NetCDF 3.5 memory limitation (?) w/ ifc and pgf90 
Author Message
 NetCDF 3.5 memory limitation (?) w/ ifc and pgf90

Hi,

I'm allocating (either through ALLOCATE or the traditional method) two
integer arrays of size (4, 9200000) named IPOS and TEMPNLEGS.  Whee.
The fortran side works fine, and the array is allocated and all is
good.  However, when I try to use netcdf to drop a chunk of data into
that array:

     file2_stat = NF90_OPEN(coeffile, NF90_SHARE, ncid3)
     status = NF90_INQ_VARID(ncid3, "position"           , positionid)
     status = NF90_INQ_VARID(ncid3, "nlegen"             , nlegenid)
     status = NF90_GET_VAR(ncid3, positionid, IPOS)
     IF (status /= NF90_NOERR) CALL handle_err(status,
'nf90_get_var1')
     status = NF90_GET_VAR(ncid3, nlegenid, TEMPNLEGS)
     IF (status /= NF90_NOERR) CALL handle_err(status,
'nf90_get_var2')

Okay, the resulting error from netcdf is error -57, nf90_eedge.  I'm
unable to find documentation on the meaning of this error, but I found
if I reduce the size of my arrays to approximately (4,8000000), then
the error no longer appears. Me: Doctor, my arm hurts when I do this.
Doctor: Don't do that.

However, the kicker is that sometimes I don't get the error message
when reading in certain datasets using the same executable. The arrays
are the same size between datasets, and the data being dumped into
them are the same size i.e., same number of values being pushed into
them.

The same datasets fail consistently as well as the working datasets
work consistently (i.e., it's not a random error from run to run).

Something that may help me get around this is how to determine the
size of the UNLIMITED dimension in netcdf.  That would allow me to
specify a smaller upper bound on the size of the array, however, it
doesn't address why it sometimes works and sometimes doesn't.

Comments, questions, and suggestions are appreciated.

-Ben



Wed, 19 Jul 2006 08:23:50 GMT  
 NetCDF 3.5 memory limitation (?) w/ ifc and pgf90

Update from the OP:

I think I've found the source of the problem.  The array size is 4,9200000
however the size of the unlimited dimension LESS than 9200000 in the
failing files and GREATER than 9200000 in the working files.  This implies
to me that something may be automatically failing when it's not supposed
to.  Here's the ncdump output (note: netcdf/ncdump orders array subscripts
in reverse compared to fortran):

WORKING:
netcdf 03031300_085500_080_010_00-05-15_00_coef {
dimensions:
        cols = 4 ;
        compact_coefficients = UNLIMITED ; // (14109368 currently)
variables:
        int position(compact_coefficients, cols) ;
        int nlegen(compact_coefficients, cols) ;

NOT WORKING:
netcdf 03031223_085500_080_010_00-05-15_00_coef {
dimensions:
        cols = 4 ;
        compact_coefficients = UNLIMITED ; // (8082891 currently)
variables:
        int position(compact_coefficients, cols) ;
        int nlegen(compact_coefficients, cols) ;

I suspect that netcdf is seeing me try to read a number of values less
than the allocated amount and complains.  This is why reducing the array
size to 4,8000000 worked -- because it was less than the 8082891 above.

The problem I face is that a priori, I do not know how big the netcdf
unlimited dimension is, I only know the max number of possible
values to be read is 9200000.  The unlimited dimension can theoretically
vary anywhere from 1 to 9200000*60 in the current setup, however most of
the time it is > 9200000.  The dataset is sparse meaning that I will
always have less than 9200000 actual values that are being read -- i'm not
concerned about this problem.  

If there's a way to inquire the size of the unlimited dimension, I could
allocate the arrays to be less than or equal to 9200000 as necessary.  
If you know how, please let me/us know as well.

Regards,

Ben


Quote:
> Hi,

> I'm allocating (either through ALLOCATE or the traditional method) two
> integer arrays of size (4, 9200000) named IPOS and TEMPNLEGS.  Whee.
> The fortran side works fine, and the array is allocated and all is
> good.  However, when I try to use netcdf to drop a chunk of data into
> that array:

>      file2_stat = NF90_OPEN(coeffile, NF90_SHARE, ncid3)
>      status = NF90_INQ_VARID(ncid3, "position"           , positionid)
>      status = NF90_INQ_VARID(ncid3, "nlegen"             , nlegenid)
>      status = NF90_GET_VAR(ncid3, positionid, IPOS)
>      IF (status /= NF90_NOERR) CALL handle_err(status,
> 'nf90_get_var1')
>      status = NF90_GET_VAR(ncid3, nlegenid, TEMPNLEGS)
>      IF (status /= NF90_NOERR) CALL handle_err(status,
> 'nf90_get_var2')

> Okay, the resulting error from netcdf is error -57, nf90_eedge.  I'm
> unable to find documentation on the meaning of this error, but I found
> if I reduce the size of my arrays to approximately (4,8000000), then
> the error no longer appears. Me: Doctor, my arm hurts when I do this.
> Doctor: Don't do that.

> However, the kicker is that sometimes I don't get the error message
> when reading in certain datasets using the same executable. The arrays
> are the same size between datasets, and the data being dumped into
> them are the same size i.e., same number of values being pushed into
> them.

> The same datasets fail consistently as well as the working datasets
> work consistently (i.e., it's not a random error from run to run).

> Something that may help me get around this is how to determine the
> size of the UNLIMITED dimension in netcdf.  That would allow me to
> specify a smaller upper bound on the size of the array, however, it
> doesn't address why it sometimes works and sometimes doesn't.

> Comments, questions, and suggestions are appreciated.

> -Ben



Wed, 19 Jul 2006 08:49:36 GMT  
 NetCDF 3.5 memory limitation (?) w/ ifc and pgf90

Update #2:

I found out how to read the size of the unlimited dimension and thus I
believe I can solve the problems from here.

http://www.unidata.ucar.edu/packages/netcdf/f90/Documentation/f90-htm...
http://www.unidata.ucar.edu/packages/netcdf/f90/Documentation/f90-htm...

As usual, RTFM. :-o

Hopefully this will answer someone else's questions in the future.

-Ben



Wed, 19 Jul 2006 09:42:08 GMT  
 NetCDF 3.5 memory limitation (?) w/ ifc and pgf90

Quote:

> Update #2:

> I found out how to read the size of the unlimited dimension and thus I
> believe I can solve the problems from here.

> http://www.unidata.ucar.edu/packages/netcdf/f90/Documentation/f90-htm...
> http://www.unidata.ucar.edu/packages/netcdf/f90/Documentation/f90-htm...

> As usual, RTFM. :-o

> Hopefully this will answer someone else's questions in the future.

I presume you read about the function to inquire about the dimensions.

BTW, you can get a meaningful error message from netCDF if you use the strerror function.
E.g. see below, from some of my f90 code, if the global attribute write is unsuccessful,
the nf90_status value is set and I output my own error message to the end of which I
append the netCDF error string, NF90_STRERROR( NF90_Status ).

      NF90_Status = NF90_PUT_ATT( NC_FileID,    &
                                  NF90_GLOBAL,   &
                                  PLATFORM_NAME_GATTNAME, &
                                  TRIM( Platform_Name ) )

      IF ( NF90_Status /= NF90_NOERR ) THEN
        Error_Status = WARNING
        CALL Display_Message( ROUTINE_NAME, &
                              'Error writing '//PLATFORM_NAME_GATTNAME//' attribute to '//
&
                              TRIM( NC_fileNAME )//' - '// &
                              TRIM( NF90_STRERROR( NF90_Status ) ), &
                              Error_Status, &
                              Message_Log = Message_Log )
      END IF

cheers,

paulv

--
Paul van Delst



Sun, 23 Jul 2006 21:38:22 GMT  
 
 [ 4 post ] 

 Relevant Pages 

1. pgf90 compiler and netcdf help

2. xlf / nag / ifort / pgf90 vs g95 / sun f90 / ifc 7.1

3. LAPACK95 and Intel IFC and MKL or PGF90

4. two compilers different results (ifc & pgf90)

5. Moving from Lahey/IMSL to ifc/pgf90

6. VSE memory limitations?

7. Ruby windows memory limitation

8. Sudden memory limitations running PCScheme 4.02PL1 (Geneva) in Windows 3.11

9. ModelSim Evaluation memory limitation has been reached

10. Memory limitation

11. memory error/limitation on VisualFortran6.1?

12. Memory limitations of g77 under Linux

 

 
Powered by phpBB® Forum Software