writing large files 
Author Message
 writing large files

I'm using g95 and I'm having trouble writing direct access files of
more than 2GB in size from fortran.  I assume that this is an
addressability problem.  I'm using g95, but unless I'm mistaken, it
doesn't allow for 64-bit compilation (I can run 64-bit executables on
this machine, which is an AIX server).  Am I stuck?  Have I perhaps
mis-diagnosed the problem?


Tue, 06 Sep 2011 22:39:15 GMT  
 writing large files
It could be the executable, it could be the file system.


Wed, 07 Sep 2011 02:49:20 GMT  
 writing large files

Quote:

> I'm using g95 and I'm having trouble writing direct access files of
> more than 2GB in size from Fortran.  I assume that this is an
> addressability problem.  I'm using g95, but unless I'm mistaken, it
> doesn't allow for 64-bit compilation (I can run 64-bit executables on
> this machine, which is an AIX server).  Am I stuck?  Have I perhaps
> mis-diagnosed the problem?

One solution is to use more than one file, and select
the record from the appropriate file based on its record
number.  

Not so long ago many file systems ran into the problem
that the file system itself used 32 bit offsets.
That should have been fixed on most by now.

Even after that, the C fseek() and ftell() routines,
which used a C (long int) to specify the offset became
the limiting factor.  (Many Fortran libraries now use
the C library to access files.)  For 32 bit code,
the fseek64() and ftell64() functions were added, but
a better solution is to go to 64 bits.

Since the system couldn't always tell which programs
could properly access files larger than 2GB and
which couldn't, some systems require a bit in
the executable file, something like the
"64 bit file aware" bit, to allow such access.
That avoids accidentally writing over the beginning
of a file when the offset wraps.

-- glen



Wed, 07 Sep 2011 02:44:47 GMT  
 writing large files

Quote:
> Even after that, the C fseek() and ftell() routines,
> which used a C (long int) to specify the offset became
> the limiting factor.  (Many Fortran libraries now use
> the C library to access files.)  For 32 bit code,
> the fseek64() and ftell64() functions were added, but
> a better solution is to go to 64 bits.

I've never heard of fseek64 or ftell64. POSIX defined fseeko and
ftello, which take an off_t argument instead of long. For the lower
level POSIX I/O functions (open, read, lseek, etc.) there are xxx64
variants.

Quote:
> Since the system couldn't always tell which programs
> could properly access files larger than 2GB and
> which couldn't, some systems require a bit in
> the executable file, something like the
> "64 bit file aware" bit, to allow such access.
> That avoids accidentally writing over the beginning
> of a file when the offset wraps.

I've never heard of such a bit. On unix systems you typically have to
define a bunch or preprocessor flags to enable large file support when
compiling. The command "getconf LFS_CFLAGS" shows which flags are
needed; on Linux 32-bit and Solaris ?? this prints out
"-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64", on 64-bit Linux it's
empty. You also have to open files with O_LARGEFILE, otherwise the
kernel prevents any access beyond the 2 GB limit.

It might be that AIX does this slightly differently, and since
presumably AIX is not a very common platform for g95 some bug in
setting these preprocessor flags has gone unnoticed.

--
JB



Wed, 07 Sep 2011 16:16:27 GMT  
 writing large files

Quote:
> Even after that, the C fseek() and ftell() routines,
> which used a C (long int) to specify the offset became
> the limiting factor.  (Many Fortran libraries now use
> the C library to access files.)  For 32 bit code,
> the fseek64() and ftell64() functions were added, but
> a better solution is to go to 64 bits.

I've never heard of fseek64 or ftell64. POSIX defined fseeko and
ftello, which take an off_t argument instead of long. The way large
file support is done in POSIX is with the xxx64 functions,
e.g. open64, lseek64, fseeko64 etc., where off_t is typedefed to
off64_t and so forth.

Quote:
> Since the system couldn't always tell which programs
> could properly access files larger than 2GB and
> which couldn't, some systems require a bit in
> the executable file, something like the
> "64 bit file aware" bit, to allow such access.
> That avoids accidentally writing over the beginning
> of a file when the offset wraps.

I've never heard of such a bit. On unix systems you typically have to
define a bunch or preprocessor flags to enable large file support when
compiling. The command "getconf LFS_CFLAGS" shows which flags are
needed; on Linux 32-bit and Solaris ?? this prints out
"-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64", on 64-bit Linux it's
empty. You also have to open files with O_LARGEFILE, otherwise the
kernel prevents any access beyond the 2 GB limit.

It might be that AIX does this slightly differently, and since
presumably AIX is not a very common platform for g95 some bug in
setting these preprocessor flags has gone unnoticed.

--
JB



Wed, 07 Sep 2011 22:15:18 GMT  
 
 [ 5 post ] 

 Relevant Pages 

1. Problems writing out large files

2. Need help with large file reading and writing.

3. Use of Arrays Vs output .dbf files - for counts in large files

4. File size and large files

5. newbie question: Why is an .exe file much larger than a .com file:

6. end of file for large files

7. Weird behaviour of file.write() (no data written)?

8. Toy VRML parser update to speed reading/writing of large MFNumbers

9. Problem with WRITE for large arrays

10. Most efficient way to write a large matrix to disk

11. Reading from input file writing to output file

12. Write to 2 file from pipe without using temp file

 

 
Powered by phpBB® Forum Software