writing about 100 colomns of real data into text file 
Author Message
 writing about 100 colomns of real data into text file

I am trying to write about 100 columns of real data into a text file,
but somehow I get an 'Aborted' message when my write statement is
executed.

I've decided to test if there were some limitations on the number of
columns I could write, and apparently the following program works fine:

! This program demonstrates that it is possible to use *a lot* of columns with
! data when writing data-output to a textfile.

program test_max_write_columns

  implicit none

  integer, parameter                       :: dp = &
                                            selected_real_kind(2*precision(1.0))
  integer                                  :: cols
  real(kind=dp), dimension(:), allocatable :: myvar
  character(len=100)                       :: formatstring

  open(unit=20, file="test_max_write_columns.dat", &
       status="replace", access="sequential", action="write")

  print *, "How many columns of real data do you want?"
  read *, cols

  allocate(myvar(cols))
  call random_number(myvar)

  ! Build up formatstring dynamically with 'cols' columns.
  write(unit=formatstring, fmt="(A, I0, A)") "(", cols, "ES25.15e2)"

  ! Write to file and see if we can get that many columns...
  write(unit=20, fmt=formatstring) myvar

  close(unit=20)

  deallocate(myvar)

end program test_max_write_columns

If I enter for example that I want 104 columns, then i cleanly get a .dat file
containing 104 columns of real data-values.

Now I have a bigger application in which i basically does the same.  The write
statement at which my program gets aborted is the following:

        write(unit=unit_number, fmt=formatstring) 2**i-1,       &
              mymeans(i),                                       &
              minval(myrel_errs(i,:)), maxval(myrel_errs(i,:)), &
              myvariances(i),                                   &
              myrel_errs(i,:)

Formatstring gets built as follows:

    write(unit=formatstring, fmt="(A, I3, A)") "(I10, ", nbruns+4, "ES25.15e2)"

but apparently things fail when I use that format string in the first write
statement: it works when nb_runs is 36, but it fails when nb_runs is 37.  If
i change the format-descriptor to for example ES10.5e2 it works for larger
values than 37.

So my main guess is that it is a problem which is related to the
format-descriptor or so, but then I don't understand why my little test-program
*can* in fact write 104 columns, and this code from my application *can't*...

Does anybody have any suggestions on what could be wrong or things to look for?

Thanks,
Bart

--
        "Share what you know.  Learn what you don't."



Mon, 26 Nov 2007 03:05:34 GMT  
 writing about 100 colomns of real data into text file
When I need many columns to be written, I use ADVANCE='NO' and write
each column in a different statement. Maybe it helps.


Mon, 26 Nov 2007 03:25:49 GMT  
 writing about 100 colomns of real data into text file


Quote:
> I am trying to write about 100 columns of real data into a text file,
> but somehow I get an 'Aborted' message when my write statement is
> executed.

> I've decided to test if there were some limitations on the number of
> columns I could write, and apparently the following program works fine:

  [example elided]

Ok. Note that compilers are allowed to place limits on record size.
Including a maxrec= specifier in the OPEN can help, though the compiler
may also have a limit on the allowable size of maxrec.

However, the fact that your test program ran ok argues against that as
the problem. It doesn't absolutely prove it, but argues against it. I do
note that you didn't show the OPEN from the "real" program. That can be
relevant. I'm guessing that it is probably ok, but that is only a guess
on my part.

Quote:

>         write(unit=unit_number, fmt=formatstring) 2**i-1,       &
>               mymeans(i),                                       &
>               minval(myrel_errs(i,:)), maxval(myrel_errs(i,:)), &
>               myvariances(i),                                   &
>               myrel_errs(i,:)

> Formatstring gets built as follows:

>     write(unit=formatstring, fmt="(A, I3, A)") "(I10, ", nbruns+4,
>     "ES25.15e2)"

> but apparently things fail when I use that format string in the first write
> statement: it works when nb_runs is 36, but it fails when nb_runs is 37.  If
> i change the format-descriptor to for example ES10.5e2 it works for larger
> values than 37.

Hmm. I don't see the connection between nbruns and the number of data
items in the write statement. I suppose that is probably ok because I
don't see why changing the ES25.15e2 to ES10.5e2 would change that...

Oh. Hold on a sec. What is the declaration of formatstring? Any chance
that the character string is one character too short for the ES25.15e2
form? Hard to say what the effects of truncating the last character
would be, but it just might possibly happen to work as long as the
number of items was small enough.

--
Richard Maine                       |  Good judgment comes from experience;
email: my first.last at org.domain  |  experience comes from bad judgment.
org: nasa, domain: gov              |        -- Mark Twain



Mon, 26 Nov 2007 03:27:26 GMT  
 writing about 100 colomns of real data into text file

Quote:

> Ok. Note that compilers are allowed to place limits on record size.
> Including a maxrec= specifier in the OPEN can help, though the compiler
> may also have a limit on the allowable size of maxrec.

> However, the fact that your test program ran ok argues against that as
> the problem. It doesn't absolutely prove it, but argues against it.

This is why I wrote this simple test-program.  Apparently, my compiler
can handle such cases... (I'm using the F-compiler for this, with as
only option the -O3 option)

Quote:
> I do
> note that you didn't show the OPEN from the "real" program. That can be
> relevant. I'm guessing that it is probably ok, but that is only a guess
> on my part.

This is the OPEN from the "real" program:

    open(unit=unit_number, file=filename, &
         iostat=ios, status="replace", access="sequential", action="write")

Quote:
>> but apparently things fail when I use that format string in the first write
>> statement: it works when nb_runs is 36, but it fails when nb_runs is 37.  If
>> i change the format-descriptor to for example ES10.5e2 it works for larger
>> values than 37.

Now I have for example ES9.2e2 and i can use at least nb_runs=100.  So depending on the value of this edit-descriptor, my program gets aborted or not...

Quote:
> Hmm. I don't see the connection between nbruns and the number of data
> items in the write statement. I suppose that is probably ok because I
> don't see why changing the ES25.15e2 to ES10.5e2 would change that...

There are actually nb_runs + 5 columns in my data file.  The first column is
an integer value, and the other nb_runs+4 columns are reals of a certain
real kind that i want to write out saving as much as precision as i can.

Quote:
> Oh. Hold on a sec. What is the declaration of formatstring?

    character(len=50)                        :: formatstring

Quote:
> Any chance
> that the character string is one character too short for the ES25.15e2
> form? Hard to say what the effects of truncating the last character
> would be, but it just might possibly happen to work as long as the
> number of items was small enough.

I guess len=50 should be enough in my case.  I've PRINTed format string right
before it's used and it looks ok.

I really don't understand why i can write 100 colmns with my example test
program, but not in my other program... they both practically do the same, but
still the "real" program aborts if the width from my edit descriptor gets too
big.

Any other things I could check?

Regards,
Bart

--
        "Share what you know.  Learn what you don't."



Mon, 26 Nov 2007 04:28:40 GMT  
 writing about 100 colomns of real data into text file

Quote:

> Ok. Note that compilers are allowed to place limits on record size.
> Including a maxrec= specifier in the OPEN can help, though the compiler
> may also have a limit on the allowable size of maxrec.

> However, the fact that your test program ran ok argues against that as
> the problem. It doesn't absolutely prove it, but argues against it.

Hmm... Richard... I must say 'sorry' again... i though I was compiling
my testprogram with the same compiler as my "real" program, but
apparently this was not the case...

The test-program compiled with g95 so apparently g95 can handle that
case, but when I compile my test-program with the F-compiler, i get the
following message:

Buffer overflow on output
Program terminated by fatal I/O error
Aborted

So it is indeed probably a problem on the limit of the compiler.  Now my
question is: is there a way I *can* write the real values with the
ES25.15e2 format descriptor?  Maybe by writing them as individual values
instead of row by row?
I don't have much experience in writing to files like this.  I mostly
write a matrix line by line, but apparently now i can't do this because
of this compiler limitation...  Is the ADVANCE=NO option as suggested by
another poster the way to go?

Regards,
Bart

--
        "Share what you know.  Learn what you don't."



Mon, 26 Nov 2007 04:37:30 GMT  
 writing about 100 colomns of real data into text file

Quote:

> I am trying to write about 100 columns of real data into a text file,
> but somehow I get an 'Aborted' message when my write statement is
> executed.

(snip)

Quote:
>   ! Build up formatstring dynamically with 'cols' columns.
>   write(unit=formatstring, fmt="(A, I0, A)") "(", cols, "ES25.15e2)"

While there are cases that require dynamic format strings, this
(usually) isn't one of them.  If you put the largest number you
could expect, (or that your compiler allows), such as
  "(I10,9999ES25.15e2)"   and note that there aren't many printers
that will print lines that long.  There is no penalty for extra
format items at the end that are not used.

(snip)

Quote:
> Formatstring gets built as follows:
>     write(unit=formatstring, fmt="(A, I3, A)") "(I10, ", nbruns+4, "ES25.15e2)"
> but apparently things fail when I use that format string in the first write
> statement: it works when nb_runs is 36, but it fails when nb_runs is 37.  If
> i change the format-descriptor to for example ES10.5e2 it works for larger
> values than 37.

It seems to fail on the change from a 1010 character record to a 1035
character record, which looks suspiciously like a 1024 character
boundary.   Is it possible that your real program sets a record length
limit on the OPEN statement?

-- glen



Mon, 26 Nov 2007 05:26:23 GMT  
 writing about 100 colomns of real data into text file


Quote:


> > Ok. Note that compilers are allowed to place limits on record size.
> > Including a maxrec= specifier in the OPEN can help,
...
> The test-program compiled with g95 so apparently g95 can handle that
> case, but when I compile my test-program with the F-compiler, i get the
> following message:

> Buffer overflow on output
> Program terminated by fatal I/O error
> Aborted

I suggest trying the specifier in the OPEN statement, except that I
"spelled" it incorrectly above. It is recl= instead of maxrec=. I had
also noticed that you seemed to be hitting it at about 1024 characters,
as glenn commented, and that did seem like a plausible default record
size limit. But the reported success of the test case threw me off
track. I suggest trying recl=some-sufficiently-large-number in the OPEN.

Quote:
>Is the ADVANCE=NO option as suggested by
> another poster the way to go?

I'll not say that it couldn't work.... but I wouldn't expect it to.

Advance='no' is convenient for when you want or need to use multiple
write statements. But it is still writing to a sequential formatted
file. The fundamental file characteristics (like record length limits)
should not be any different. I use the word "should" a lot here. It is
possible that the implementation might happen to work in such a way that
the effective limit is different, but I wouldn't expect it.

The f2003 access='stream' is a different matter. In fact, avoiding
record length limitations has been cited as a reason for doing the
formatted version of access='stream'.  (I don't fully understand why it
would help, and I recall someone here - Giles I think - emphasizing that
it shouldn't... but whether it should or not, that *WAS* advanced as an
argument). Anyway, you don't have f2003 yet.

--
Richard Maine                       |  Good judgment comes from experience;
email: my first.last at org.domain  |  experience comes from bad judgment.
org: nasa, domain: gov              |        -- Mark Twain



Mon, 26 Nov 2007 06:47:26 GMT  
 writing about 100 colomns of real data into text file
(snip)

Quote:
> The f2003 access='stream' is a different matter. In fact, avoiding
> record length limitations has been cited as a reason for doing the
> formatted version of access='stream'.  (I don't fully understand why it
> would help, and I recall someone here - Giles I think - emphasizing that
> it shouldn't... but whether it should or not, that *WAS* advanced as an
> argument). Anyway, you don't have f2003 yet.

Well, the record oriented I/O is from the beginnings of fortran,
allocate a buffer (or two) the length of the record (print line) fill
it, and then write it out.  Each WRITE always starts with an empty
buffer.

Stream is the name C uses for its I/O system, sort of like Fortran
I/O UNITs.  For text streams, C makes no guarantee that there isn't a
record length limit, and there likely will be if the underlying file
system has one.  For unix systems, the file system does not have a
record structure, and bytes can be written independent of any record
length.  My interpretation of access='stream' would be a C stream.
(Especially if the C library is used for I/O.)   The C fopen function
can open a stream in text or binary mode.  Some things depend on which
mode is used.

Using C files with the IBM VSE, MVS, or CMS systems can be somewhat
complicated.  Files are written with fixed length records, or with
variable length records with a length indication at the beginning.
The C/370 library can even convert to/from ASA carriage control
characters for text files.  I expect access='stream' to have similar
complications under those systems.

The DEC/Compaq/HP VMS system also has a record oriented file system,
but I believe it also has the ability to do stream files without record
marks.

-- glen



Mon, 26 Nov 2007 07:31:59 GMT  
 writing about 100 colomns of real data into text file

Quote:


> (snip)

> > The f2003 access='stream' is a different matter. In fact, avoiding
> > record length limitations has been cited as a reason for doing the
> > formatted version of access='stream'.  (I don't fully understand why it
> > would help, and I recall someone here - Giles I think - emphasizing that
> > it shouldn't... but whether it should or not, that *WAS* advanced as an
> > argument). Anyway, you don't have f2003 yet.

> Well, the record oriented I/O is from the beginnings of Fortran,
> allocate a buffer (or two) the length of the record (print line) fill
> it, and then write it out.  Each WRITE always starts with an empty
> buffer.

There is no feature of Formatted I/O that requires that model.
The buffer needs to be at least as long as the amount of a given
record read or written by a single READ or WRITE statement in
order for the tab edit descriptors (particularly the tab left
and absolute tab specifiers) to work correctly.  But that's a
requirement for Fortran 2003's stream I/O too.  *If* stream I/O
had disallowed the use of the left tab and absolute tab specifiers
in the format, that would have constituted a significant change.
Even with the tab specifiers, arbitrary length records should
still be possible using multiple READ or WRITE statements and
non-advancing I/O - even without the stream feature.

The only difference I can find between stream I/O and non-stream
I/O in F2003 is that stream I/O requires the programmer to handle
record marks manually.  But, since stream I/O doesn't provide
any control over what the record mark is, that manual handling
confers no new capabilities to the programmer, just a burden.

Quote:
> [...] My interpretation of access='stream' would be a C stream.

But that *isn't* the F2003 interpretation.  Any analogy to C is
probably misleading.

--
J. Giles



Mon, 26 Nov 2007 07:59:47 GMT  
 writing about 100 colomns of real data into text file

Richard E Maine schrieb:

Quote:


....
> >Is the ADVANCE=NO option as suggested by
> > another poster the way to go?

> I'll not say that it couldn't work.... but I wouldn't expect it to.

> Advance='no' is convenient for when you want or need to use multiple
> write statements. But it is still writing to a sequential formatted
> file. The fundamental file characteristics (like record length limits)
> should not be any different. I use the word "should" a lot here. It is
> possible that the implementation might happen to work in such a way that
> the effective limit is different, but I wouldn't expect it.

I just tried opening with recl= some value, and at least in gfortran,
you get the runtime error:
Fortran runtime error: End of record
when you try to write more columns. I know that in CVF, recl is set to
2147483647 or 2147483640  or some other value (depends on the other
open options), when it's not specified.
Remaking your open should work.

Something else, I see the use of "I0" in the format to make the
formatstring, is this use allowed by the standard (I talk about the
zero width, which means in CVF and gfortran automatic width)?



Mon, 26 Nov 2007 08:03:59 GMT  
 writing about 100 colomns of real data into text file

Quote:

> Something else, I see the use of "I0" in the format to make the
> formatstring, is this use allowed by the standard (I talk about the
> zero width, which means in CVF and gfortran automatic width)?

Yes. That is an f95 feature. It was one of the few new features in f95.

--
Richard Maine
email: my last name at domain
domain: summertriangle dot net



Mon, 26 Nov 2007 09:03:49 GMT  
 writing about 100 colomns of real data into text file

Quote:

> It seems to fail on the change from a 1010 character record to a 1035
> character record, which looks suspiciously like a 1024 character
> boundary.

Yep, this 1024 boundary was indeed the problem.  I'm using the NAGWare
f95 compiler and according to the manpage it has a default maximum
record length of 1024 for formatted output.

Quote:
> Is it possible that your real program sets a record length
> limit on the OPEN statement?

I didn't set 'recl' in my OPEN statement, so I was using the default of
1024.  The reason why my testprogram *did* work with g95 is because it
has a default recl of 50000000.

Adding 'recl=<whatever i want>' solves my problem.

Thanks for helping me figuring this out guys!

Regards,
Bart

--
        "Share what you know.  Learn what you don't."



Mon, 26 Nov 2007 17:08:05 GMT  
 writing about 100 colomns of real data into text file

Quote:

> I suggest trying the specifier in the OPEN statement, except that I
> "spelled" it incorrectly above. It is recl= instead of maxrec=. I had
> also noticed that you seemed to be hitting it at about 1024 characters,
> as glenn commented, and that did seem like a plausible default record
> size limit. But the reported success of the test case threw me off
> track. I suggest trying recl=some-sufficiently-large-number in the OPEN.

This is indeed the (portable!) solution to my problem.

One more question from my side: does the standard specify a limit on the value
that can be specified as recl?  Or is it dependent on the compiler, operating
system, filesystem,...?
(sorry, i do not have a copy of the official Fortran standard...)

Quote:
>>Is the ADVANCE=NO option as suggested by
>> another poster the way to go?

> I'll not say that it couldn't work.... but I wouldn't expect it to.

> Advance='no' is convenient for when you want or need to use multiple
> write statements. But it is still writing to a sequential formatted
> file. The fundamental file characteristics (like record length limits)
> should not be any different. I use the word "should" a lot here. It is
> possible that the implementation might happen to work in such a way that
> the effective limit is different, but I wouldn't expect it.

You are right.  When I tried adding advance="no" to my write statements,
i still got stuck on this boundary of 1024.

Again, thanks for your help and suggestions!

Bart

--
        "Share what you know.  Learn what you don't."



Mon, 26 Nov 2007 17:11:37 GMT  
 writing about 100 colomns of real data into text file

Quote:
> One more question from my side: does the standard specify a limit on the value
> that can be specified as recl?  Or is it dependent on the compiler, operating
> system, filesystem,...?

THe limits in the standard are to be understood as guarantees, not as limits.
There is, I believe, no limit for the record length. Usually, such a limit
is set by the RTL or even the OS. For instance, if you want to write a for-
matted record to a normal text file on VMS, that record cannot be longer
than 32766 characters.

        Jan



Mon, 26 Nov 2007 18:02:35 GMT  
 writing about 100 colomns of real data into text file

Quote:

> THe limits in the standard are to be understood as guarantees, not as limits.
> There is, I believe, no limit for the record length.

Just being curious, i've tried to look this up in the September 23, 2002
committee draft 'ISO/IEC 1539-1' but i could not find anything
concerning the guaranteed/maximum record lengths.

Where exactly in the standards (or my committee draft) is this
described?

Regards,
Bart

--
        "Share what you know.  Learn what you don't."



Mon, 26 Nov 2007 20:11:32 GMT  
 
 [ 25 post ]  Go to page: [1] [2]

 Relevant Pages 

1. I have 100 data files, I want to join them together as one data file

2. Real time data display with LabVIEW 6i and WebDAQ/100

3. Please help Visual Basic Programer with simple read/write text file routine---Real Basic 2.1

4. 100% private, 100% offshore 100% crap 8347.

5. 100% Offshore, 100% Private, 100% Online 8347

6. Weird behaviour of file.write() (no data written)?

7. SRC: Music Synthesizer written in 100% Python

8. Write 2D Text Array in a text file

9. extracting top 100 data using awk or sed/perl

10. 100 digits x $ 100 challenge

11. AMD 486DX/100 or Intel 486DX/100?

12. Reading a binary file / writing binary data to a file

 

 
Powered by phpBB® Forum Software