help reading fortran generated binary data... 
Author Message
 help reading fortran generated binary data...

Hi All,

First, for those not interested in the "long" story (below), here's my
basic problem...

I've read a value from binary file using the following code:


point value)

The value *should* be any of 0, 90, 180, or 270.

What I get instead is 4 variations of an *extreme* (large or small) number,
such as:

5.4388530464437965e+185

So, is there some magic that needs to be performed on the above to extract
the value I'm actually after?

Details below...

--------------------------------------------------------------------------------------

I have a tcl program designed to read an old fortran Unformatted Sequential
file format (as described in (much) earlier threads).  Up until know, I
have only needed certain data from the file - which I have been able to
successfully retrieve.  Up until now, I have just "read through" data that
I didn't care about.  Now, I have a need to use a few pieces of data that I
used to just ignore.  Ultimately, I'm having trouble pulling "sensible"
data from this section of the file and thought someone could give me a push
in the right direction...

Basically, I have some old FORTRAN code that is capable of reading the same
file - which I have used as a guide when writing my tcl version.  In the
case of the data in question, the FORTRAN code reads the data into a common
block defined as follows:

      INTEGER NUMEX2,EXTNUM
      REAL*8 EXTORI,EXTX,EXTY,EXTZ
      LOGICAL EXTMIR
      INTEGER*2 EXTASM,EXTSUB,EXTPAR,EXTJOB
      CHARACTER*1 EXTSTK(MAXSTK)
      INTEGER EXTDUM(MAXSTK+12)
      EQUIVALENCE(EXTORI,EXTDUM)
      COMMON/NSTEXT/EXTORI,EXTX,EXTY,EXTZ,EXTMIR,EXTASM,EXTSUB,
     *              EXTPAR,EXTJOB,EXTNUM,EXTSTK,NUMEX2

The FORTRAN reads the data into the above common block by reading a known
number of INTEGER values into the EXTDUM array as defined above.  The code
for the read is as follows:

               IF (NUMEX2.GT.0)
     *            READ(FIL,ERR=900,END=900)(EXTDUM(I),I=1,NUMEX2)

Again, currently I am successfully "reading through" this data (which is
required so the read process doesn't get lost), but now I need to "keep"
some of it.  What I'm currently after is the data stored in the EXTORI
variable above - which is defined as a REAL*8.

So, once I'm looking at the proper section of the file, it seems to me that
the following should return the proper data:


While this seems to read the proper "amount" of data (again, I know this
because if I don't account for the data, the read "gets lost" and the
process will hang), the resulting value is not sensible.  The values stored
at this location in the file are angles, and are basically 0, 90, 180, or
270 (as returned by the original FORTRAN reader).  The value stored my the
tcl "theData" variable above contains *huge* exponent values, both positive
and negative (depending upon the file being read).  For example, I might
get the following back:

5.4388530464437965e+185

Interestingly, I only get 4 different values back (which again should
equate to 0, 90, 180, and 270), so I *think* I'm getting the correct data,
and likely just not interpreting it correctly.

So, is there some mathematical magic that needs to performed on the above
value to reveal the number I'm really after?

Thanks for any input, as I'm really stumped here...

Jeff



Mon, 04 Feb 2008 00:36:03 GMT  
 help reading fortran generated binary data...
On Wed, 17 Aug 2005 16:36:03 GMT, "Jeff Godfrey"

Quote:

>Hi All,

>First, for those not interested in the "long" story (below), here's my
>basic problem...

>I've read a value from binary file using the following code:


>point value)

>The value *should* be any of 0, 90, 180, or 270.

>What I get instead is 4 variations of an *extreme* (large or small) number,
>such as:

>5.4388530464437965e+185

Hi Jeff,
what I would try next is to write out the 4 values you expect into a
binary file and have a look at it with a hex editor: question of
finding out what Tcl expects to see as 0, 90, 180, or 270 represented
as double.
Then look at the raw binary data you get. Can you locate the patterns
from above  anywhere ?
If not, maybe FORTRAN stored doubles in a different format?
HTH and good luck
Helmut Giese


Mon, 04 Feb 2008 00:50:22 GMT  
 help reading fortran generated binary data...
I suppose this is to obvious ... but have you already considered that
the byte order assumptions of the two systems might be different!
Perhaps you can switch the interpretation order of the bytes from the
file.  As I recall binary scan had some options for doing that.

Jim



Mon, 04 Feb 2008 00:50:20 GMT  
 help reading fortran generated binary data...


Quote:
> On Wed, 17 Aug 2005 16:36:03 GMT, "Jeff Godfrey"

>>Hi All,

>>First, for those not interested in the "long" story (below), here's my
>>basic problem...

>>I've read a value from binary file using the following code:


>>floating
>>point value)

>>The value *should* be any of 0, 90, 180, or 270.

>>What I get instead is 4 variations of an *extreme* (large or small)
>>number,
>>such as:

>>5.4388530464437965e+185

> Hi Jeff,
> what I would try next is to write out the 4 values you expect into a
> binary file and have a look at it with a hex editor: question of
> finding out what Tcl expects to see as 0, 90, 180, or 270 represented
> as double.
> Then look at the raw binary data you get. Can you locate the patterns
> from above  anywhere ?
> If not, maybe FORTRAN stored doubles in a different format?
> HTH and good luck
> Helmut Giese

Helmut,

Short of anyone just "knowing" the answer, that's probably what I'll need
to do.

Thanks for the suggestion.

Jeff



Mon, 04 Feb 2008 03:00:08 GMT  
 help reading fortran generated binary data...
Quote:
----- Original Message -----

Newsgroups: comp.lang.tcl
Sent: Wednesday, August 17, 2005 11:50 AM
Subject: Re: help reading fortran generated binary data...

>I suppose this is to obvious ... but have you already considered that the
>byte order assumptions of the two systems might be different! Perhaps you
>can switch the interpretation order of the bytes from the file.  As I
>recall binary scan had some options for doing that.

> Jim

Hi Jim,

I'm guessing that my problem *is* related to something like you've
mentioned
(data interpretation), but I haven't figured out how to correct it yet.  As
I don't think I mentioned it before, the FORTRAN program that created the
file originally, the FORTRAN program that can read the file, and my tcl
program are all running under WinXP Pro on an Intel platform - so I assume
byte order is the same in all cases - unless FORTRAN stores doubles
"oddly"???

Thanks,

Jeff



Mon, 04 Feb 2008 03:05:34 GMT  
 help reading fortran generated binary data...
On Wed, 17 Aug 2005 19:00:08 GMT, "Jeff Godfrey"

Quote:



>> On Wed, 17 Aug 2005 16:36:03 GMT, "Jeff Godfrey"

>>>Hi All,

>>>First, for those not interested in the "long" story (below), here's my
>>>basic problem...

>>>I've read a value from binary file using the following code:


>>>floating
>>>point value)

>>>The value *should* be any of 0, 90, 180, or 270.

>>>What I get instead is 4 variations of an *extreme* (large or small)
>>>number,
>>>such as:

>>>5.4388530464437965e+185

>> Hi Jeff,
>> what I would try next is to write out the 4 values you expect into a
>> binary file and have a look at it with a hex editor: question of
>> finding out what Tcl expects to see as 0, 90, 180, or 270 represented
>> as double.
>> Then look at the raw binary data you get. Can you locate the patterns
>> from above  anywhere ?
>> If not, maybe FORTRAN stored doubles in a different format?
>> HTH and good luck
>> Helmut Giese

>Helmut,

>Short of anyone just "knowing" the answer

That would probably have been Arjen, Tcl's FORTRAN guru :) but he
seems to not be listening right now.

My FORTRAN days are long gone and anyway I never had to deal with
floating point values as raw binary data, so I am only guessing here.

But I wouldn't be surprised if FORTRAN (or at least a sufficiently old
compiler) did use a different means of storing doubles - I mean,
FORTRAN probably predates the IEEE standard on floating point
calculation by many years. And of course you can write them out and
read them back in using programs compiled with the same compiler -
like Borland Pascal users could when using the funny 6 byte floating
point type called Real which Borland invented - but of course nobody
else could.
If you know more details about the interpretation of your 4 values you
may want to turn to a FORTRAN news group.
Good luck
Helmut Giese



Mon, 04 Feb 2008 03:33:32 GMT  
 help reading fortran generated binary data...


Quote:
> On Wed, 17 Aug 2005 19:00:08 GMT, "Jeff Godfrey"



>>> On Wed, 17 Aug 2005 16:36:03 GMT, "Jeff Godfrey"

>>>>Hi All,

>>>>First, for those not interested in the "long" story (below), here's my
>>>>basic problem...

>>>>I've read a value from binary file using the following code:


>>>>floating
>>>>point value)

>>>>The value *should* be any of 0, 90, 180, or 270.

>>>>What I get instead is 4 variations of an *extreme* (large or small)
>>>>number,
>>>>such as:

>>>>5.4388530464437965e+185

>>> Hi Jeff,
>>> what I would try next is to write out the 4 values you expect into a
>>> binary file and have a look at it with a hex editor: question of
>>> finding out what Tcl expects to see as 0, 90, 180, or 270 represented
>>> as double.
>>> Then look at the raw binary data you get. Can you locate the patterns
>>> from above  anywhere ?
>>> If not, maybe FORTRAN stored doubles in a different format?
>>> HTH and good luck
>>> Helmut Giese

>>Helmut,

>>Short of anyone just "knowing" the answer
> That would probably have been Arjen, Tcl's FORTRAN guru :) but he
> seems to not be listening right now.

> My FORTRAN days are long gone and anyway I never had to deal with
> floating point values as raw binary data, so I am only guessing here.

> But I wouldn't be surprised if FORTRAN (or at least a sufficiently old
> compiler) did use a different means of storing doubles - I mean,
> FORTRAN probably predates the IEEE standard on floating point
> calculation by many years. And of course you can write them out and
> read them back in using programs compiled with the same compiler -
> like Borland Pascal users could when using the funny 6 byte floating
> point type called Real which Borland invented - but of course nobody
> else could.
> If you know more details about the interpretation of your 4 values you
> may want to turn to a FORTRAN news group.

Helmut,

I'm guessing you're right - that the doubles are being stored in some 'odd'
format in the file.  I had a thought, though I don't know if 1) it makes
any sense, and 2) that I know how to implement (part of) it...

As previously mentioned, the FORTRAN "reader" program currently populates
the entire COMMON block in question (which is a mix of data types) by
reading this particular file section in as an array of INTEGERS.

Since I know that I can interpret INTEGERS correctly from the file, would
it be possible for me to do the same?  That is, read the appropriate number
of INTEGER values from the file (this I can do), and then "construct" the
proper datatypes from those?

Any thoughts?

Thanks,

Jeff



Mon, 04 Feb 2008 04:11:50 GMT  
 help reading fortran generated binary data...
Quote:
>Helmut,

>I'm guessing you're right - that the doubles are being stored in some 'odd'
>format in the file.  I had a thought, though I don't know if 1) it makes
>any sense, and 2) that I know how to implement (part of) it...

>As previously mentioned, the FORTRAN "reader" program currently populates
>the entire COMMON block in question (which is a mix of data types) by
>reading this particular file section in as an array of INTEGERS.

>Since I know that I can interpret INTEGERS correctly from the file, would
>it be possible for me to do the same?  That is, read the appropriate number
>of INTEGER values from the file (this I can do), and then "construct" the
>proper datatypes from those?

>Any thoughts?

Jeff,
this requires more detailed knowledge than I have. But you could of
course check, if you find 0, 90, 180 or 270 /as integers/ in your
binary data.
Hm, the following doesn't probably tell you anything new, but in case
it helps ...
        ---
#
# bin2Hex: convert binary data into hex representation, adding $delim
if defined
#
proc bin2Hex {binStr {delim ""}} {
    set s ""
    for {set i 0} {$i < [string length $binStr]} {incr i} {
        binary scan [string index $binStr $i] H2 cc
        append s $cc $delim
    }
    return $s
Quote:
}

        ---

You might have to change 'H2' to 'h2' - byte order stuff.
Good luck
Helmut Giese



Mon, 04 Feb 2008 05:12:17 GMT  
 help reading fortran generated binary data...

Quote:
>As previously mentioned, the FORTRAN "reader" program currently populates
>the entire COMMON block in question (which is a mix of data types) by
>reading this particular file section in as an array of INTEGERS.
>Since I know that I can interpret INTEGERS correctly from the file, would
>it be possible for me to do the same?  That is, read the appropriate number
>of INTEGER values from the file (this I can do), and then "construct" the
>proper datatypes from those?

I doubt that this will work, that is, not without understanding the format
of the data. I suspect that what Fortran is doing is really writing
and re-reading raw binary data. Calling it "integer" data really just
means "write it out in chunks of 4 bytes without messing with it"
and "read it in in chunks of 4 bytes without messing with it".
So if you read the this data into Tcl as integers, your integers
aren't going to be meaningful as such; they're just going to contain
bit fields in the format used by Fortran. There's no guarantee that
a Tcl integer will even correspond to a meaningful unit.

For example, suppose that the Fortran doubles occupy 64 bits, with
24 bits used for the mantissa and 40 bits for the exponent. (I'm just
making this up for the sake of a simple example. As far as I know
this isn't a real floating point format.) Suppose that a Fortran
INTEGER is 32 bits. So Fortran writes out one double as a pair of
two INTEGERS, each 4 bytes long.

Now you read these into Tcl as two 32-bit integers.  Even assuming that
there are no differences in byte-ordering to worry about, neither of
your Tcl integers can be interpreted by itself. One of them consists of
the mantissa plus 8 bits of the exponent; the other consists of the
remainng 32 bits of the exponent. Of course, if you know what the
Fortran double format is you can compute the real value by shifting
and masking operations.

If I had to deal with this I think I would just write out some known
values, print them out in binary, and figure out what the floating
point format is.

Bill

--
Bill Poser, Linguistics, University of Pennsylvania



Mon, 04 Feb 2008 05:32:19 GMT  
 help reading fortran generated binary data...
I overlooked this thread or it started after I left for home :)
One thing:
- have you thought of byte-ordering? This is important when the file is
created
  on a different machine than where you are reading it. Also if the
machines
  in question do not both use the same floating-point representation
you can
  get into trouble.

Also:
- the data that are actually written to file (or read as exempified by
the FORTRAN
  fragment) consist of different length - integer of two bytes and a
set of characters
  Are you sure you have computed the offset correctly?
- FORTRAN always uses the floating-point types that are available, it
does not
  try to enforce any peculiar type by itself, at least not when you
stay within the
  standard. So, double precision reals ought to be simply 8 bytes.

Just a few thoughts early in the morning ...

Regards,

Arjen



Mon, 04 Feb 2008 14:32:24 GMT  
 help reading fortran generated binary data...

Quote:

> I don't think I mentioned it before, the FORTRAN program that created the
> file originally, the FORTRAN program that can read the file, and my tcl
> program are all running under WinXP Pro on an Intel platform - so I assume
> byte order is the same in all cases - unless FORTRAN stores doubles
> "oddly"???

You write "FORTRAN stores doubles" as if there would be a standard
that all Fortran implementations would follow.  There isn't.  Without
a hex dump of sample data (together with what you expect it to be,
if known) or at least information about what Fortran compiler you're
using on what platform, we're in the dark.  Binary files in Fortran
are notoriously non-portable, and Tcl 8.4's [binary] command is
also somewhat lacking in capability where it comes to floating-point
data (8.5 has additional format groups for floating-point with a
specific byte ordering).  There are workarounds for the latter
problem.

In short, you've not given us enough information to attack the
problem.  A hexdump of even *one* of the doubles you're reading
might be enough; replacing


with


     puts [format %016lx $dataAsWide]

would tell us a lot.

--
73 de ke9tv/2, Kevin



Mon, 04 Feb 2008 21:24:37 GMT  
 help reading fortran generated binary data...
Could you then include information about the FORTRAN compiler too?
While _unformatted_ FORTRAN files are notoriously non-portable, there
seems to be a certain convergence towards a de facto standard. The
reason they are non-portable is that they include information about the
record structure. But most not-too-old compilers use one of a small
number of schemes for this and the most popular scheme is very simple.

In other words: all hope is not lost yet :)

Regards,

Arjen



Mon, 04 Feb 2008 22:06:22 GMT  
 help reading fortran generated binary data...


Quote:
> You write "FORTRAN stores doubles" as if there would be a standard
> that all Fortran implementations would follow.  There isn't.

Interesting - I'm not overly familiar with the details of data storage -
obviously... ;^)

Quote:
>Without
> a hex dump of sample data (together with what you expect it to be,
> if known) or at least information about what Fortran compiler you're
> using on what platform, we're in the dark.

The compiler is "Microsoft Fortran Powerstation, Version 4.00".  Regarding
a HEX dump of data, I'll see if I can provide that soon...

Quote:
> In short, you've not given us enough information to attack the
> problem.  A hexdump of even *one* of the doubles you're reading
> might be enough; replacing


> with


>     puts [format %016lx $dataAsWide]

As I'm sure you know, my intention was not to leave anyone in the dark...
Unfortunately, I'm far enough out of my element that I don't really know
what *is* important to know in order to solve the problem.  Now that you
have given me some ideas, I'll see what I can do to provide more useful
information.

Thanks for the input Kevin.

Jeff



Mon, 04 Feb 2008 23:48:12 GMT  
 help reading fortran generated binary data...


Quote:
> Could you then include information about the FORTRAN compiler too?
> While _unformatted_ FORTRAN files are notoriously non-portable, there
> seems to be a certain convergence towards a de facto standard. The
> reason they are non-portable is that they include information about the
> record structure. But most not-too-old compilers use one of a small
> number of schemes for this and the most popular scheme is very simple.

Arjen,

The compiler being used is "Microsoft Fortran PowerStation, Version 4.00".

The format of the file is described here:

http://h18009.www1.hp.com/fortran/docs/vf-html/pg/pggfmsp.htm

In the "Unformatted Sequential Files" section....

I used the info from the above page to decode the basic file storage format
and create an appropriate tcl-based reader (that allows for the "length
bytes" and 128 byte record lengths)...

Anyway, my tcl reader has been in use for almost 2 years now, so I am
confident in it's general abilities to properly interpret the file's
contents.  Because of that, I'm quite confident that I am reading the
appropriate info from the file with regard to my DOUBLE problem, but I'm
obviously not interpreting what I read properly (yet)...

Thanks for any additional input.

Jeff



Mon, 04 Feb 2008 23:55:58 GMT  
 help reading fortran generated binary data...
Quote:

> Hi All,

> First, for those not interested in the "long" story (below), here's my
> basic problem...

> I've read a value from binary file using the following code:


> point value)

> The value *should* be any of 0, 90, 180, or 270.

> What I get instead is 4 variations of an *extreme* (large or small) number,
> such as:

> 5.4388530464437965e+185

Hi Jeff,

could it be possible that your index ins one off
following are the [binary scan h*] dumps of the value you gave as an example
and the 4 values you are expecting:
If we assume that you are one off your example is missing the last byte
and would stand for 180

4500000000000866 : 5.4388530464437965e+185
0000000000000000 : 0.0
0000000000086504 : 90.0
0000000000086604 : 180.0
00000000000e0704 : 270.0

uwe



Tue, 05 Feb 2008 01:16:37 GMT  
 
 [ 25 post ]  Go to page: [1] [2]

 Relevant Pages 

1. Read binary data generated by Scientific Atlanta 1590 Pattern Recorder

2. Reading Fortran generated unformatted binary files in tcl

3. Fortran/C code to read LV binary data?

4. Using DEC Fortran to read raw binary data?

5. Problems reading Fortran binary data files

6. Reading a binary data set created by SAS into FORTRAN

7. Read the fortran unformatted binary data.

8. Reading a binary file / writing binary data to a file

9. reading binary data fast / help with optimizing ( again )

10. Reading Arguments from Command Line and reading binary data

11. Help : Reading MS Fortran binary files.....

12. ifort Readin binary data generated by pgi

 

 
Powered by phpBB® Forum Software