struct module + array module = ? ( reading C structs ) 
Author Message
 struct module + array module = ? ( reading C structs )

As I'm hacking Yet Another python Module than reads in a C-struct
from disk and translates it to a Python object, I have to ask
if there is a better way ?

To be more specific: The struct module translates c structs into
a tuple of values - it handles repetition counts, but it doesn't
handle arrays.

The array class has a fromstring method to read in binary objects.

But if you have a struct that contains c-strings and other type
arrays, you either have to use struct to generate a large tuple,
part of which you need to pack back into a sequence of some type,
or you have to alternate between struct and array conversion.

I do either, depending on the struct and which looks easier.

However, as I find myself writing something (so far) exactly like:

| hfmt1 = 'hhhxx'
| hfmt1_size = struct.calcsize( hfmt1 )
| FLOATSIZE = struct.calcsize( 'f' )
| SHORTSIZE = struct.calcsize( 'h' )
| MAXREFS = 20
|
| def xlate( hdr ):
|        nrefs, nconc, iub = struct.unpack( hfmt1, hdr[:hfmt1_size] )
|        hdr = hdr[hfmt1_size:]
|        elems = []
|        index = []
|        for i in range( nrefs ):
|                estart =  i * 4
|                istart =  ( MAXREFS * 4 ) + ( i * 6 )
|                elems.append( array( 'c' ).fromstring( hdr[estart:estart+4] )
|                index.append( array( 'h' ).fromstring( hdr[istart:istart+6] )
|        i = ( MAXREFS * 4 ) + ( MAXREFS * 6 )
|        weights = array( 'f', hdr[i:i+(nconc*FLOATSIZE)] )
|        i = i + ( MAXREFS * FLOATSIZE  )
|        rasums = array( 'f', hdr[i:i+(nrefs*FLOATSIZE)] )
|        i = i + ( MAXREFS * FLOATSIZE )
|        rbsums = array( 'f', hdr[i:i+(nrefs*FLOATSIZE)] )

        [ ... and on to the short integer arrays ... ]

I have to think that either there is a better trick that I'm missing,
or we need to invent one.

Extending struct module to handle arrays would be one idea - but
we'ld have to figure out a syntax for format strings ( that wouldn't
conflict with other format string conventions - '6c' is six characters,
NOT a 6 character string. )

I guess a more immediate fix would be a higher level pre-parser routine
that took in a C-struct definition and ( using struct and array ) returned
the desired Python object.

Anybody do anything like this already, or have any tricks up there
sleeve ?

[ Sorry to lapse back to such practical matters, but I have to make
 my boss happy and actually work on some practical application Python
 code for a while, instead of playing with *extending* Python. So -
 here I go - thinking up new extensions! Well - I tried! ;-)    ]


-- UVA Department of Molecular Physiology and Biological Physics --
-- Box 449 Health Science Center        C{*filter*}tesville,VA 22908 --



Sat, 03 May 1997 03:10:43 GMT  
 struct module + array module = ? ( reading C structs )

I started off trying to make a smarter wrapper around struct that
took an extension for arrays in the format string (  'hh[32c]' for
example ) but before I got very far, I decided, that since I wanted
eventually to turn the fields into class attributes, to mimic the
c-structs, it would be better and simpler to table-drive the conversion.

I'm in too much of a hurry to finish this particular problem to
overgeneralize the solution right now, but it would seem like a good
project to use Aaron's parser to parse C structs from .h files and
automatically construct a class definition using the table.

All I need to initially fill the table are field names and struct
module format strings. struct.calcsize produces the sizes and offsets.
Of course, the generalized solution would have to handle structs and
typedef's recursively. And that recursive processing could probably
be made to handle what is special cased in my code: handling the
two-dimensional arrays and character strings.

 In this particular example, I was really taking the defs from a fortran
include file, but I need to read C include files more often. 'elems' is
declared  "character*4 elems(20)" and index is "Integer*2 indx(3,20)"
They are the two fields that require special post-processing.

So now I have something like the following:

_table = (      [ 'elems',      '80c' ],
                [ 'index',      '60h' ],
                [ 'weight',     '20f' ],
                [ 'rasums',     '20f' ],
                [ 'rbsums',     '20f' ],
                [ 'iotoph',     '4h'  ],
                [ 'ifrang',     '2h'  ],
                [ 'iarang',     '2h'  ],
                [ 'ibrang',     '2h'  ],
                [ 'irlist',     '6h'  ] )

# _table element is initially: [ struct-name, format-string ]

# first three fields + pad word are simple scalars, so we
# didn't bother to put them in the table.

_initial = 'hhhxx'
_initskip = struct.calcsize( _initial )

i = _initskip

for x in _table:
        x.append( i )
        x.append( struct.calcsize( x[1] ) )
        i = i + x[-1]
# _table element is now: [struct-name, format-string, offset, size ]

class MlRefhdr:
        def __init__( self, *hdr ):
                if hdr : self.xlate( hdr[0] )
        def xlate( self, hdr ):
                self.nrefs, self.nconcs, self.iub = struct.unpack( _initial, hdr[:_initskip] )
                nrefs = self.nrefs
                for entry in _table:
                        setattr( self, entry[0], struct.unpack( entry[1], hdr[entry[2]:entry[2]+entry[3]] ))
                elems = []
                index = []
                for i in range( nrefs ):
                   # join elems into a string...
                   elems.append( string.joinfields(self.elems[i*4:(i+1)*4],'') )
                   # and split index into a 2-d "array"
                   index.append( self.index[i*3:(i+1)*3] )
                self.elems = elems
                self.index = index
                self.weight = self.weight[:nrefs] # truncate these
                self.rasums = self.rasums[:nrefs] # to actual number
                self.rbsums = self.rbsums[:nrefs]


-- UVA Department of Molecular Physiology and Biological Physics --
-- Box 449 Health Science Center        C{*filter*}tesville,VA 22908 --



Sat, 03 May 1997 08:18:55 GMT  
 struct module + array module = ? ( reading C structs )

Ancient discussion, I know.

I've considered doing a /proc module in C instead of python, for this
very reason.  Setting up the struct-module stuff to translate C
structs to python is facilitated by the struct-module, but it's still
a pain, if you're dealing with a struct of much size, or doing so on a
regular basis.

It seems to me, that a gcc option might be pretty helpful - tho the
GNU folks Might think it a {*filter*} hack that doesn't belong in gcc.  The
notion would be to add a "-fdump-struct structname" to gcc, that would
parse things up, emit no code (per se), and output the sizes and
paddings of the structure "structname" in some
rediculously-easy-to-translate format - not necessarily the python
(struct module) format - we'd translate to that, from there.

Yeah, we could do a parser for C and cpp, but...


 write:

Quote:

>As I'm hacking Yet Another Python Module than reads in a C-struct
>from disk and translates it to a Python object, I have to ask
>if there is a better way ?

>To be more specific: The struct module translates c structs into
>a tuple of values - it handles repetition counts, but it doesn't
>handle arrays.

>The array class has a fromstring method to read in binary objects.

>But if you have a struct that contains c-strings and other type
>arrays, you either have to use struct to generate a large tuple,
>part of which you need to pack back into a sequence of some type,
>or you have to alternate between struct and array conversion.

>I do either, depending on the struct and which looks easier.

>However, as I find myself writing something (so far) exactly like:

>| hfmt1 = 'hhhxx'
>| hfmt1_size = struct.calcsize( hfmt1 )
>| FLOATSIZE = struct.calcsize( 'f' )
>| SHORTSIZE = struct.calcsize( 'h' )
>| MAXREFS = 20
>|
>| def xlate( hdr ):
>|        nrefs, nconc, iub = struct.unpack( hfmt1, hdr[:hfmt1_size] )
>|        hdr = hdr[hfmt1_size:]
>|        elems = []
>|        index = []
>|        for i in range( nrefs ):
>|                estart =  i * 4
>|                istart =  ( MAXREFS * 4 ) + ( i * 6 )
>|                elems.append( array( 'c' ).fromstring( hdr[estart:estart+4] )
>|                index.append( array( 'h' ).fromstring( hdr[istart:istart+6] )
>|        i = ( MAXREFS * 4 ) + ( MAXREFS * 6 )
>|        weights = array( 'f', hdr[i:i+(nconc*FLOATSIZE)] )
>|        i = i + ( MAXREFS * FLOATSIZE  )
>|        rasums = array( 'f', hdr[i:i+(nrefs*FLOATSIZE)] )
>|        i = i + ( MAXREFS * FLOATSIZE )
>|        rbsums = array( 'f', hdr[i:i+(nrefs*FLOATSIZE)] )

>    [ ... and on to the short integer arrays ... ]

>I have to think that either there is a better trick that I'm missing,
>or we need to invent one.

>Extending struct module to handle arrays would be one idea - but
>we'ld have to figure out a syntax for format strings ( that wouldn't
>conflict with other format string conventions - '6c' is six characters,
>NOT a 6 character string. )

>I guess a more immediate fix would be a higher level pre-parser routine
>that took in a C-struct definition and ( using struct and array ) returned
>the desired Python object.

>Anybody do anything like this already, or have any tricks up there
>sleeve ?

>[ Sorry to lapse back to such practical matters, but I have to make
> my boss happy and actually work on some practical application Python
> code for a while, instead of playing with *extending* Python. So -
> here I go - thinking up new extensions! Well - I tried! ;-)    ]


>-- UVA Department of Molecular Physiology and Biological Physics --
>-- Box 449 Health Science Center        C{*filter*}tesville,VA 22908 --




Thu, 19 Jun 1997 01:27:44 GMT  
 
 [ 3 post ] 

 Relevant Pages 

1. sorry for this stupid question about struct module

2. wish: "struct" module extension proposal

3. packing (struct module)

4. Improved struct module

5. A case for the buffer interface (was: Improved struct module)

6. Bug in struct module?

7. Struct module

8. struct module

9. revamped struct module

10. struct module endian-ness idea

11. struct module?

12. Problems with the struct module on the alpha

 

 
Powered by phpBB® Forum Software