cross-version voodoo: handling 8.3 / 8.4 private structure changes 
Author Message
 cross-version voodoo: handling 8.3 / 8.4 private structure changes

I've added a page to the wiki to discuss how I handled a special case of
accessing private Tcl structures where the location of elements were
shifted due to a change in the size of Tcl_HashTable in 8.4 (added one
pointer).  In this case it was TclX's use of the private Tcl Namespace
structure (not the public Tcl_Namespace), which includes 3 Tcl_HashTables
directly into it, causing other elements to shift.

For a bit of a write-up, see:

        http://www.*-*-*.com/

The trick above is half-generalized.  It only works for Namespace, but
the proper support is there to switch on the type of structure you need
to adjust for, with you just need to provide size_t pairings.

I've tested this for 8.3->8.4 and 8.4->8.3, and I'm fairly certain of
the 8.3->8.4 shifts, but the 8.4->8.3 shifts may need more boundary
testing.

--
  Jeff Hobbs                     The Tcl Guy
  Senior Developer               http://www.*-*-*.com/



Wed, 29 Dec 2004 07:34:39 GMT  
 cross-version voodoo: handling 8.3 / 8.4 private structure changes

Quote:

> I've added a page to the wiki to discuss how I handled a special case of
> accessing private Tcl structures where the location of elements were
> shifted due to a change in the size of Tcl_HashTable in 8.4 (added one
> pointer).  In this case it was TclX's use of the private Tcl Namespace
> structure (not the public Tcl_Namespace), which includes 3 Tcl_HashTables
> directly into it, causing other elements to shift.

Is there binary compatibility with 8.4?  In previous releases, I know
you put lots of effort to not reorder structure fields (even on
private structures) and to leave public structures alone (replicating
them if necessary).  Unfortunately, the Tcl_HashTable is a public
structure.  Many extensions use it in their own code.

The bad part is that I'm not sure that breaking compatibility is
necessary.  With a little bit of recoding, the structures could stay
the same size.  For example, storing the hash value makes the
bucketPtr redundant.  You can always calculate it from the hash value.

   bucketPtr = tablePtr->buckets + (entryPtr->hash & tablePtr->numBuckets);

Is there any code anywhere using the new custom key features?  I've
argued this before--I don't really think it serves any real purpose.
Given how little code implements the hash table and the cost of
changing it, would it be worth it to spin-off the new features in its
own Tcl_CustomHashTable API?

I'm not a big believer in binary compatibility.  It was bound to break
sooner or later and only has became a constraint on what types of
improvements can be implemented.  It's not a problem either way for
me.  I'll still create BLT binaries for each Tcl release.  But
shouldn't there be something in the release notes saying that 8.4 is
not binary compatible with previous 8.x releases?  Or does noone
expect compatibility any more?

--gah



Thu, 30 Dec 2004 08:04:14 GMT  
 cross-version voodoo: handling 8.3 / 8.4 private structure changes

Quote:


> > I've added a page to the wiki to discuss how I handled a special case of
> > accessing private Tcl structures where the location of elements were
> > shifted due to a change in the size of Tcl_HashTable in 8.4 (added one
> > pointer).  In this case it was TclX's use of the private Tcl Namespace
> > structure (not the public Tcl_Namespace), which includes 3 Tcl_HashTables
> > directly into it, causing other elements to shift.

> Is there binary compatibility with 8.4?  In previous releases, I know

The updated hash tables were done in such a way as to carefully ensure
binary compatability with previous versions.  For example, extensions
like tktable that make extensive use of hash tables and is stubified,
and it works just fine without any of the above tricks.  In fact, it
can probably be compile with 8.3 or 8.4 and be used across the boundary
because it stores pointers to hash tables and then allocates them,
instead of inlining the hash tables.

However, it would be fine also to inline the hash tables.  There is
extra care taken in the new 8.4 hash table code to not use the extra
pointer unless it knows that the calling code was compiled with 8.4.

The problem comes with Tclx because it is making internal references to
hash tables in private structures in Tcl, and these have been shifted
in 8.4 due to the change in Tcl_HashTable size.  It is really only a
problem when poking into other extensions' (or core's) structures which
contain inlined Tcl_HashTables, where you have one compiled in 8.3 and
the other 8.4.

Quote:
> The bad part is that I'm not sure that breaking compatibility is
> necessary.  With a little bit of recoding, the structures could stay
> the same size.  For example, storing the hash value makes the
> bucketPtr redundant.  You can always calculate it from the hash value.

>  bucketPtr = tablePtr->buckets + (entryPtr->hash & tablePtr->numBuckets);

That may be the case, that the bucketPtr could have been reused for the
typePtr (the new field) with careful coding (like the above where this
field isn't used in 8.4).  I'm not sure if that option was analysed.

Quote:
> Is there any code anywhere using the new custom key features?  I've

Not that I've done ... this feature was adapted from Feather, IIRC.

Quote:
> changing it, would it be worth it to spin-off the new features in its
> own Tcl_CustomHashTable API?

I think the problems caused, like the above, are very limited, and the
added functionality to hash tables useful for those willing to work in
8.4+.

Quote:
> shouldn't there be something in the release notes saying that 8.4 is
> not binary compatible with previous 8.x releases?  Or does noone

In this case, I don't think that is necessary, just due to the limited
effect it has.  I don't know of a case where it actually affects people
who stick with the public APIs, which is what binary compatability is
promised for.

--
  Jeff Hobbs                     The Tcl Guy
  Senior Developer               http://www.ActiveState.com/
      Tcl Support and Productivity Solutions



Fri, 31 Dec 2004 00:45:55 GMT  
 cross-version voodoo: handling 8.3 / 8.4 private structure changes

Quote:

> The updated hash tables were done in such a way as to carefully ensure
> binary compatability with previous versions.  For example, extensions
> like tktable that make extensive use of hash tables and is stubified,
> and it works just fine without any of the above tricks.  In fact, it
> can probably be compile with 8.3 or 8.4 and be used across the boundary
> because it stores pointers to hash tables and then allocates them,
> instead of inlining the hash tables.

If you say there's only forward compatibility--compile with 8.4 and it
works with version 8.4 or greater--I'll buy that.

Here's a case where the new Tcl_HashTable breaks code.

        typedef struct {
            Display *display;  
            Tk_Window tkwin;

            Tcl_HashTable thingTable; /* Formerly 10 words, now 9 words. */

            int relief;

        }  MyWidget;

        MyWidget *widgPtr;

The size of the hash table structure has changed and so have the
offsets of every field below it in the MyWidget structure. [The
Tcl_HashEntry structure grows but it's dynamically allocated].  The
location of "relief" changes depending upon which version I compile
with.  I can compile this code under 8.3 and run it with any 8.x wish
including 8.4.  If I compile it under 8.4, it only runs under 8.4.  I
don't know why 8.4 doesn't pad the structure by another word.

But let's say you're more interested in compatibility than features or
that you're an expert hash table user with your own "find" and
"create" routines. So you compile with

        -DTCL_PRESERVE_BINARY_COMPATIBILITY

Now the size of the Tcl_HashTable is 11 words!  No extension compiled
with 8.0 - 8.3 will work with 8.4.

Quote:
>> Is there any code anywhere using the new custom key features?  I've
> Not that I've done ... this feature was adapted from Feather, IIRC.

If there's no code anywhere that benefits from it, why not then revert
back to the compatible version of the hash table?  What problem is it
solving?

Quote:
>> changing it, would it be worth it to spin-off the new features in its
>> own Tcl_CustomHashTable API?

It's my own view that the new changes don't improve the Tcl hash
table, just make it more complicated.  The real answer is to fix the
suboptimal array hashing function, not to add hooks to work around it.
The new custom key features could then have easily been added by an
API layer on top of the existing hash table.

--gah



Fri, 31 Dec 2004 03:53:43 GMT  
 cross-version voodoo: handling 8.3 / 8.4 private structure changes

Quote:

> The problem comes with Tclx because it is making internal references to
> hash tables in private structures in Tcl, and these have been shifted
> in 8.4 due to the change in Tcl_HashTable size.  It is really only a
> problem when poking into other extensions' (or core's) structures which
> contain inlined Tcl_HashTables, where you have one compiled in 8.3 and
> the other 8.4.

Why not modify Tclx?

Chang



Fri, 31 Dec 2004 12:49:23 GMT  
 cross-version voodoo: handling 8.3 / 8.4 private structure changes

Quote:


>>The problem comes with Tclx because it is making internal references to
>>hash tables in private structures in Tcl, and these have been shifted
>>in 8.4 due to the change in Tcl_HashTable size.  It is really only a
>>problem when poking into other extensions' (or core's) structures which
>>contain inlined Tcl_HashTables, where you have one compiled in 8.3 and
>>the other 8.4.

> Why not modify Tclx?

Jeff did exactly that, if i read his wiki page correct.

Michael Schlenker



Fri, 31 Dec 2004 20:46:21 GMT  
 cross-version voodoo: handling 8.3 / 8.4 private structure changes

Quote:


        ...
> If you say there's only forward compatibility--compile with 8.4 and it
> works with version 8.4 or greater--I'll buy that.

Yes, that is correct.  Also, there is forward compatability with 8.3 to
8.3+ - if you aren't mixing extensions compiled with 8.4+ and then poke
into each others internal structures where Tcl_HashTables are used.

        ... [code with inline hash table removed] ...

Quote:
> with.  I can compile this code under 8.3 and run it with any 8.x wish
> including 8.4.  If I compile it under 8.4, it only runs under 8.4.  I
> don't know why 8.4 doesn't pad the structure by another word.

> But let's say you're more interested in compatibility than features or
> that you're an expert hash table user with your own "find" and
> "create" routines. So you compile with

>         -DTCL_PRESERVE_BINARY_COMPATIBILITY

> Now the size of the Tcl_HashTable is 11 words!  No extension compiled
> with 8.0 - 8.3 will work with 8.4.

Note that TCL_P_B_C is 1 by default, but the final statement is not
correct.  That last word is not accessed by 8.4+ unless the extension
which uses it was compiled with 8.4+.  This way you maintain
compatability.  It can be a bit hard to follow the magic in the code
that ensures this, but trust me - it is there.

Quote:
> It's my own view that the new changes don't improve the Tcl hash
> table, just make it more complicated.  The real answer is to fix the
> suboptimal array hashing function, not to add hooks to work around it.
> The new custom key features could then have easily been added by an
> API layer on top of the existing hash table.

This may be a good way to approach it, but then what was added does
add the functionality with compatability.  I've seen some of your
other recommendations for updating hash tables - all good, but most
lacking in binary compatability.  That's good for 9.0, but not 8.x.

--
  Jeff Hobbs                     The Tcl Guy
  Senior Developer               http://www.ActiveState.com/
      Tcl Support and Productivity Solutions



Sat, 01 Jan 2005 00:15:10 GMT  
 cross-version voodoo: handling 8.3 / 8.4 private structure changes

Quote:

> Note that TCL_P_B_C is 1 by default, but the final statement is not
> correct.  That last word is not accessed by 8.4+ unless the extension
> which uses it was compiled with 8.4+.  This way you maintain
> compatability.  It can be a bit hard to follow the magic in the code
> that ensures this, but trust me - it is there.

What I'm saying is that you can't have an extension that uses a
Tcl_HashTable structure in its public structures.

file Object.h

        typedef struct {
            Tcl_HashTable nameTable;
            int result;
        } Object;

Let's say I have an extension B that creates Objects.  I compile that
with 8.3, it's 56 bytes.  Later on I compile extension C that uses
Object with 8.4.  An Object is now 58 bytes.  Now I load both
extensions.

        load A.so
        load B.so
        set object [make_object]
        use_object $object

When B uses the Object passed to it from A, it will reach past the end
of the structure for the "result" field.

I wish this was a bit more contrived but the Tcl_HashTable is both a
common user-code data structure and it's not normally malloc'ed but
allocated with the rest of the structure.  I build up applications
from lots of smaller dynamic libraries.  Since the size of the
structure changes with 8.4, I have to either 1) make sure I don't use
8.4 to compile any code or 2) recompile everything with 8.4.

Quote:
>> It's my own view that the new changes don't improve the Tcl hash
>> table, just make it more complicated.  The real answer is to fix
>> the suboptimal array hashing function, not to add hooks to work
>> around it.  The new custom key features could then have easily been
>> added by an API layer on top of the existing hash table.
> This may be a good way to approach it, but then what was added does
> add the functionality with compatability.  I've seen some of your
> other recommendations for updating hash tables - all good, but most
> lacking in binary compatability.  That's good for 9.0, but not 8.x.

That's not necessarily true.  TIP#69 was also about fixing the hash
table for 64-bit platforms which will require reordering structures.
The only proposed change was to reduce size of a Tcl_HashEntry by 20%
at the cost of changing the call the Tcl_DeleteHashEntry.  I'll let
others decide if it's worthwhile.  

The difference between 8.4's hash table and TIP#69 is that TIP#69
actually fixes those problems and doesn't leave it for users to
implement new allocation and hash routines.  Again, what problem does
the 8.4 hash table solve?  I don't know.

--gah



Sat, 01 Jan 2005 07:53:13 GMT  
 cross-version voodoo: handling 8.3 / 8.4 private structure changes

Quote:


> > Note that TCL_P_B_C is 1 by default, but the final statement is not
> > correct.  That last word is not accessed by 8.4+ unless the extension
> > which uses it was compiled with 8.4+.  This way you maintain
> What I'm saying is that you can't have an extension that uses a
> Tcl_HashTable structure in its public structures.
        ...
> I wish this was a bit more contrived but the Tcl_HashTable is both a
> common user-code data structure and it's not normally malloc'ed but
> allocated with the rest of the structure.  I build up applications
> from lots of smaller dynamic libraries.  Since the size of the
> structure changes with 8.4, I have to either 1) make sure I don't use
> 8.4 to compile any code or 2) recompile everything with 8.4.

Yes, that is correct.  Building up from multiple libraries will expose
this problem again, when you are crossing the 8.3/8.4 boundary.

--
  Jeff Hobbs                     The Tcl Guy
  Senior Developer               http://www.ActiveState.com/
      Tcl Support and Productivity Solutions



Mon, 03 Jan 2005 23:16:01 GMT  
 cross-version voodoo: handling 8.3 / 8.4 private structure changes

Quote:
> What I'm saying is that you can't have an extension that uses a
> Tcl_HashTable structure in its public structures.

> file Object.h

>    typedef struct {
>        Tcl_HashTable nameTable;
>        int result;
>         } Object;

> Let's say I have an extension B that creates Objects.  I compile that
> with 8.3, it's 56 bytes.  Later on I compile extension C that uses
> Object with 8.4.  An Object is now 58 bytes.  Now I load both
> extensions.

>    load A.so
>    load B.so
>    set object [make_object]
>    use_object $object

> When B uses the Object passed to it from A, it will reach past the end
> of the structure for the "result" field.

Agree. That is a real problem for the relationship among Tcl
extensions.

For example, suppose I use the ORATCL (it used the Tcl_HashTable) in
8.3 version, and MyExtension in 8.4 version, that may cause troubles
somewhere. Then I have to recompile ORATCL with 8.4. So what is the
advantage of stub for ORATCL?

This is the Hash table used in the Oratcl.

typedef struct OratclState {
        Tcl_HashTable   *logHash;       /* login handle hash            */
        int             logid;          /* login handle id              */
        Tcl_HashTable   *stmHash;       /* statement handle hash        */
        int             stmid;          /* statement handle id          */
        Tcl_HashTable   *lobHash;       /* statement handle hash        */
        int             lobid;          /* statement handle id          */

Quote:
} OratclState;

Chang


Tue, 04 Jan 2005 07:08:36 GMT  
 cross-version voodoo: handling 8.3 / 8.4 private structure changes

Quote:

> For example, suppose I use the ORATCL (it used the Tcl_HashTable) in
> 8.3 version, and MyExtension in 8.4 version, that may cause troubles
> somewhere. Then I have to recompile ORATCL with 8.4. So what is the
> advantage of stub for ORATCL?

> This is the Hash table used in the Oratcl.

> typedef struct OratclState {
>         Tcl_HashTable   *logHash;       /* login handle hash            */
>         int             logid;          /* login handle id              */
>         Tcl_HashTable   *stmHash;       /* statement handle hash        */
>         int             stmid;          /* statement handle id          */
>         Tcl_HashTable   *lobHash;       /* statement handle hash        */
>         int             lobid;          /* statement handle id          */
> } OratclState;

This is perfectly fine and will not require recompilation.  If you follow
the earlier posts in the thread, it is the direct include of the
Tcl_HashTable structure into a struct, not a pointer to one, that will
cause the problem.

--
  Jeff Hobbs                     The Tcl Guy
  Senior Developer               http://www.ActiveState.com/
        Tcl Support and Productivity Solutions
 Join us in Sept. for Tcl'2002: http://www.tcl.tk/community/tcl2002/



Tue, 04 Jan 2005 07:24:48 GMT  
 
 [ 11 post ] 

 Relevant Pages 

1. trace incompatibility in 8.4 and 8.3 ?

2. problem about tcl 8.3 and 8.4

3. Q: speed open call Tcl 8.3 vs 8.4

4. glob difference between 8.3 and 8.4

5. Tk 8.3 -> 8.4, surprisingly slow text widget

6. Tcl 8.4 more DOSsy than 8.3 ?

7. Stubs with Tcl 8.3 vs Tcl 8.4

8. CONST Tcl 8.3 vs Tcl 8.4

9. 8.4 vs 8.3

10. Serial comm: RTS handling fot Tcl 8.3

11. Postcommand behavior change on Windows Tk/8.3

12. Looking for Tk 8.3 text widget behavior change explanation

 

 
Powered by phpBB® Forum Software