TclMM - Tcl shared memory interface 
Author Message
 TclMM - Tcl shared memory interface

I have created an easy-to-use interface to shared memory for Unix
systems, based on top of libmm (which is itself a multiplatform
abstraction).

I borrowed the hash table implementation from Tcl to build the whole
thing to run out of shared memory, so you can do stuff like this:

Quote:
> mm_set foo bar
> mm_set foo
bar
> mm_unset foo

across different processes (as long as they have a common ancestor!).

It's available from:

http://www.*-*-*.com/

In all fairness, though, I should say that socket implementations of a
similar concept are probably more desirable in some ways, in that they
aren't that much slower, can be written in pure Tcl, and are thus
cross-platform.  See my multiplexer
( http://www.*-*-*.com/
implementation of this).

Ciao,
--
David N. Welton
   Consulting: http://www.*-*-*.com/
     Personal: http://www.*-*-*.com/
Free Software: http://www.*-*-*.com/
   Apache Tcl: http://www.*-*-*.com/



Fri, 30 Jul 2004 20:06:38 GMT  
 TclMM - Tcl shared memory interface

Quote:
> I have created an easy-to-use interface to shared memory for Unix
> systems, based on top of libmm (which is itself a multiplatform
> abstraction).

Hmmm, very interesting...

I've done similar thing for the Thread extension (thread shared arrays).

Enables you to share data between threads in an protected fashion.

If I understand you correctly, you've done basically the same thing, but

between distinct processes.

What about joining efforts and bringing out a shared variable interface

which would unite both approaches ?

Cheer's

Zoran



Sat, 31 Jul 2004 17:02:49 GMT  
 TclMM - Tcl shared memory interface

Quote:

> I've done similar thing for the Thread extension (thread shared
> arrays).  Enables you to share data between threads in an protected
> fashion.  If I understand you correctly, you've done basically the
> same thing, but between distinct processes.

Yes, using Unix's shared memory facilities.

Quote:
> What about joining efforts and bringing out a shared variable
> interface which would unite both approaches ?

Hrm, good idea, but there are a few limitations.  First, my code is
definitely not pure Tcl.  Far from it - it's very unix oriented.

Maybe the best thing to look at would be a common API that could then
be implemented by other projects (tequila and multiplexer come to
mind).  Something like

        share_init backend ?implementation_specific_list?

    Inits shared system.  Pass it the name of the backend to use, and
    a list which contains implementation specific start up information.

        share_set variable ?value?

    Set variable to value, or if value is not present, return value of
    variable.

        share_exists variable

    Does variable exist?

        share_exit

    Cleanup.

How do you do what you do?  For TclMM I just use mm_set and mm_unset,
which basically hash values into and out of shared memory.  This is
nice and simple, and high-level.

--
David N. Welton
   Consulting: http://www.dedasys.com/
     Personal: http://www.dedasys.com/davidw/
Free Software: http://www.dedasys.com/freesoftware/
   Apache Tcl: http://tcl.apache.org/



Sat, 31 Jul 2004 19:25:16 GMT  
 TclMM - Tcl shared memory interface

Quote:

>         share_init backend ?implementation_specific_list?
>         share_set variable ?value?
>         share_exists variable
>         share_exit

YM:
   share::init ...
   share::set ...
   share::exists ...
   share::finalize

Donal.
--

-- Always running as a superuser is not a fault, it's an OS preference.



Sat, 31 Jul 2004 19:41:54 GMT  
 TclMM - Tcl shared memory interface
Hi David,

I basically went to same way in 1999 when create TStore (sure
with Tcl binding: tclStore). Since there was no libMM available
I did on my own (ie. own allocator). And my main intend was
however to use mmapp'ed file in order to provide a very
fast persistent store/ a main memory database.
Actually it is only slightly slower than a normal Tcl array access.
In that way a very fast  (s|n)dbm/Berkeley db replacement.
And also the idea was to use it in conjunction with my tDOM
as very high performance XML/DOM store.

Here is the usage from the C code:

static char tstore_usage[] =
                "Usage tstore <subCommand> <args>, where subCommand can
be:\n"
                "             create <databaseFileName> <size>
\n"
                "             open <databaseFileName>
\n"
                ;

static char tstoreobj_usage[] =
                "Usage tstoreObject <method> <args>, where method can
be:\n"
                "             close ?-async?
\n"
                "             sync ?-async?
\n"
                "             set <varName> ?<value>?
\n"
                "             info
\n"
                ;

An example:

   set t [tstore create mytables.db 100000000]
   $t set $key $data
   puts stderr [$t set $key]

Re-using my tstore code I created several months ago 'shmarray',
which focus is just to provide an (almost)read-only and fast
data store using shared memory only (no mmap files). It is basically
the same as tstore, but also allows to link standard Tcl array
to the shared arrays:

static char shmarr_usage[] =
                "Usage: shmarray <subCommand> <args...>     \n"
                "   where subCommand can be:                \n"
                "        setsize <size>                     \n"
                "        init   <fileRef> <size>            \n"
                "        attach <fileRef>                   \n"
                "        lock                               \n"
                "        unlock                             \n"
                "        link   <arrName> <tclArrayName>    \n"
                "        set    <arrName> <index> ?<value>? \n"
                "        unset  <arrName> <index>           \n"
                "        exists <arrName> <index>           \n"
                "        names  <arrName> ?<filterPattern>? \n"
                "        vars                               \n"    
Example:

   load shmarray.sl
   shmarray init common.db 10000000
   shmarray lock
   shmarray link BacklogSalesGroup BacklogSalesGroup

   set BacklogSalesGroup(foo) bar
   set BacklogSalesGroup(tcltk) easy

   shmarray unlock

Actually this is used on a 8 processor N-Class server (HP/UX)
with 40 worker processes in a large world-wide 'datawarehouse'
like backend application.

I'm still very interested in tstore/shmarray even working
on it since 1999 (besides tDOM). The ultimate goal would be
to create a simple, small, fast, easy to use and _robust_
storage system featuring the 'column-wise approach'
(similar to Metakit), and that in plain C.

Regards,
Jochen.



Quote:

> I have created an easy-to-use interface to shared memory for Unix
> systems, based on top of libmm (which is itself a multiplatform
> abstraction).

> I borrowed the hash table implementation from Tcl to build the whole
> thing to run out of shared memory, so you can do stuff like this:

> > mm_set foo bar
> > mm_set foo
> bar
> > mm_unset foo

> across different processes (as long as they have a common ancestor!).

> It's available from:

> http://www.dedasys.com/freesoftware/files/TclMM.tgz

> In all fairness, though, I should say that socket implementations of a
> similar concept are probably more desirable in some ways, in that they
> aren't that much slower, can be written in pure Tcl, and are thus
> cross-platform.  See my multiplexer
> (http://www.dedasys.com/freesoftware/files/multiplexer.tgz for an
> implementation of this).

> Ciao,

--
Posted via Mailgate.ORG Server - http://www.Mailgate.ORG


Sat, 31 Jul 2004 21:16:46 GMT  
 TclMM - Tcl shared memory interface

Quote:

> Hrm, good idea, but there are a few limitations.  First, my code is
> definitely not pure Tcl.  Far from it - it's very unix oriented.

Huh. Mine is all C. Apart from that, it works on Win as well.
Maybe even on MacX if I get some spare time (difficult task!).

Quote:
> Maybe the best thing to look at would be a common API that could then
> be implemented by other projects (tequila and multiplexer come to
> mind).  Something like

>         share_init backend ?implementation_specific_list?

>     Inits shared system.  Pass it the name of the backend to use, and
>     a list which contains implementation specific start up information.

This sounds very good, indeed. This way we could have a nice Tcl
wrapper
arround low-level implementation. The only thing is to get more/less
common functionality (ideal would be same functionality) in all
low-level
implementations.

Quote:
> How do you do what you do?  For TclMM I just use mm_set and mm_unset,
> which basically hash values into and out of shared memory.  This is
> nice and simple, and high-level.

I have a pretty complex setup of (fixed) number of buckets, each
protected
with a mutex, each having one or more hash-tables which store
key/value pairs. Data is stored in Tcl_Obj's instead of strings so I
had to write my own DupInternalRep calls for some object types which
are
known to produce shallow object copies (listType, most notably).
All this complication is because one should not share Tcl_Obj's among
threads and I wanted to keep the object internal rep alive.
Basic bucket-idea is borrowed from AOLserver nsv_* commands, but I've
rewritten it altogether. You might want to checkout latest thread
extension status (the HEAD, not 2.2) and look into tsv:* commands.

I'll have a look on what you do in the meantime.

Cheer's
Zoran



Sat, 31 Jul 2004 22:33:04 GMT  
 TclMM - Tcl shared memory interface
Between the three of you, I see a lot of excellent stuff which would
not take too much to package into something *really* cool.

Why don't you leverage on the momentum created, create a SF prg
(or a module next to thread) and spend a few hours at it.

We (Tcl community) would really appreciate ;-)

Thank you in advance,
-Acacio

Quote:

> Hi David,

...


Sun, 01 Aug 2004 09:44:36 GMT  
 TclMM - Tcl shared memory interface

Quote:

> Between the three of you, I see a lot of excellent stuff which would
> not take too much to package into something *really* cool.

> Why don't you leverage on the momentum created, create a SF prg
> (or a module next to thread) and spend a few hours at it.

> We (Tcl community) would really appreciate ;-)

I agree.

Threading package already includes thread-shared
variables. David and Jochen have solutions for
sharing stuff across distinct processes.
What would be needed is a generic Tcl wrapper,
as David already proposed, which might unify
capabilities of all three approaches.

Provided the sources at some accessible
place (SF?) I could spend some time in
evaluating how to bring all of this together.

Jochen, David, what do you mean?

Zoran



Sun, 01 Aug 2004 20:38:56 GMT  
 TclMM - Tcl shared memory interface

Quote:

> I have created an easy-to-use interface to shared memory for Unix
> systems, based on top of libmm (which is itself a multiplatform
> abstraction).

> I borrowed the hash table implementation from Tcl to build the whole
> thing to run out of shared memory, so you can do stuff like this:
> across different processes (as long as they have a common ancestor!).

It means that shared memory segment should be created by tcl parent
process, doesn't it ?

I have the system (under Unix) where several proccesses (_not
multi-threaded_) use shared memory segments to exchage data. Shared
memory segments are created by low-level C-functions shmget, shmat. I
would like to attach to the same shared segments from inside tcl/tk
scripts. I've written a small-and-dirty tcl extension for shm* family
functions but was not satisfied by its quality :) , may be because of
low level of abstraction used.

Is it possible to add to TclMM extension the possibility to attach to
already existing shared memory segment with known id and size ? It will
provide a great mechanism for writing, for example, Tk-based frontends
for data aquisition systems, while backend could be written in C.

Alexandr Bravo



Mon, 02 Aug 2004 18:10:37 GMT  
 TclMM - Tcl shared memory interface


                        .
                        .
                        .
Quote:
>Is it possible to add to TclMM extension the possibility to attach to
>already existing shared memory segment with known id and size ? It will
>provide a great mechanism for writing, for example, Tk-based frontends
>for data aquisition systems, while backend could be written in C.

                        .
                        .
                        .
I agree.  That's the need I've always had for shared memory.
--


Business:  http://www.Phaseit.net
Personal:  http://starbase.neosoft.com/~claird/home.html



Mon, 02 Aug 2004 21:59:59 GMT  
 TclMM - Tcl shared memory interface

Quote:



> >Is it possible to add to TclMM extension the possibility to attach
> >to already existing shared memory segment with known id and size ?
> >It will provide a great mechanism for writing, for example,
> >Tk-based frontends for data aquisition systems, while backend could
> >be written in C.
> I agree.  That's the need I've always had for shared memory.

How about the tclshmem.c system, available from:

http://www.dedasys.com/freesoftware/files/

It's more of a low-level system.  It's been a while since I touched
the code, and it doesn't have documentation or a test suite.  I might
be convinced to remedy that if people actually find the code useful.

--
David N. Welton
   Consulting: http://www.dedasys.com/
     Personal: http://www.dedasys.com/davidw/
Free Software: http://www.dedasys.com/freesoftware/
   Apache Tcl: http://tcl.apache.org/



Tue, 03 Aug 2004 00:48:17 GMT  
 TclMM - Tcl shared memory interface

Quote:

> > I have created an easy-to-use interface to shared memory for Unix
> > systems, based on top of libmm (which is itself a multiplatform
> > abstraction).  I borrowed the hash table implementation from Tcl
> > to build the whole thing to run out of shared memory, so you can
> > do stuff like this: across different processes (as long as they
> > have a common ancestor!).
> It means that shared memory segment should be created by tcl parent
> process, doesn't it ?

Yes, exactly.  Plus, it's based on libmm, which doesn't even
necessarily use the SySV shared mem system.

Quote:
> I have the system (under Unix) where several proccesses (_not
> multi-threaded_) use shared memory segments to exchage data. Shared
> memory segments are created by low-level C-functions shmget,
> shmat. I would like to attach to the same shared segments from
> inside tcl/tk scripts. I've written a small-and-dirty tcl extension
> for shm* family functions but was not satisfied by its quality :) ,
> may be because of low level of abstraction used.

I wrote something similar, but if you are looking at the low level,
well - then it's low level.  Not much to do about that...  Anything
you build on top of it has to know about what it's looking at.

Quote:
> Is it possible to add to TclMM extension the possibility to attach
> to already existing shared memory segment with known id and size ?
> It will provide a great mechanism for writing, for example, Tk-based
> frontends for data aquisition systems, while backend could be
> written in C.

I don't think it would be good at all for comunicating with C, because
of all the abstractions.  What's more, internally, it uses Tcl's hash
table implementation, which would make it even harder to comunicate
with bits of binary data written by C.

--
David N. Welton
   Consulting: http://www.dedasys.com/
     Personal: http://www.dedasys.com/davidw/
Free Software: http://www.dedasys.com/freesoftware/
   Apache Tcl: http://tcl.apache.org/



Tue, 03 Aug 2004 00:51:22 GMT  
 TclMM - Tcl shared memory interface

Quote:

> I have a pretty complex setup of (fixed) number of buckets, each
> protected with a mutex, each having one or more hash-tables which
> store key/value pairs. Data is stored in Tcl_Obj's instead of
> strings so I had to write my own DupInternalRep calls for some
> object types which are known to produce shallow object copies
> (listType, most notably).  All this complication is because one
> should not share Tcl_Obj's among threads and I wanted to keep the
> object internal rep alive.  Basic bucket-idea is borrowed from
> AOLserver nsv_* commands, but I've rewritten it altogether.

Sounds quite complex.  How do you fit the DupInternalRep calls in? I
mean, how do you supplant the Tcl supplied ones?

Quote:
> You might want to checkout latest thread extension status (the HEAD,
> not 2.2) and look into tsv:* commands.

Where at?

Quote:
> I'll have a look on what you do in the meantime.

Mine's quite simple, really.  Small, simple, and easy to use.

--
David N. Welton
   Consulting: http://www.dedasys.com/
     Personal: http://www.dedasys.com/davidw/
Free Software: http://www.dedasys.com/freesoftware/
   Apache Tcl: http://tcl.apache.org/



Tue, 03 Aug 2004 01:10:51 GMT  
 
 [ 16 post ]  Go to page: [1] [2]

 Relevant Pages 

1. Tcl and Shared memory Question.

2. Is it possible to acces shared memory in TCL

3. shared memory in Tcl/tk

4. Help: Shared Memory in Tcl

5. (q) Has anybody access shared memory from an Tcl/Tk application

6. Anyone add SYSV shared memory extensions to tcl?

7. ANNOUNCE: fbsql (MySQL interface for Tcl) - first patch (fixes memory exception)

8. SQUEAK: Can it interface to shared libraries?

9. Java Native Interface with Fortran shared libraries?

10. Proposal for sharing C interfaces between extension modules

11. C-level interface to Shared Libraries

12. Shared Memory, Data Compression Documentation Questions

 

 
Powered by phpBB® Forum Software