NODE tree introspection 
Author Message
 NODE tree introspection

I've been considering writing something for Ruby similar to the Perl
B.pm module; this is a module which provides Perl-level access to the
bytecode op tree, so that other modules can walk the op tree in
various interesting ways. This is the way the Perl to C compiler was
implemented. (however unsuccessfully...)

It seems reasonably easy to do in Ruby, just by wrapping the NODE*
structures as Ruby objects. Has anyone already done this or tried
working on it?

--
"The best index to a person's character is a) how he treats people who can't
do him any good and b) how he treats people who can't fight back."
-- Abigail Van Buren



Sun, 10 Apr 2005 23:42:22 GMT  
 NODE tree introspection


Quote:

>I've been considering writing something for Ruby similar to the Perl
>B.pm module; this is a module which provides Perl-level access to the
>bytecode op tree, so that other modules can walk the op tree in
>various interesting ways. This is the way the Perl to C compiler was
>implemented. (however unsuccessfully...)

>It seems reasonably easy to do in Ruby, just by wrapping the NODE*
>structures as Ruby objects. Has anyone already done this or tried
>working on it?

Hmmmm.... interesting idea.  So basically what you're proposing would be
able to walk the AST of the currently running program (or any arbitrary
snippet that you send off to eval?)?   I actually have some use for this
kind of thing so if you do it please keep us informed.

I don't think anyone has approached the problem in the way you propose.  
Can you give some more details of how you're thinking of doing it?  I know
that Bob Calco is working on a Ruby parser based on Matz' Ruby yacc file
and he intends it to be a Ruby extension (on second thought, maybe what
he is doing is similar to what you're thinking of doing).

Phil



Mon, 11 Apr 2005 01:02:50 GMT  
 NODE tree introspection

Quote:

> I've been considering writing something for Ruby similar to the Perl
> B.pm module; this is a module which provides Perl-level access to the
> bytecode op tree, so that other modules can walk the op tree in
> various interesting ways. This is the way the Perl to C compiler was
> implemented. (however unsuccessfully...)

> It seems reasonably easy to do in Ruby, just by wrapping the NODE*
> structures as Ruby objects. Has anyone already done this or tried
> working on it?

http://www.rubygarden.org/ruby?RubyInRuby
http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/8693
http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/6885

I believe somebody attempted to ... uhmmm... do something using rockit.

--
 _           _                            
| |__   __ _| |_ ___ _ __ ___   __ _ _ __  
| '_ \ / _` | __/ __| '_ ` _ \ / _` | '_ \
| |_) | (_| | |_\__ \ | | | | | (_| | | | |
|_.__/ \__,_|\__|___/_| |_| |_|\__,_|_| |_|
        Running Debian GNU/Linux Sid (unstable)
batsman dot geo at yahoo dot com

We are using Linux daily to UP our productivity - so UP yours!
        -- Adapted from Pat Paulsen by Joe Sloan



Mon, 11 Apr 2005 01:21:52 GMT  
 NODE tree introspection

Quote:

> Hmmmm.... interesting idea.  So basically what you're proposing would be
> able to walk the AST of the currently running program (or any arbitrary
> snippet that you send off to eval?)?  

Yes. That's what Perl's B.pm does.

 % perl -MB -le '$x= B::main_start; print $x->name while $x=$x->next'
nextstate
pushmark
gv
entersub
gvsv
sassign
...

(New statement, add a mark to the stack, add a symbol table entry
[B::main_start] to the stack, call it, put a variable on the stack,
assign the result of the call to the variable...)

Quote:
> I don't think anyone has approached the problem in the way you propose.
> Can you give some more details of how you're thinking of doing it?

I can even give you some sample code, because I've already got a fair bit
of it working. :)

It says a lot for Ruby's ease of use, clarity of code and
well-developed extensions system that I've only been *using* the
{*filter*}y language for three days and I've already got most of a bytecode
walker constructed.

Here's the core of it:

static VALUE
bytecode_get_root(VALUE class)
{
    return Data_Wrap_Struct(class, 0, 0, ruby_eval_tree);

Quote:
}

void Init_Bytecode () {
    cByteNode = rb_define_class("ByteNode", rb_cObject);
    rb_define_singleton_method(cByteNode, "root", bytecode_get_root, 0);
    rb_define_method(cByteNode, "type", bytecode_nd_type, 0);
    rb_define_method(cByteNode, "line", bytecode_nd_line, 0);
    rb_define_method(cByteNode, "head", bytecode_nd_head, 0);
    rb_define_method(cByteNode, "next", bytecode_nd_next, 0);
    ...

Quote:
}

The bytecode_nd_* functions just do a Data_Get_Struct, call the relevant
nd_* macro, and wrap the result.

Now I can say:

% ruby -e 'require "Bytecode"; p ByteNode.root; p ByteNode.root.next.type'
#<ByteNode:0x401a6550>
4

Is this making more sense now?

Of course, it'll make even more sense when I write something which converts
type back into a string rather than an enum, etc. :)

--
Darn it, who spiked my coffee with water?!



Mon, 11 Apr 2005 01:44:37 GMT  
 NODE tree introspection


Quote:

>> Hmmmm.... interesting idea.  So basically what you're proposing would be
>> able to walk the AST of the currently running program (or any arbitrary
>> snippet that you send off to eval?)?  

>Yes. That's what Perl's B.pm does.

> % perl -MB -le '$x= B::main_start; print $x->name while $x=$x->next'
>nextstate
>pushmark
>gv
>entersub
>gvsv
>sassign
>...

>(New statement, add a mark to the stack, add a symbol table entry
>[B::main_start] to the stack, call it, put a variable on the stack,
>assign the result of the call to the variable...)

>> I don't think anyone has approached the problem in the way you propose.
>> Can you give some more details of how you're thinking of doing it?

>I can even give you some sample code, because I've already got a fair bit
>of it working. :)

>It says a lot for Ruby's ease of use, clarity of code and
>well-developed extensions system that I've only been *using* the
>{*filter*}y language for three days and I've already got most of a bytecode
>walker constructed.

;-)

- Show quoted text -

Quote:

>Here's the core of it:

>static VALUE
>bytecode_get_root(VALUE class)
>{
>    return Data_Wrap_Struct(class, 0, 0, ruby_eval_tree);
>}

>void Init_Bytecode () {
>    cByteNode = rb_define_class("ByteNode", rb_cObject);
>    rb_define_singleton_method(cByteNode, "root", bytecode_get_root, 0);
>    rb_define_method(cByteNode, "type", bytecode_nd_type, 0);
>    rb_define_method(cByteNode, "line", bytecode_nd_line, 0);
>    rb_define_method(cByteNode, "head", bytecode_nd_head, 0);
>    rb_define_method(cByteNode, "next", bytecode_nd_next, 0);
>    ...
>}

>The bytecode_nd_* functions just do a Data_Get_Struct, call the relevant
>nd_* macro, and wrap the result.

>Now I can say:

>% ruby -e 'require "Bytecode"; p ByteNode.root; p ByteNode.root.next.type'
>#<ByteNode:0x401a6550>
>4

>Is this making more sense now?

Yes and no.  Since Ruby doesn't use a bytecode interpretting virtual
machine (instead it walks an AST) what exactly do you mean by bytecode
here (or are you defining a bytecode for Ruby).

Quote:

>Of course, it'll make even more sense when I write something which converts
>type back into a string rather than an enum, etc. :)

I think what you're working on is very important and would be useful for a
variety of projects (Like Cardinal for example).  Please if possible can
you start up a sourceforge or savannah project for this?  After the Ruby
conference I think I might have some time to help out with this.

Phil



Mon, 11 Apr 2005 03:06:01 GMT  
 NODE tree introspection


Quote:

>> I've been considering writing something for Ruby similar to the Perl
>> B.pm module; this is a module which provides Perl-level access to the
>> bytecode op tree, so that other modules can walk the op tree in
>> various interesting ways. This is the way the Perl to C compiler was
>> implemented. (however unsuccessfully...)

>> It seems reasonably easy to do in Ruby, just by wrapping the NODE*
>> structures as Ruby objects. Has anyone already done this or tried
>> working on it?

>http://www.rubygarden.org/ruby?RubyInRuby
>http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/8693
>http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/6885

>I believe somebody attempted to ... uhmmm... do something using rockit.

Yes there are other projects out there with the goal of doing what Simon
is trying to accomplish (see also Ruth).  However, it seems that perhaps
the approach that he is taking might lead to something....

Phil



Mon, 11 Apr 2005 03:08:02 GMT  
 NODE tree introspection

Quote:

> Yes and no.  Since Ruby doesn't use a bytecode interpretting virtual
> machine (instead it walks an AST) what exactly do you mean by bytecode
> here (or are you defining a bytecode for Ruby).

Sorry; "bytecode" is probably the wrong word because it implies some kind
of serialized assembler. I'm using it in a much looser sense. Perl walks
an AST, too. But being able to get at and fiddle with the AST leads to a
lot of very interesting things: serialization of the internal form to disk
so you *do* get true bytecode, translation to C, graphing the internal
structure of the program, and so on.

My ulterior motive, of course, is to enable a Ruby->Parrot backend.

Quote:
> I think what you're working on is very important and would be useful for a
> variety of projects (Like Cardinal for example).  Please if possible can
> you start up a sourceforge or savannah project for this?

I'll need some help; as I'm very new to Ruby, I don't know exactly what's
required to set up and distribute a Ruby extension the politically correct
way. :)

What I'll do is dump the code into my CVS tree, and make it publically
available, and I can give you or whoever else commit access to
that. I'll also send you the code off-list for now, because I have to head
away for a few days.

--



Mon, 11 Apr 2005 05:08:55 GMT  
 NODE tree introspection

Quote:




> >> I've been considering writing something for Ruby similar to the Perl
> >> B.pm module; this is a module which provides Perl-level access to the
> >> bytecode op tree, so that other modules can walk the op tree in
> >> various interesting ways. This is the way the Perl to C compiler was
> >> implemented. (however unsuccessfully...)

> >> It seems reasonably easy to do in Ruby, just by wrapping the NODE*
> >> structures as Ruby objects. Has anyone already done this or tried
> >> working on it?

> >http://www.rubygarden.org/ruby?RubyInRuby
> >http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/8693
> >http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/6885

> >I believe somebody attempted to ... uhmmm... do something using rockit.

> Yes there are other projects out there with the goal of doing what Simon
> is trying to accomplish (see also Ruth).  However, it seems that perhaps
> the approach that he is taking might lead to something....

Funny, I should have read more about previous attempts to do 'Ruby in
Ruby', for I really thought that Simon's approach was THE approach :-)

--
 _           _                            
| |__   __ _| |_ ___ _ __ ___   __ _ _ __  
| '_ \ / _` | __/ __| '_ ` _ \ / _` | '_ \
| |_) | (_| | |_\__ \ | | | | | (_| | | | |
|_.__/ \__,_|\__|___/_| |_| |_|\__,_|_| |_|
        Running Debian GNU/Linux Sid (unstable)
batsman dot geo at yahoo dot com

Problem solving under Linux has never been the circus that it is under
AIX.
        -- Pete Ehlke in comp.unix.aix



Mon, 11 Apr 2005 17:09:58 GMT  
 NODE tree introspection
-----BEGIN PGP SIGNED MESSAGE-----



Quote:

>> Yes and no.  Since Ruby doesn't use a bytecode interpretting virtual
>> machine (instead it walks an AST) what exactly do you mean by bytecode
>> here (or are you defining a bytecode for Ruby).

>Sorry; "bytecode" is probably the wrong word because it implies some kind
>of serialized assembler. I'm using it in a much looser sense. Perl walks
>an AST, too. But being able to get at and fiddle with the AST leads to a
>lot of very interesting things: serialization of the internal form to disk
>so you *do* get true bytecode, translation to C, graphing the internal
>structure of the program, and so on.

>My ulterior motive, of course, is to enable a Ruby->Parrot backend.

>> I think what you're working on is very important and would be useful for a
>> variety of projects (Like Cardinal for example).  Please if possible can
>> you start up a sourceforge or savannah project for this?

>I'll need some help; as I'm very new to Ruby, I don't know exactly what's
>required to set up and distribute a Ruby extension the politically correct
>way. :)

- - Don't worry, nobody else knows how to do it the "politically
  correct" way yet either. There is no standard way yet to
  set up a module. There is the ruby equivalent of Config.pm
  in rbconfig.rb and rpkg is an emerging standard, but there
  is still not consensus or anything remotely like h2xs,
  Makefile.PL and CPAN.

Quote:

>What I'll do is dump the code into my CVS tree, and make it publically
>available, and I can give you or whoever else commit access to
>that. I'll also send you the code off-list for now, because I have to head
>away for a few days.

- - Did you look at Ruth? It's not quite the same as what you're
  trying to do, but it does walk the AST tree fairly well.

- - Booker C. Bense

-----BEGIN PGP SIGNATURE-----
Version: 2.6.2

iQCVAwUBPblb8mTWTAjn5N/lAQEkqAP/WwbQYYW4i6IbwsnaT+wEsZPYGpapY+fc
uK48/n3mBJhqFGN4mRvsu95gal0aFwAlqFc9cP+ipRoXMcMFOSbRUd6Ht509VKca
oZ7JOwFcYmU3Ot/mah3+cFItqfZaoF23w4j4LFbeAJh/Zm1b/ekAONhthcMwNAPf
mRlhvD3w+zY=
=Hi/c
-----END PGP SIGNATURE-----



Tue, 12 Apr 2005 22:57:55 GMT  
 NODE tree introspection


Quote:

>> Hmmmm.... interesting idea.  So basically what you're proposing would be
>> able to walk the AST of the currently running program (or any arbitrary
>> snippet that you send off to eval?)?  

>Yes. That's what Perl's B.pm does.

> % perl -MB -le '$x= B::main_start; print $x->name while $x=$x->next'
>nextstate
>pushmark
>gv
>entersub
>gvsv
>sassign
>...

>(New statement, add a mark to the stack, add a symbol table entry
>[B::main_start] to the stack, call it, put a variable on the stack,
>assign the result of the call to the variable...)

>> I don't think anyone has approached the problem in the way you propose.
>> Can you give some more details of how you're thinking of doing it?

>I can even give you some sample code, because I've already got a fair bit
>of it working. :)

>It says a lot for Ruby's ease of use, clarity of code and
>well-developed extensions system that I've only been *using* the
>{*filter*}y language for three days and I've already got most of a bytecode
>walker constructed.

>Here's the core of it:

>static VALUE
>bytecode_get_root(VALUE class)
>{
>    return Data_Wrap_Struct(class, 0, 0, ruby_eval_tree);
>}

>void Init_Bytecode () {
>    cByteNode = rb_define_class("ByteNode", rb_cObject);
>    rb_define_singleton_method(cByteNode, "root", bytecode_get_root, 0);
>    rb_define_method(cByteNode, "type", bytecode_nd_type, 0);
>    rb_define_method(cByteNode, "line", bytecode_nd_line, 0);
>    rb_define_method(cByteNode, "head", bytecode_nd_head, 0);
>    rb_define_method(cByteNode, "next", bytecode_nd_next, 0);
>    ...
>}

>The bytecode_nd_* functions just do a Data_Get_Struct, call the relevant
>nd_* macro, and wrap the result.

>Now I can say:

>% ruby -e 'require "Bytecode"; p ByteNode.root; p ByteNode.root.next.type'
>#<ByteNode:0x401a6550>
>4

I wonder if we could do this a lot faster by using swig to wrap the
Ruby C parser?  Swig Ruby itself to make an extension for Ruby.

...just a thought.

Phil



Sat, 16 Apr 2005 05:24:01 GMT  
 NODE tree introspection

Quote:

> I wonder if we could do this a lot faster by using swig to wrap the
> Ruby C parser?  Swig Ruby itself to make an extension for Ruby.

First, SWIG is there when your own language's extension mechanism is a
pain in the ass; Ruby's extension system is just fine, (much more
sensible than Perl's or Python's) and so I don't think SWIG is the
answer. (to anything) And, to be blunt, I prefered Perl's XS to SWIG.
Your milage, of course, is sure to vary.

Second, SWIG is particularly useful when you're trying to do cross-language
library extensions. But we're only trying to export Ruby's AST to Ruby, so
cross-langage concerns are irrelevant.

We dealt with SWIG in Extending and Embedding Perl
(http://www.amazon.com/exec/obidos/ASIN/1930110820/) and, to be frank,
I really don't want to deal with it again. Ruby's extension API,
although I've beaten it up in the past
(http://www-6.ibm.com/jp/developerworks/linux/001027/ruby_qa.html)
kicks seven shades of shite out of Perl's.

--
WILLIAM: Just incidentally -- why are you wearing that?
SPIKE: Ahm -- combination of factors really. No clean clothes...
W: There never will be, you know, unless you actually clean your clothes.
S: Right.  Vicious circle.



Sat, 16 Apr 2005 06:48:47 GMT  
 NODE tree introspection


Quote:

>> I wonder if we could do this a lot faster by using swig to wrap the
>> Ruby C parser?  Swig Ruby itself to make an extension for Ruby.

>First, SWIG is there when your own language's extension mechanism is a
>pain in the ass; Ruby's extension system is just fine, (much more
>sensible than Perl's or Python's) and so I don't think SWIG is the
>answer. (to anything) And, to be blunt, I prefered Perl's XS to SWIG.
>Your milage, of course, is sure to vary.

Where SWIG really shines is when you've got some substantial C or C++
library that you want to wrap pretty-much automatically (and I'm always
amazed how _little_ I have to do to use SWIG).

Quote:
>Second, SWIG is particularly useful when you're trying to do cross-language
>library extensions. But we're only trying to export Ruby's AST to Ruby, so
>cross-langage concerns are irrelevant.

Well, I guess I should put it differently: We could use SWIG to wrap
ruby.h and node.h and basically do what you're trying to do, but more
automatically.

<TANGENT>
That brings up an interesting point: What if one were to wrap
python with SWIG so that the Python interpreter was available from within
Ruby?
</TANGENT>

Quote:

>We dealt with SWIG in Extending and Embedding Perl
>(http://www.amazon.com/exec/obidos/ASIN/1930110820/) and, to be frank,
>I really don't want to deal with it again.

What version of SWIG did you use?  I guess I started out with SWIG 1.3.13
and now use 1.3.15 and have find it very easy to use.  I've heard that
earlier versions were more of a pain.  Also, I suspect you were using SWIG
to make Perl extensions - I've only used it to create Ruby extensions, so
perhaps there are some differences with Perl.

Quote:
>Ruby's extension API,
>although I've beaten it up in the past
>(http://www-6.ibm.com/jp/developerworks/linux/001027/ruby_qa.html)
>kicks seven shades of shite out of Perl's.

Yes, Ruby's extension API is great, but when you've got lots of existing
code to wrap SWIG works great in my experience.

Phil



Sat, 16 Apr 2005 11:45:02 GMT  
 NODE tree introspection

Quote:




>>Second, SWIG is particularly useful when you're trying to do cross-language
>>library extensions. But we're only trying to export Ruby's AST to Ruby, so
>>cross-langage concerns are irrelevant.

>Well, I guess I should put it differently: We could use SWIG to wrap
>ruby.h and node.h and basically do what you're trying to do, but more
>automatically.

OK, I see the error of my ways - I just tried to wrap node.h with SWIG
and yeah, that's not gonna work.... you end up wrapping stuff that
obviously doesn't need to be wrapped (he says sheepishly:)

Have you added any more code to your bytecode.c?

Quote:

><TANGENT>
>That brings up an interesting point: What if one were to wrap
>Python with SWIG so that the Python interpreter was available from within
>Ruby?
></TANGENT>

But this might still be a viable idea for someone who might want to
evaluate Python code from within Ruby (or I suppose the same could be
done with Perl or Tcl).

Phil



Sat, 16 Apr 2005 12:49:28 GMT  
 
 [ 13 post ] 

 Relevant Pages 

1. Tree nodes and CwWidget

2. ActiveX tree control's wiat event does not abort after nodes are added

3. Building Tree for list of nodes in level order

4. <tree-node> in DUIM

5. Help: TREEs, NODEs, and COLLECTIONs

6. random node in tree

7. Internal Nodes of an XOR tree

8. Internal nodes of an XOR tree

9. Binary trees storing huge amounts of data in nodes

10. help with node height in a tree

11. Bwidget tree node limits

12. Tree node in motion and scollbar

 

 
Powered by phpBB® Forum Software