Ruby parsers in Ruby 
Author Message
 Ruby parsers in Ruby

Wouldn't it be cool to have Ruby playing with Parrot before python or even
perhaps Perl?  That could be a real coup.

Dan Sugalski has suggested that it would be very helpful if we had a Ruby
parser written in Ruby.  It would initially need to spit out Parrot
bytecode, but eventually we could give him an AST.

Now, it seems to me that we have at least a couple of projects where we
parse Ruby in Ruby (actually, three that I know of):
1) Markus Liedl's Ruby parser - it was announced a while back on
Rubygarden that it was recently imported into the CVS tree (I think that
means Ruby's CVS tree).
2) JRuby - It must have a frontend parser written in Ruby and then emits
Java bytecode, right?
3) Matju's AST - which produces an abstract syntax tree of a Ruby program.

Is there any possibility that efforts can be merged in this area?  It
seems that we have three options available and there must be a pretty good
amount of code out there already for a Ruby parser written in pure Ruby,
so instead of maintaining three different code bases perhaps we could
create one Ruby in Ruby parser frontend (based on picking the best
elements of the above three) and then create different 'backends'.
The frontend would create an AST which could be fed to the different
backends:
JRuby - emits Java bytecode
PRuby - emits Parrot bytecode
RRuby - emits Ruby bytecode for upcomeing Rite VM

While competition has it merits, I think this is an area where we as a
community could benefit from joining forces instead of duplicating effort.

Can anyone involved in the above three projects (or any other similar ones
I might have missed) comment on the status of these projects and the
feasability of merging them?

Phil



Thu, 17 Jun 2004 14:44:38 GMT  
 Ruby parsers in Ruby


Quote:
> Wouldn't it be cool to have Ruby playing with Parrot before Python or even
> perhaps Perl?  That could be a real coup.

> Dan Sugalski has suggested that it would be very helpful if we had a Ruby
> parser written in Ruby.  It would initially need to spit out Parrot
> bytecode, but eventually we could give him an AST.

> Now, it seems to me that we have at least a couple of projects where we
> parse Ruby in Ruby (actually, three that I know of):
> 1) Markus Liedl's Ruby parser - it was announced a while back on
> Rubygarden that it was recently imported into the CVS tree (I think that
> means Ruby's CVS tree).
> 2) JRuby - It must have a frontend parser written in Ruby and then emits
> Java bytecode, right?

Wrong, this wouldn't work well since it would need ruby first and in any
case
we are not yet emitting java bytecode.
It started like a translation of the c ruby interpreter in java and is being
refactored to enable it to emit java bytecode.  So it has a java ruby parser
which builds an AST which is intepreted.

Quote:
> 3) Matju's AST - which produces an abstract syntax tree of a Ruby program.

> Is there any possibility that efforts can be merged in this area?  It
> seems that we have three options available and there must be a pretty good
> amount of code out there already for a Ruby parser written in pure Ruby,
> so instead of maintaining three different code bases perhaps we could
> create one Ruby in Ruby parser frontend (based on picking the best
> elements of the above three) and then create different 'backends'.
> The frontend would create an AST which could be fed to the different
> backends:
> JRuby - emits Java bytecode
> PRuby - emits Parrot bytecode
> RRuby - emits Ruby bytecode for upcomeing Rite VM

This would be nice, however as far as JRuby is concerned this would be like
starting from scratch, the code we are writing is all in java and the Java
bytecode will be emited by a java library.  Of course we could use our Ruby
java interop layer to call in the java library from Ruby but in this case
only
jruby would be able to interpret the JRuby backend and we have a chicken and
egg
problem.

Quote:

> While competition has it merits, I think this is an area where we as a
> community could benefit from joining forces instead of duplicating effort.

> Can anyone involved in the above three projects (or any other similar ones
> I might have missed) comment on the status of these projects and the
> feasability of merging them?

> Phil

Benoit
one of the JRuby developers


Thu, 17 Jun 2004 19:17:47 GMT  
 Ruby parsers in Ruby

Quote:



>> Wouldn't it be cool to have Ruby playing with Parrot before Python or even
>> perhaps Perl?  That could be a real coup.

>> Dan Sugalski has suggested that it would be very helpful if we had a Ruby
>> parser written in Ruby.  It would initially need to spit out Parrot
>> bytecode, but eventually we could give him an AST.

>> Now, it seems to me that we have at least a couple of projects where we
>> parse Ruby in Ruby (actually, three that I know of):
>> 1) Markus Liedl's Ruby parser - it was announced a while back on
>> Rubygarden that it was recently imported into the CVS tree (I think that
>> means Ruby's CVS tree).
>> 2) JRuby - It must have a frontend parser written in Ruby and then emits
>> Java bytecode, right?
>Wrong, this wouldn't work well since it would need ruby first and in any
>case
>we are not yet emitting java bytecode.
>It started like a translation of the c ruby interpreter in java and is being
>refactored to enable it to emit java bytecode.  So it has a java ruby parser
>which builds an AST which is intepreted.

Sorry, my misunderstanding.  So you parse Ruby in Java and emit Java
bytecode, of course that makes sense.

It still leaves us with two different projects which create a Ruby parser
in Ruby (unless there are others I don't know about).

Phil



Fri, 18 Jun 2004 01:25:14 GMT  
 Ruby parsers in Ruby

Quote:

> > The frontend would create an AST which could be fed to the different
> > backends:
> > JRuby - emits Java bytecode
> > PRuby - emits Parrot bytecode
> > RRuby - emits Ruby bytecode for upcomeing Rite VM
>This would be nice, however as far as JRuby is concerned this would be like
>starting from scratch, the code we are writing is all in java and the Java
>bytecode will be emited by a java library.

Hmmm. I suppose we could run the java version of the ruby parser through
the java bytecode->parrot bytecode translator. Dunno what that'd look like
for speed, though.

                                        Dan

--------------------------------------"it's like this"-------------------
Dan Sugalski                          even samurai

                                      teddy bears get drunk



Fri, 18 Jun 2004 02:49:55 GMT  
 Ruby parsers in Ruby

Quote:
> Hmmm. I suppose we could run the java version of the ruby parser through
> the java bytecode->parrot bytecode translator. Dunno what that'd look like
> for speed, though.

> Dan

In any case for now the ruby to java bytecode is a little bit of wishfull
thinking
what we do right now is interpret the AST in java, like I said we now have a
port
to java of the C interpreter but with a little bit more object orientation
to it (as
java allows).
Dan,
I don't think this would be a good idea since it would loose most of the
advantages
of parrot for ruby which is to be a VM built for dynamic languages.  Dynamic
languages translated in java bytecode usually have performance problems and
other problems due mostly to java being made for static languages.

Benoit
PS: I'm happy that you are taking so much interest in ruby Dan, I see parrot
as the future for ruby for plenty of reason, one of them being that it gives
ruby program access to a wider install base and a potentially bigger body of
extensions.



Fri, 18 Jun 2004 18:38:04 GMT  
 Ruby parsers in Ruby
Robert Feldt is working on a Ruby parser written in Ruby. See:
  http://groups.google.com/groups?q=rubyinruby&hl=en&btnG=Google+Search

Curt



Fri, 18 Jun 2004 19:14:23 GMT  
 Ruby parsers in Ruby

Quote:
> > Hmmm. I suppose we could run the java version of the ruby parser through
> > the java bytecode->parrot bytecode translator. Dunno what that'd look like
> > for speed, though.

> > Dan
>In any case for now the ruby to java bytecode is a little bit of wishfull
>thinking
>what we do right now is interpret the AST in java, like I said we now have a
>port
>to java of the C interpreter but with a little bit more object orientation
>to it (as
>java allows).
>Dan,
>I don't think this would be a good idea since it would loose most of the
>advantages
>of parrot for ruby which is to be a VM built for dynamic languages.  Dynamic
>languages translated in java bytecode usually have performance problems and
>other problems due mostly to java being made for static languages.

Oh, we don't have to use it as an endpoint, just as a starting point.

Quote:
>PS: I'm happy that you are taking so much interest in ruby Dan, I see parrot
>as the future for ruby for plenty of reason, one of them being that it gives
>ruby program access to a wider install base and a potentially bigger body of
>extensions.

I rather like Ruby--I think it's a nice language. I'd really like some of
the core Ruby datatypes to get into the core Parrot source and test suite.
It'll help keep us honest. (We need to get method dispatch done first, though)

                                        Dan

--------------------------------------"it's like this"-------------------
Dan Sugalski                          even samurai

                                      teddy bears get drunk



Fri, 18 Jun 2004 23:23:45 GMT  
 Ruby parsers in Ruby


Quote:

>> > Hmmm. I suppose we could run the java version of the ruby parser through
>> > the java bytecode->parrot bytecode translator. Dunno what that'd look like
>> > for speed, though.

I don't think we need to do that.  Hopefully we can get either the
ruby-parser in the 1.7 CVS going, or perhaps (and I kind of prefer this)
build on the ruby parser example that comes with Rockit.  Rockit seems
like a more advanced parser builder than the other options.

Quote:

>> > Dan
>>In any case for now the ruby to java bytecode is a little bit of wishfull
>>thinking
>>what we do right now is interpret the AST in java, like I said we now have a
>>port
>>to java of the C interpreter but with a little bit more object orientation
>>to it (as
>>java allows).
>>Dan,
>>I don't think this would be a good idea since it would loose most of the
>>advantages
>>of parrot for ruby which is to be a VM built for dynamic languages.  Dynamic
>>languages translated in java bytecode usually have performance problems and
>>other problems due mostly to java being made for static languages.

>Oh, we don't have to use it as an endpoint, just as a starting point.

I think we might be wasting effort trying to go down that road.  Effort
that would be better spent working on a true RubyinRuby solution which can
be used by several other projects as well.

Phil



Sat, 19 Jun 2004 02:24:16 GMT  
 Ruby parsers in Ruby

Quote:

>Robert Feldt is working on a Ruby parser written in Ruby. See:
>  http://groups.google.com/groups?q=rubyinruby&hl=en&btnG=Google+Search

Is it the same as the ruby parser example that comes with Rockit?

I also see mention of something called Ruth which uses Matz's parser but
converts the internal tree nodes to Ruby objects, so there's another
potential option.

I guess what I'm trying to figure out is which of the options is furthest
along and if perhaps there might be some cooperation possible amoung them.

Phil



Sat, 19 Jun 2004 02:33:00 GMT  
 Ruby parsers in Ruby

Quote:

> Robert Feldt is working on a Ruby parser written in Ruby. See:
>   http://groups.google.com/groups?q=rubyinruby&hl=en&btnG=Google+Search

For what it's worth, RDoc contains a full Ruby lexer and a fair amount
of a parser, all in pure Ruby. Most of the credit belongs to Keiju
Ishitsuka, who wrote the irb code that I ripped off.

Dave



Sun, 20 Jun 2004 12:30:19 GMT  
 Ruby parsers in Ruby

Quote:
> Now, it seems to me that we have at least a couple of projects where we
> parse Ruby in Ruby (actually, three that I know of):

What I would like more than a joint parser project is a clear Ruby
specification.
It's impossible to do proper parallel implementations without a proper spec.

What is needed is the exact grammar and lexer rules.
And an up to date specification of the standard libraries. The spec need not
be as well-written as say the pick-axe book, but it should unambigiously
define behanviour of the classes.
This is particularly important because ever so often a new function is
introduced which is not covered by the printed publications.
The spec. should preferably also mention the version a given feature were
introduced (and deprecated where applicable).

Wrt. the lexer/parser spec. It may be that the Rockit language is suitable
for this. It would be nice with a compilable specification. A distinct
feature of Rockit is that it doesn't blend implementation with specification
and that it does not have as many problems with conflicts as many other
grammar languages.

MikkelFJ



Sun, 20 Jun 2004 21:48:46 GMT  
 Ruby parsers in Ruby

Quote:

> And an up to date specification of the standard libraries. The spec
> need not be as well-written as say the pick-axe book, but it should
> unambigiously define behanviour of the classes.  This is
> particularly important because ever so often a new function is
> introduced which is not covered by the printed publications.

Imagine... that the C source of the built-in libraries in Rite
included RDoc documentation of each class, module, and method
(bootstrapped from the current ri descriptions, perhaps)...

You may say I'm a dreamer....

Dave



Sun, 20 Jun 2004 23:24:39 GMT  
 Ruby parsers in Ruby
Hi,

In message "[ruby-talk:30040] Re: Ruby parsers in Ruby"

|What I would like more than a joint parser project is a clear Ruby
|specification.
|It's impossible to do proper parallel implementations without a proper spec.
|
|What is needed is the exact grammar and lexer rules.

The "ruby" source?  parse.y is compilable unambiguous grammer
description. ;-)

Seriously, Ruby had been too "hot" to write human readable "clear
specification".  Recently Ruby (at least 1.7) grammer and behaviors
are becoming relatively stable, so that such attempt may be
reasonable.

I would welcome volunteers for such effort, as long as the process for
me to upgrade the spec. is provided.

                                                        matz.



Sun, 20 Jun 2004 23:51:15 GMT  
 Ruby parsers in Ruby

Quote:

> |What is needed is the exact grammar and lexer rules.

> The "ruby" source?  parse.y is compilable unambiguous grammer
> description. ;-)

> Seriously, Ruby had been too "hot" to write human readable "clear
> specification".  Recently Ruby (at least 1.7) grammer and behaviors
> are becoming relatively stable, so that such attempt may be
> reasonable.

This has been a long-term goal for me but I keep postponing it since my
thesis writing is currently taking up all my time.

If anyone care to do this I think you can be helped by Rulator which is in
the Rockit CVS at

http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/rockit/src/examples/ruby/

Its a hybrid RubyInRuby parser with the lexer specified with Rockit 0.4
parser generators/combinators and the parser itself being generated by
Racc from the grammar in parse.y. It can be used both to build a RubyAST
tree or as an event based parser.

Rich Kilmer has helped out by writing tests for the lexer but more tests
are needed to weed out bugs (in the parser). Any help appreciated.

Regards,

Robert



Mon, 21 Jun 2004 00:07:18 GMT  
 Ruby parsers in Ruby

Quote:


> > And an up to date specification of the standard libraries. The spec
> > need not be as well-written as say the pick-axe book, but it should
> > unambigiously define behanviour of the classes.  This is
> > particularly important because ever so often a new function is
> > introduced which is not covered by the printed publications.

> Imagine... that the C source of the built-in libraries in Rite
> included RDoc documentation of each class, module, and method
> (bootstrapped from the current ri descriptions, perhaps)...

Do you mean something like Emacs Lisp documentation strings?

--
Pierre-Charles David (pcdavid <at> emn <dot> fr)
Computer Science PhD Student, cole des Mines de Nantes, France
Homepage: http://purl.org/net/home/pcdavid



Mon, 21 Jun 2004 00:17:56 GMT  
 
 [ 20 post ]  Go to page: [1] [2]

 Relevant Pages 

1. Ruby Syntax Highlighting (and a Ruby Parser BUG)

2. pthread trouble with ruby-opengl on FreeBSD (was: Re: [announcement] Ruby 3D Ruby)

3. Ruby books (Ruby NG FAQ, Ruby FAQ, home page)

4. Why are parser tools rarely used in ruby?

5. Why are parser tools rarely used in Ruby?

6. Bug in cvs ruby parser

7. Ruby, XML schema( RELAX NG) and a Q&D parser

8. ruby-htmltools, a tree-building HTML parser version 1.01

9. ruby-htmltools, a tree-building HTML parser

10. Regexp::Parser ported to Ruby...?

11. Markus' ruby-parser (part 2)

12. Markus' ruby-parser

 

 
Powered by phpBB® Forum Software