CMUCL performance in the Shootout 
Author Message
 CMUCL performance in the Shootout

Why did CMUCL do so poorly in the "Shootout" ? It is a few times
slower than C for matrix multiplication. All types were declared. On
my machine, when I test it from REPL, so there are no start-up costs,
Lisp is 4-5 times slower than C. Why?

http://www.*-*-*.com/ ~doug/shootout/bench/matrix/

P.S.   I hope no one gets over-sensitive because of this question.



Wed, 09 Nov 2005 10:10:22 GMT  
 CMUCL performance in the Shootout

--On Friday, May 23, 2003 7:10 PM -0700 Michael Park

Quote:

> Why did CMUCL do so poorly in the "Shootout" ? It is a few times
> slower than C for matrix multiplication. All types were declared. On
> my machine, when I test it from REPL, so there are no start-up costs,
> Lisp is 4-5 times slower than C. Why?

> http://www.bagley.org/~doug/shootout/bench/matrix/

Why did C do so poorly in the "Shootout"?  It is a few times slower that
CMUCL for regular expression matching.  Why?

http://www.bagley.org/~doug/shootout/bench/regexmatch/

I'd hardly call 6th overall "doing so poorly"!

I guess when Doug states in his "Best of Show":

"I've dabbled with Lisp (e.g. CMUCL) on and off for many years, but the
shootout gave me an opportunity to look more at Common Lisp and at Scheme
(Bigloo) too, which I hadn't done before. This family of languages has so
much to offer, it's a shame that they are not more popular."

he simply didn't know what he was talking about.

Quit trying to get a rise out of the group...



Wed, 09 Nov 2005 14:04:02 GMT  
 CMUCL performance in the Shootout

Quote:
> Why did C do so poorly in the "Shootout"?  It is a few times slower that
> CMUCL for regular expression matching.  Why?

In this case, Shootout does not measure C performance, but PCRE
performance, 8-)


Wed, 09 Nov 2005 21:13:25 GMT  
 CMUCL performance in the Shootout

Quote:

> Why did CMUCL do so poorly in the "Shootout" ? It is a few times
> slower than C for matrix multiplication. All types were declared. On
> my machine, when I test it from REPL, so there are no start-up costs,
> Lisp is 4-5 times slower than C. Why?

> http://www.bagley.org/~doug/shootout/bench/matrix/

> P.S.   I hope no one gets over-sensitive because of this question.

I have benchmarked a 300x300 matrix multiplication. The fastest CMUCL
version was only 2 times slower than the C++ version using TNT::Array2D
(Template Numerical Library). The CMUCL program still made array bound
checks while the C++ program did not. So I think CMUCL is quite fast in
this example although not as fast as C++ (a C program using raw C-arrays
may be even faster).

The main advantage of CMUCL for me is the combination of fast
interactive and dynamic developement and reasonable fast compiled code.
There is no other language/implementation that is comparable to CMUCL
in that respect.

Rolf Wester



Wed, 09 Nov 2005 21:16:04 GMT  
 CMUCL performance in the Shootout

Quote:

>  Why did CMUCL do so poorly in the "Shootout" ? It is a few times
>  slower than C for matrix multiplication. All types were declared. On
>  my machine, when I test it from REPL, so there are no start-up costs,
>  Lisp is 4-5 times slower than C. Why?

Looking at the disassembled code from both versions
(which I'll call "C" and "L" with the obvious meanings):

  - C has an inner loop 9 instructions long; L has an
    inner loop 21 instructions long.

  - C keeps the accumulated sum in a register
    throughout the calculation; L spills it.

  - C keeps the inner loop's counter in a register
    throughout the calculation; L spills it.

  - C moves one indexing operation out of the inner loop;
    L doesn't.

Quote:
>  P.S.   I hope no one gets over-sensitive because of this question.

If you want to avoid people getting over-sensitive,
it might be advisable to phrase the question in a
less provocative way. You found *one* test in which
CMUCL does much worse than C; that doesn't constitute
"doing poorly" on its own. (My vague recollection:
there are a couple of tests on which it's substantially
worse than C, and for the rest it's similar in speed.)

--

.sig under construc



Wed, 09 Nov 2005 21:54:33 GMT  
 CMUCL performance in the Shootout



Quote:

> --On Friday, May 23, 2003 7:10 PM -0700 Michael Park

> > Why did CMUCL do so poorly in the "Shootout" ? It is a few times
> > slower than C for matrix multiplication. All types were declared. On
> > my machine, when I test it from REPL, so there are no start-up costs,
> > Lisp is 4-5 times slower than C. Why?

> > http://www.bagley.org/~doug/shootout/bench/matrix/

> Why did C do so poorly in the "Shootout"?  It is a few times slower that
> CMUCL for regular expression matching.  Why?

> http://www.bagley.org/~doug/shootout/bench/regexmatch/

Doesn't this graph really show that we should all be using OCaml?

;-)



Wed, 09 Nov 2005 22:55:23 GMT  
 CMUCL performance in the Shootout

Quote:

> Doesn't this graph really show that we should all be using OCaml?

Btw -- there is somewhere an interesting variation on the shootout, with
the *bytes* of code metric, using both plain and bzip2'edded
sources. Can't seem to find it right now, though.

IIRC Ocaml did a lot worse in those if any significant weight was given to
the source size.

What I'd like to see (or do, if I find the time) would be a version of the
shootout using various CL implementations -- both commercial and free --
plus maybe g++ and Sun javac to give a non-CL baseline. Also, a FP
intensive test would be interesting as well.

Cheers,

  -- Nikodemus



Wed, 09 Nov 2005 23:27:11 GMT  
 CMUCL performance in the Shootout


Quote:
> Doesn't this graph really show that we should all be using OCaml?

I've played with OCaml and it certainly seems to be a very attractive
language if you already know what you want before you start, but I never
seem to.  The OCaml guys have had some spectacular sucesses in the ICFP
contest (e.g. the Ray Tracer in '00), but also some big failures (e.g.
last year).

Four weeks to the next one!

-- Bruce
   sticking with Dylan, but hoping to see more CL entries



Thu, 10 Nov 2005 08:55:43 GMT  
 CMUCL performance in the Shootout

Quote:

> The OCaml guys have had some spectacular sucesses in the ICFP
> contest (e.g. the Ray Tracer in '00), but also some big
> failures (e.g.  last year).

I think you mean two years ago.  Last year first place was taken
by a team using OCaml.

  Neil



Fri, 11 Nov 2005 00:34:04 GMT  
 CMUCL performance in the Shootout


Quote:

> > The OCaml guys have had some spectacular sucesses in the ICFP
> > contest (e.g. the Ray Tracer in '00), but also some big
> > failures (e.g.  last year).

> I think you mean two years ago.  Last year first place was taken
> by a team using OCaml.

Hmm ... you're correct.

I was thinking of "the OCaml guys" at INRIA (Camls 'R Us), who came in
81st last year.

There were a number of other teams using OCaml who did better.  Which is
of course a good reflection on the language.

-- Bruce



Fri, 11 Nov 2005 05:35:59 GMT  
 CMUCL performance in the Shootout
I love trolls.  My favorite troll is one that has been sautaid (ispell
sucks) in a mushroom, garlic, butter sauce.  hmmm mmm good!

Just don't expose them to sunlight.  Stone is much harder to eat.

--
(describe 'describe)



Fri, 11 Nov 2005 06:30:49 GMT  
 CMUCL performance in the Shootout

Quote:

> I love trolls.  My favorite troll is one that has been sautaid (ispell
> sucks) in a mushroom, garlic, butter sauce.  hmmm mmm good!

> Just don't expose them to sunlight.  Stone is much harder to eat.

But then you get lawn trolls, get enough and you can start
your own miniature golf course/theme park.

marc



Fri, 11 Nov 2005 06:50:51 GMT  
 CMUCL performance in the Shootout



|

|| | The OCaml guys have had some spectacular sucesses in the ICFP
|| | contest (e.g. the Ray Tracer in '00), but also some big
|| | failures (e.g.  last year).
||
|| I think you mean two years ago.  Last year first place was taken
|| by a team using OCaml.
|
| Hmm ... you're correct.
|
| I was thinking of "the OCaml guys" at INRIA (Camls 'R Us), who came in
| 81st last year.
|
| There were a number of other teams using OCaml who did better.  Which is
| of course a good reflection on the language.

My team came in 24th out of 168, and the three of us probably had
around 10 man-hours of experience in OCaml before the contest started.
That slowed us down, but there was a lot about OCaml that sped us up
even more.  (I do like Lisp too.)

--
http://www.dfan.org



Fri, 11 Nov 2005 11:18:04 GMT  
 CMUCL performance in the Shootout

Quote:


> >  Why did CMUCL do so poorly in the "Shootout" ? It is a few times
> >  slower than C for matrix multiplication. All types were declared. On
> >  my machine, when I test it from REPL, so there are no start-up costs,
> >  Lisp is 4-5 times slower than C. Why?

> Looking at the disassembled code from both versions
> (which I'll call "C" and "L" with the obvious meanings):

>   - C has an inner loop 9 instructions long; L has an
>     inner loop 21 instructions long.

>   - C keeps the accumulated sum in a register
>     throughout the calculation; L spills it.

>   - C keeps the inner loop's counter in a register
>     throughout the calculation; L spills it.

>   - C moves one indexing operation out of the inner loop;
>     L doesn't.

1.) Is there any way I can convince CMUCL to perform these particular
optimizations? Or do some kind of inline assembly? "Silly loops" like
these are kind of important in the code I tend to write.

2.) I always kind of envied people who could read assembly. With no
prior knowledge of x86 registers or assembly language, how much
learning would be required (in earth years) to learn this kind of
stuff? Any specific tutorials suggestions? Thanks

P.S. I don't think "bounds checking" necessarily needs to be
compromised. Bounds checks can probably be optimised away, as no one
resizes the matrices while they are being multiplied.

P.P.S. I want Lisp programs to run faster, I must be a troll. Thanks
for noticing.



Fri, 11 Nov 2005 14:22:55 GMT  
 CMUCL performance in the Shootout

Quote:



> > Doesn't this graph really show that we should all be using OCaml?

> I've played with OCaml and it certainly seems to be a very attractive
> language if you already know what you want before you start, but I never
> seem to.  The OCaml guys have had some spectacular sucesses in the ICFP
> contest (e.g. the Ray Tracer in '00), but also some big failures (e.g.
> last year).

> Four weeks to the next one!

Four weeks or four months?

Quote:
>    sticking with Dylan, but hoping to see more CL entries

How does one submit a CL entry? As code only? As an image? As fasl?
What if their version of CMUCL is different?


Fri, 11 Nov 2005 14:59:40 GMT  
 
 [ 71 post ]  Go to page: [1] [2] [3] [4] [5]

 Relevant Pages 

1. Dynamic language performance shootout

2. Ruby performance on the Language Shootout

3. Performance/compatibility question: Allegro CL vs CMUCL?

4. ECLS/CLISP/CMUCL performance -- revised

5. Numerical performance of CMUCL

6. CL (CLISP vs CMUCL) performance on Linux

7. CL (CLISP vs CMUCL) performance on Linux

8. Gforth and the language shootout

9. language shootout page

10. Paper online: "Web Framework Shootout"

11. Ruby code is the shortest (language shootout)

12. Computer Language Shootout...

 

 
Powered by phpBB® Forum Software