Benchmark Results for CL's onWin32 
Author Message
 Benchmark Results for CL's onWin32

A while ago I mentioned in a posting that I had done some benchmarking
of Common Lisp's for Win32. I received a few emails requesting the
results. So....here they are.

CL-TYPE           Allegro CL LispWorks Poplog CL CLISP      CormanLisp
CL-VER            5.0.1      4.1.18    2.0       1999-07-22 1.3
BOYER              1.000     10.050     1.033     4.040      1.708
BROWSE             1.000      4.487     2.184     5.142      3.355
CTAK               1.000      1.839     5.903     4.258      3.757
DDERIV             1.288      3.220     2.190     3.220      1.000
DERIV              1.483      3.758     1.928     3.791      1.000
DESTRU             1.000      2.639     2.222     5.361      1.500
DIV2               1.279      4.256     1.790     5.864      1.000
FFT                1.000     41.050    67.000    46.050     53.998
FPRINT             1.630      1.935     4.261     1.000      6.168
FREAD              1.458      6.958     1.250     1.000      8.534
FRPOLY             1.084      1.737     1.000     3.106      1.299
PUZZLE             1.441      1.816     1.000     4.850      5.409
STAK               1.000      1.333     2.310     4.167      3.910
TAK                1.000      3.500     3.500    21.025      1.863
TAKL               1.000      2.321     2.321    16.357      1.608
TAKR               1.000      4.273     2.000     9.182      1.450
TRAVERSE           1.000      5.156     3.370    13.167      5.829
TRIANG             1.287      1.000     2.779    15.688      8.254
RHYME              1.000      2.178     1.831     2.324     95.729
PRIMECOUNTFIXNUM   1.000      1.740     2.122     3.878      6.463
PRIMECOUNTBIGNUM   2.296      1.000     1.063     1.489     26.100
PNPOLY             1.000      1.290    32.031    42.536     42.301
GENERICFUNCTIONS   1.000      1.111     5.926     3.951     37.946
SLOTVALUES         1.074      1.000     1.407     1.639      5.027
TOTAL              1.000      3.872     5.382     7.877     11.483

Notes:

1) All benchmarks run on a Pentium II at 366Mhz with Windows NT.

2) The benchmark values are normalized to the best performer. The total
is computed
as the sum of the individual normalized values and then is normalized
itself. This
means that each benchmark is weighted equally in the total.

3) Allegro CL http://www.*-*-*.com/ (I used the commercial version)

4) Lispworks http://www.*-*-*.com/ (I used the commercial version)

5) Poplog CL http://www.*-*-*.com/
(Free with source)

7) Clisp http://www.*-*-*.com/ (Free with source)

8) CormanLisp http://www.*-*-*.com/ (Free with source) Cormanlisp is the
least complete and robust. Write-line and asin do not work. Loop macro
doesn't understand minimizing or of-type. Defstruct cannot do (:type
list). with-open-file :if-exists :append appears to :supersede
instead. Environment information functions are not implemented.

9) Gabriel benchmarks downloaded from
ftp://ftp.cs.umass.edu/pub/eksl/gabriel/. Benchmarks run with
(optimize (speed 3) (safety 1) (space 0)).

10) Primecount benchmarks also make use of a Lispworks nonANSI
declaration
(optimize (fixnum 0)).

11) Rhyme exercises read and write of ascii files along with string sort
and list reversal.

12) Primecountfixnum is a fixnum integer intensive computation.

13) Primecountbignum is a bignum integer intensive computation.

14) Pnpoly is a floating point intensive computation. Pnpoly is
optimized
for floating point using declarations and optimized for (speed 3) and
(safety 0). I believe that only Allegro CL and Lispworks pay any
attention. Unfortunately Lispworks also requires a (declare (optimize
(float 0))) to make any real gains as well. This is an nonANSI
feature. It is also not a very robust one. FFT is floating point
intensive but the use of (float 3) only came up with Lispworks
compiler errors.

15) Genericfunctions and slotvalues test generic function calling and
slot
value accessing. They were obtained from the CormanLisp distribution.

16) My source:

(defun rhyme (from to &aux dict)
  (declare (optimize (speed 3) (safety 0)))
     (with-open-file (in from :direction :input)
       (setq dict (loop for w = (read-line in nil nil)
                    until (not w) collect w)))
     (setq dict (mapc #'nreverse dict))
     (setq dict (sort dict #'string<))
     (setq dict (mapc #'nreverse dict))
     (with-open-file (out to :direction :output :if-exists :supersede)
       (dolist (x dict)
         #-cormanlisp (write-line x out)
         #+cormanlisp (progn (write-string x out) (terpri out))
         )))

(defun prime-count (s e)
  (declare (optimize (speed 3) (safety 0))
           #+Lispworks (optimize (fixnum-safety 3))
           (integer s e))
  (macrolet ((search-template (typeint)
               `(loop for n of-type ,typeint from (if (oddp x) x (1+
x)) to y by 2
                    for stop of-type ,typeint = (isqrt n)
                    with count of-type ,typeint = (if (<= x 2 y) 1 0) ;
to count 2
                    when (= n (loop for test-div of-type ,typeint from
3 to stop by 2
                                  when (zerop (mod n test-div)) do
(return test-div)
                                  finally (return n)))
                    do (incf count)
                    finally (print count))))
    (flet ((search-fixnum-range (x y) (search-template fixnum))
           (search-bignum-range (x y) (search-template integer)))
      (if (> e most-positive-fixnum)
          (search-bignum-range s e)
        (search-fixnum-range s e)))))

(defun pnpoly (npol xp yp x y)
  (declare (optimize (speed 3) (safety 0))
           #+Lispworks (optimize (float 0))
           (fixnum npol)
           (double-float x y)
           (type (simple-array double-float (*)) xp yp))
  (let* ((c nil)
         (j (1- npol)))
    (declare (fixnum j))
    (dotimes (i npol c)
      (declare (fixnum i))
      (if (and (or (and (<= (aref yp i) y) (< y (aref yp j)))
                   (and (<= (aref yp j) y) (< y (aref yp i))))
               (< x (+ (aref xp i) (/ (* (- (aref xp j) (aref xp i))
                                         (- y (aref yp i)))
                                      (- (aref yp j) (aref yp i))))))
          (setq c (not c)))
      (setq j i))))

(defun pnpolymain (npol div)
  (declare (optimize (speed 3) (safety 0))
           #+Lispworks (optimize (float 0))
           (fixnum npol div))
  (let* ((xp (make-array npol :element-type 'double-float))
         (yp (make-array npol :element-type 'double-float))
         (theta 0.0d0)
         (a 10.0d0)
         #+clisp (pi (float pi 0.0d0))
         (fdiv (/ (* 2 a) div))
         (count 0))
    (declare (double-float fdiv a theta)
             (fixnum count)
             (type (simple-array double-float (*)) xp yp))
    (dotimes (i npol)
      (declare (fixnum i))
      (setq theta (/ (* 2 i pi) npol))
      (setf (aref xp i) (+ a (* a (expt (cos theta) 3)))
            (aref yp i) (+ a (* a (expt (sin theta) 3)))))
    (dotimes (u (1+ div))
      (declare (fixnum u))
      (dotimes (v (1+ div))
        (declare (fixnum v))
        (if (pnpoly npol xp yp
                    #+Lispworks (* (coerce u 'double-float) fdiv)
                    #-Lispworks (* u fdiv)
                    #+Lispworks (* (coerce v 'double-float) fdiv)
                    #-Lispworks (* v fdiv)
                    )
            (incf count))))
    (format t "Area:  ~a" (/ (* count 4 a a) (* (1+ div) (1+ div))))))

(defclass foo () ())

(defclass bar () (a b c))

(defclass baz () (x y))

(defgeneric gf-1-1 (a))
(defmethod gf-1-1 ((a foo)))

(defgeneric gf-1-2 (a))
(defmethod gf-1-2 ((a foo)))
(defmethod gf-1-2 ((a symbol)))

(defgeneric gf-1-3 (a))
(defmethod gf-1-3 ((a integer)))
(defmethod gf-1-3 ((a bar)))
(defmethod gf-1-3 ((a t)))

(defgeneric gf-2-3 (a b))
(defmethod gf-2-3 ((a integer) (b bar)))
(defmethod gf-2-3 ((a foo) (b t)))
(defmethod gf-2-3 ((a t) (b integer)))

(defun bench-gfs ()
  (let ((foo (make-instance 'foo))
        (bar (make-instance 'bar))
        (baz (make-instance 'baz)))
    (dotimes (i 10000)
      (gf-1-1 foo)
      (gf-1-1 foo)
      (gf-1-2 foo)
      (gf-1-2 'quux)
      (gf-1-3 5)
      (gf-1-3 foo)
      (gf-2-3 5 bar)
      (gf-2-3 foo 'quux)
      (gf-2-3 baz 7))))

(defun bench-slot-value ()
  (let ((bar (make-instance 'bar))
        (baz (make-instance 'baz)))
    (dotimes (i 100000)
      (setf (slot-value bar 'a) 10)
      (setf (slot-value baz 'y) 20)
      (slot-value bar 'a)
      (setf (slot-value bar 'c) 777)
      (slot-value bar 'c)
      (slot-value baz 'y))))

(defmacro ttime (&rest body)

(defun testrhyme ()
  (print (ttime (rhyme "c:/personal/wattojd/benchmarks/gabriel/dict.txt"

"c:/personal/wattojd/benchmarks/gabriel/rhyme.txt"))))

(defun testprimecountfixnum ()
  (print (ttime (prime-count 0 200000))))

(defun testprimecountbignum ()
  (print (ttime (prime-count 1000000000050 1000000000100))))

(defun testpnpoly ()
  (print (ttime (pnpolymain 200 400))))

(defun testgenericfunctions ()
  (print (ttime (bench-gfs))))

(defun testslotvalues ()
  (print (ttime (bench-slot-value))))

--
John Watton
Alcoa Inc.

Sent via Deja.com http://www.*-*-*.com/
Before you buy.



Sun, 19 May 2002 03:00:00 GMT  
 Benchmark Results for CL's onWin32


Quote:

>A while ago I mentioned in a posting that I had done some benchmarking
>of Common Lisp's for Win32. I received a few emails requesting the
>results. So....here they are.

<snip>

Thanks for posting it. :-)  BTW, is there a benchmark for some unix
flavour?  I'd like to see how CMUCL performs. :-)

PS Is someone working on a win32 of cmucl? O:-)

//-----------------------------------------------
//      Fernando Rodriguez Romero
//
//      frr at mindless dot com
//------------------------------------------------



Mon, 20 May 2002 03:00:00 GMT  
 Benchmark Results for CL's onWin32

Quote:

> 9) Gabriel benchmarks downloaded from
> ftp://ftp.cs.umass.edu/pub/eksl/gabriel/. Benchmarks run with
> (optimize (speed 3) (safety 1) (space 0)).

We don't think these short benchmarks are a good indicator of system
performance, but if you're going to play around with them, you ought
try them with (SAFETY 0).

Quote:
> FFT is floating point intensive but the use of (float 3) only came
> up with Lispworks compiler errors.

Do you mean (FLOAT 0)?  Either way, we've not seen such errors.
Evidently, the version of FFT that you used didn't have declarations
that LW could use to avoid boxing; your version of PNPOLY seems to
have the right ones, so you could try something like that in FFT.
--
Pekka P. Pirinen, Adaptive Memory Management Group, Harlequin Limited
     "A man never speaks of himself without losing something.  What he says
in his disfavor is always believed but when he commends himself, he arouses
mistrust."   - Michel Montaigne


Mon, 20 May 2002 03:00:00 GMT  
 Benchmark Results for CL's onWin32



Quote:
> 14) Pnpoly is a floating point intensive computation. Pnpoly is
> optimized
> for floating point using declarations and optimized for (speed 3) and
> (safety 0). I believe that only Allegro CL and Lispworks pay any
> attention. Unfortunately Lispworks also requires a (declare (optimize
> (float 0))) to make any real gains as well. This is an nonANSI
> feature. It is also not a very robust one. FFT is floating point
> intensive but the use of (float 3) only came up with Lispworks
> compiler errors.

I've just had a look at the problem with FFT: it is a bug that only
manifests itself with float 0 & safety > 0, whch is an unusual set of
options.

For best performance, you should also turn off all debugging support.  In
LispWorks, thie means doing:

(proclaim '(optimize (debug 0)))
(toggle-source-debugging nil)

--
Martin Simmons, Lisp Group, Harlequin Limited



Mon, 20 May 2002 03:00:00 GMT  
 Benchmark Results for CL's onWin32

| For best performance, you should also turn off all debugging support.

  all of these benchmarks seem to be testing fairly unusual conditions.
  the arguments we see here for "tuning" seem to me to imply that we have
  to chuck all the Lispy features we can think of to get a comparison.  I
  think this is completely bogus.  I'm not going to trust _myself_ enough
  to run my code with the kinds of optimization settings people advocate
  for these benchmarks.  I _want_ a lot of dynamism, debugging support,
  etc, and I'm not usually ready to run my code in production without these
  things, except in particular functions that I have determined to need it,
  and which have been specially written to be blindingly fast.  however, in
  the latter case, I'd be pretty damn stupid if I thought I could have 100%
  portable ANSI Common Lisp code _and_ super-duper maximal performance at
  the same time.  _sheesh_, guys.  if I had such needs as you imply with
  these declarations for "best performance", I would even hack the compiler
  output and look for instruction scheduling opportunities myself.  using a
  special function to turn off debugging, for instance, is just _one_ such
  measure to squeeze the bejeezus out of the system, and we're already
  outside of the portable ANSI Common Lisp domain.

  so in my view, you either want super-fast code and profile it and get the
  optimal results with hand-tweaked code that may not work anywhere else,
  or you use fully portable code under _normal_ condtions.  anything else
  is very close to cheating, rendering the benchmark results evidence of
  how good you are at cheating at random in each implementation with no way
  of comparing your skills and then _nothing_ else.  if you let instead
  consciously and openly do your extreme best to write fast code for each
  implementation, and _then_ compare the results, you at the very least get
  comparable results for comparable _people_ (i.e., experts).

  I suggest such benchmarks limit themselves to the narrowest possible type
  declarations, the highest SPEED and the lowest DEBUG and SAFETY, and
  either full local tweaking or no local tweaking.  if a declaration isn't
  enough, that's a factor that you need to take into account when and if
  you engage in _full_ local tweaking, and not at all until then.

#:Erik



Mon, 20 May 2002 03:00:00 GMT  
 Benchmark Results for CL's onWin32
Okay, I reran things incorporating the suggestions of the two Harlequin
folks so that we are compiling with (optimize (speed 3) (safety 0)
(space 0) (debug 0)). Also specifically for Lispworks I set (toggle-
source-debugging nil) and I got the (declare (optimize (float 0))) to
work in FFT. I also incorporate a suggestion into PRIMECOUNTFIXNUM
which benefits ACL (the fixnum range is declared to be positive only).

I agree with Erik to some degree and it does feel like cheating when
going outside the ANSI standard to use declarations such as Lispworks
float and fixnum-safety and their toggle-source-debugging. Otherwise
sticking with the standard I don't feel like I am cheating at all and
in fact most of my (real life) applications are sensitive to floating
and fixnum arithmetic and as a consequence they are highly optimized
with declarations and I feel that I am quite good at making the most of
this approach with the Franz ACL product.

CL-TYPE           Allegro CL LispWorks Poplog CL CLISP      CormanLisp
CL-VER            5.0.1      4.1.18    2.0       1999-07-22 1.3
BOYER              1.177      3.889     1.399     5.017      1.000
BROWSE             1.000      4.791     2.512     5.791      4.077
CTAK               1.000      1.077     7.000     5.077      4.462
DDERIV             1.000      4.517     2.667     4.167      1.188
DERIV              1.000      4.736     2.170     4.321      1.131
DESTRU             1.000      5.214     5.643    13.643      3.831
DIV2               1.000      5.091     2.205     7.273      1.239
FFT                1.000      1.500    67.000    45.550     53.126
FPRINT             1.622      1.422     4.311     1.000      6.297
FREAD              1.478      5.870     1.217     1.000      8.897
FRPOLY             1.000      1.779     1.119     3.655      1.542
PUZZLE             1.452      1.452     1.000     5.006      5.577
STAK               1.444      1.000     3.556     6.481      6.057
TAK                1.000      1.500     3.500    21.025      1.853
TAKL               1.000      1.370     2.407    16.889      1.664
TAKR               1.000      2.300     2.100    10.000      1.592
TRAVERSE           1.000      2.824     3.926    15.336      6.767
TRIANG             1.531      1.000     3.210    18.219      9.504
RHYME              1.000      2.033     1.910     2.451     87.079
PRIMECOUNTFIXNUM   1.000      4.958     6.334    11.537     19.227
PRIMECOUNTBIGNUM   2.220      1.000     1.093     1.536     26.246
PNPOLY             1.000      1.902    32.779    43.533     41.890
GENERICFUNCTIONS   1.333      1.000     7.167     4.667     51.133
SLOTVALUES         1.156      1.000     1.682     1.966      6.105
TOTAL              1.000      2.225     5.909     8.979     12.370

--
John Watton
Alcoa Inc.

Sent via Deja.com http://www.deja.com/
Before you buy.



Tue, 21 May 2002 03:00:00 GMT  
 Benchmark Results for CL's onWin32

Quote:


>> 9) Gabriel benchmarks downloaded from
>> ftp://ftp.cs.umass.edu/pub/eksl/gabriel/. Benchmarks run with
>> (optimize (speed 3) (safety 1) (space 0)).

>We don't think these short benchmarks are a good indicator of system
>performance, but if you're going to play around with them, you ought
>try them with (SAFETY 0).

        What would you consider a good indicator of system performance
then?  What benchmarking would you use?

//-----------------------------------------------
//      Fernando Rodriguez Romero
//
//      frr at mindless dot com
//------------------------------------------------



Tue, 21 May 2002 03:00:00 GMT  
 Benchmark Results for CL's onWin32


Quote:
>I've just had a look at the problem with FFT: it is a bug that only
>manifests itself with float 0 & safety > 0, whch is an unusual set of
>options.

        What about boyer?

//-----------------------------------------------
//      Fernando Rodriguez Romero
//
//      frr at mindless dot com
//------------------------------------------------



Tue, 21 May 2002 03:00:00 GMT  
 Benchmark Results for CL's onWin32



Quote:
>   I suggest such benchmarks limit themselves to the narrowest possible
type
>   declarations, the highest SPEED and the lowest DEBUG and SAFETY, and
>   either full local tweaking or no local tweaking.  if a declaration
isn't
>   enough, that's a factor that you need to take into account when and if
>   you engage in _full_ local tweaking, and not at all until then.

In fact, given the insane number of combinations for OPTIMIZE decls, this
is a good strategy for all code, not just benchmarks.

Moreover, with the luxury of hindsight, some of the extra optimization
settings for LW should probably be controlled by the existing quality
names.

--
Martin Simmons, Lisp Group, Harlequin Limited



Tue, 21 May 2002 03:00:00 GMT  
 Benchmark Results for CL's onWin32

Quote:

> Okay, I reran things incorporating the suggestions of the two Harlequin
> folks so that we are compiling with (optimize (speed 3) (safety 0)
> (space 0) (debug 0)). Also specifically for Lispworks I set (toggle-
> source-debugging nil) and I got the (declare (optimize (float 0))) to
> work in FFT. I also incorporate a suggestion into PRIMECOUNTFIXNUM
> which benefits ACL (the fixnum range is declared to be positive only).

You might also try (COMPILATION-SPEED 0) to make the LispWorks compiler work
harder on larger bodies of code. The Gabriels probably sufficiently small for
this not to make much difference.

__Jason



Fri, 24 May 2002 03:00:00 GMT  
 
 [ 10 post ] 

 Relevant Pages 

1. (forwarding) Re: Benchmark Results for CL's onWin32

2. Allegro CL Benchmarks

3. MVCL Benchmark Results

4. Smalltalk vs C vs C++ benchmark results

5. VisualWorks benchmark results

6. RFC on Benchmark results

7. Toy loop/array benchmark yields surprising results.

8. Some benchmark results

9. latest benchmark results

10. Benchmark Results so far .....

11. Benchmark results so far ....

12. Benchmark Results so far .....

 

 
Powered by phpBB® Forum Software