Coding favor 
Author Message
 Coding favor

Hi, I'm doing some benchmarking of Dylan, I'm wondering if anyone is
willing to whip up a Dylan version of this program for me. It's awfully
simple, any help earns the author credit during my seminar, and a couple
of cold beers on arrival in Perth, Australia. :-)

------- OOBench.java -------
/* Author:  Oliver White
   Date:    September 29, 1999
   Purpose: This program is for use in benchmarking against other OO
languages
*/

import java.util.Date;

public class OOBench
{
    int anInt;

    public void test()
    {
        Date start = new Date();
        for(int i = 0; i < 1000000; i++)
        {
            TestObject myObject = new TestObject(i);
            int anInt = myObject.getMember();
        }
        Date finish = new Date();
        long timeTaken = finish.getTime() - start.getTime();
        System.out.print(timeTaken);
    }

    public class TestObject
    {
        int member;

        public TestObject(int initialiser)
        {
            member = initialiser;
        }

        public int getMember()
        {
            return member;
        }
    }

    public static void main(String args[])
    {
        OOBench aBenchmarker = new OOBench();
        aBenchmarker.test();
    }

Quote:
}



Sun, 17 Mar 2002 03:00:00 GMT  
 Coding favor

Quote:

> Hi, I'm doing some benchmarking of Dylan, I'm wondering if anyone is
> willing to whip up a Dylan version of this program for me. It's awfully
> simple, any help earns the author credit during my seminar, and a couple
> of cold beers on arrival in Perth, Australia. :-)

Okay, so this works in HD 2.0 beta:

8<---------------------------------------------------------------

Module: dylan-user

define library bench
  use harlequin-dylan;
  export bench;
end library bench;

define module bench
  use harlequin-dylan;
  use simple-format;
end module bench;

8<---------------------------------------------------------------

Module: bench

define class <test-object> (<object>)
  constant slot member :: <integer>, init-keyword: member:;
end class;

define sealed domain make (subclass(<test-object>));
define sealed domain initialize (<test-object>);

define method test () => ()
  let (seconds, microseconds) =
  timing ()
    for (i :: <integer> from 0 below 1000000)
      let my-object :: <test-object> = make(<test-object>, member: i);
      let an-int :: <integer> = my-object.member;
    end for;
  end timing;
  format-out("%= seconds\n", seconds + microseconds * 1e-6);
end method;

begin
  test();
end;

8<---------------------------------------------------------------

BTW you have to be wary of benchmarks that might or might not induce GC or
which contains bits of code the compiler may ignore because they are not used.

__Jason



Mon, 18 Mar 2002 03:00:00 GMT  
 Coding favor

Quote:

>    Purpose: This program is for use in benchmarking against other OO
> languages

  ^^^^^^^^^

BTW you can't really benchmark languages, only implementations of those
languages.

__Jason



Mon, 18 Mar 2002 03:00:00 GMT  
 Coding favor



Quote:

> > Hi, I'm doing some benchmarking of Dylan, I'm wondering if anyone is
> > willing to whip up a Dylan version of this program for me. It's awfully
> > simple, any help earns the author credit during my seminar, and a couple
> > of cold beers on arrival in Perth, Australia. :-)

> Okay, so this works in HD 2.0 beta:

...

Incidentally, on my machine (a pathetic P200) the Dylan program runs in about
1.30 seconds on average and the Java program (using JDK 1.2.2 + Hotspot JIT)
runs in about 3.45 seconds on average. So the Dylan program is more than 2.5x
faster.

__Jason



Mon, 18 Mar 2002 03:00:00 GMT  
 Coding favor

Quote:

> Hi, I'm doing some benchmarking of Dylan, I'm wondering if anyone is
> willing to whip up a Dylan version of this program for me.

Jason already posted a Dylan version of the program, so I'll refrain
from doing so.  However, if you are giving a seminar talk about
different OO languages and present some benchmarks to compare them you
should think about the utility of benchmarking "different languages".

If you compare benchmark results, especially if you test high-level
languages, the speed of your benchmarks is very dependent on the
implementation of the language you use.  So you cannot claim that
language A is faster than language B, you can only state that this
implementation of language A is faster by a factor of X than that
implementation of language B for some benchmark on machine C.  If you
look at Scheme implementations you will see that this is a real
concern: In some benchmarks a globally optimizing compiler like
Jeffrey Mark Siskind's Stalin is more than 1000 times faster than
simpler implementations of the same language.  On other benchmarks the
difference is only a factor of two or even less.

Even if you stick to a single implementation each, it is almost
impossible to claim that one language is faster than another without
stating for which purpose.  If you do a lot of string processing, it
is probably hard to beat the performance of Perl scripts, however that
doesn't mean that Perl will be faster than fortran if you do a lot of
numerical calculations.

Of course, you also have to take into account at which level of
abstraction you can write programs in the different languages, you can
try to write the programs in a way that is "natural" for each language
or you can try to keep the code for different languages as similar as
possible, etc.

To put it in a nutshell: Benchmarking is hard, and you have to be
precise about specifying what the benchmark is supposed to show if you
want to get meaningful results.

Let's have a look at what the above benchmark does: It runs a loop
with an integer variable as index, in which you create an object, read
some slot and then discard the value.  So, what you are really
measuring is some combination of how fast the implementation allocates
a lot of small objects (all of the same size), how fast the garbage
collector works (if it runs at all) for this special case, how fast
you can access slots holding an integer and how much of the computation
you can do at compile time.

There are many things that influence the speed of this benchmark: The
value of the slot you access is an integer, which favors languages
like Java where ints are not objects.  So your benchmark might give
different results if the value you access is not a primitive type. You
might also get wildly different results depending on the heap size
that your application preallocates and on the strategy used by the
garbage collector.  A runtime system that tries to enlarge the heap
whenever possible might conceivably be very fast for this benchmark
but trash the system when it has to do the first collection or even
when the OS starts paging out old pages to disk.  I could go on, but I
think you get the idea.

I can show you a quite amusing example in Common Lisp (which is
considered to be a "slow" language by many (who don't know what they
are talking about)).  Your test program looks like this:

(defclass test-object ()
  ((member :reader get-member :initarg :member)))

(defmethod test ()
  (time
   (dotimes (i 1000000)
     (let* ((my-object (make-instance 'test-object :member i))
            (an-int (get-member my-object)))
       (declare (ignore an-int))))))

(define-compiler-macro test ()
  `(time
    ,(dotimes (i 1000000)
       (let* ((my-object (make-instance 'test-object :member i))
              (an-int (get-member my-object)))
         (declare (ignore an-int))))))

(defun main ()
  (test))

This program is 16 lines long, which is shorter than the Java version,
even if we don't count lines which contain only braces.  Now what is
the runtime (using ACL, but should be about the same with every Lisp
worth its salt)?

USER(15): (main)
; cpu time (non-gc) 0 msec user, 0 msec system
; cpu time (gc)     0 msec user, 0 msec system
; cpu time (total)  0 msec user, 0 msec system
; real time  0 msec
; space allocation:
;  0 cons cells, 0 symbols, -32 other bytes, 0 static bytes
NIL
USER(16):

Huh? No run-time at all? Can this be possible? Have I shown that Lisp
is infinitely faster than Java?

Yes, almost no runtime, yes it is possible and unfortunately not
(although in my experience the Lisp implementations I use are quite a
bit faster than Pizza and the kaffe Java VM).  

The run time is below the resolution of the timer because the
compiler-macro does all the computation performed in the program at
compile-time.  This means that there is nothing left to do at
run-time.  Actually this example would have been more convincing if
your program would have done something at runtime, like summing the
values of i:

(defclass test-object ()
  ((member :reader get-member :initarg :member)))

(defmethod test ()
  (let ((value 0))
    (time
     (dotimes (i 1000000)
       (let* ((my-object (make-instance 'test-object :member i))
              (an-int (get-member my-object)))
         (setf value (+ an-int value)))))
    value))

(define-compiler-macro test ()
  (let ((value 0))
    `(progn
       (time
        ,(dotimes (i 1000000)
           (let* ((my-object (make-instance 'test-object :member i))
                  (an-int (get-member my-object)))
             (setf value (+ an-int value)))))
       ,value)))

(defun main ()
  (test))

If you run this version of the program you get

USER(51): (main)
; cpu time (non-gc) 0 msec user, 0 msec system
; cpu time (gc)     0 msec user, 0 msec system
; cpu time (total)  0 msec user, 0 msec system
; real time  0 msec
; space allocation:
;  0 cons cells, 0 symbols, -32 other bytes, 0 static bytes
499999500000
USER(52):

Still all the work can be done at compile time.  The Java version
prints:


Value: 1783293664
Runtime: 3128

Oops, of course `int' is too short to hold the result of this
benchmark...

Oh, and just another note: The Lisp version of the original benchmark
without compiler-macro and with full debugging enabled is three times
faster than the Java code when run under kaffe on my machine:

USER(55): (test-1)
; cpu time (non-gc) 740 msec user, 0 msec system
; cpu time (gc)     270 msec user, 0 msec system
; cpu time (total)  1,010 msec user, 0 msec system
; real time  1,006 msec
; space allocation:
;  0 cons cells, 0 symbols, 39,999,968 other bytes, 0 static bytes
NIL
USER(56):



(Amazingly enough the Java version that does nothing is slower than
the one that performs the addition on my machine.)  The second Lisp
version (again no compiler-macro and full debugging) takes less than
two seconds:

USER(53): (test 0)
; cpu time (non-gc) 1,600 msec user, 20 msec system
; cpu time (gc)     340 msec user, 10 msec system
; cpu time (total)  1,940 msec user, 30 msec system
; real time  1,972 msec
; space allocation:
;  0 cons cells, 0 symbols, 63,213,528 other bytes, 0 static bytes
499999500000

Regards,

  Matthias



Mon, 18 Mar 2002 03:00:00 GMT  
 Coding favor

Quote:
> Hi, I'm doing some benchmarking of Dylan, I'm wondering if anyone is
> willing to whip up a Dylan version of this program for me. It's awfully
> simple, any help earns the author credit during my seminar, and a couple
> of cold beers on arrival in Perth, Australia. :-)

I may be wrong, but I couldn't find a post of this benchmark comparison in
comp.lang.eiffel
see what they have to say.

Maxwell Sayles



Mon, 18 Mar 2002 03:00:00 GMT  
 Coding favor



Quote:



> > > Hi, I'm doing some benchmarking of Dylan, I'm wondering if anyone is
> > > willing to whip up a Dylan version of this program for me. It's awfully
> > > simple, any help earns the author credit during my seminar, and a couple
> > > of cold beers on arrival in Perth, Australia. :-)

> > Okay, so this works in HD 2.0 beta:

> ...

> Incidentally, on my machine (a pathetic P200) the Dylan program runs in about
> 1.30 seconds on average and the Java program (using JDK 1.2.2 + Hotspot JIT)
> runs in about 3.45 seconds on average. So the Dylan program is more than 2.5x
> faster.

Duh. I made a mistake in the profiling. My JDK 1.2.2 itself didn't have
Hotspot in it. I'd installed it only in the JRE. With Hotspot, the Java
program now seems to run in about 0.7 seconds, nearly 2x faster than the Dylan
program. :-(

__Jason



Tue, 19 Mar 2002 03:00:00 GMT  
 Coding favor
I just want to say what an excellent exposition this was. Bravo!

David McClain

Quote:


>> Hi, I'm doing some benchmarking of Dylan, I'm wondering if anyone is
>> willing to whip up a Dylan version of this program for me.

>Jason already posted a Dylan version of the program, so I'll refrain
>from doing so.  However, if you are giving a seminar talk about
>different OO languages and present some benchmarks to compare them you
>should think about the utility of benchmarking "different languages".

>If you compare benchmark results, especially if you test high-level
>languages, the speed of your benchmarks is very dependent on the
>implementation of the language you use.  So you cannot claim that
>language A is faster than language B, you can only state that this
>implementation of language A is faster by a factor of X than that
>implementation of language B for some benchmark on machine C.  If you
>look at Scheme implementations you will see that this is a real
>concern: In some benchmarks a globally optimizing compiler like
>Jeffrey Mark Siskind's Stalin is more than 1000 times faster than
>simpler implementations of the same language.  On other benchmarks the
>difference is only a factor of two or even less.

>Even if you stick to a single implementation each, it is almost
>impossible to claim that one language is faster than another without
>stating for which purpose.  If you do a lot of string processing, it
>is probably hard to beat the performance of Perl scripts, however that
>doesn't mean that Perl will be faster than Fortran if you do a lot of
>numerical calculations.

>Of course, you also have to take into account at which level of
>abstraction you can write programs in the different languages, you can
>try to write the programs in a way that is "natural" for each language
>or you can try to keep the code for different languages as similar as
>possible, etc.

>To put it in a nutshell: Benchmarking is hard, and you have to be
>precise about specifying what the benchmark is supposed to show if you
>want to get meaningful results.

>Let's have a look at what the above benchmark does: It runs a loop
>with an integer variable as index, in which you create an object, read
>some slot and then discard the value.  So, what you are really
>measuring is some combination of how fast the implementation allocates
>a lot of small objects (all of the same size), how fast the garbage
>collector works (if it runs at all) for this special case, how fast
>you can access slots holding an integer and how much of the computation
>you can do at compile time.

>There are many things that influence the speed of this benchmark: The
>value of the slot you access is an integer, which favors languages
>like Java where ints are not objects.  So your benchmark might give
>different results if the value you access is not a primitive type. You
>might also get wildly different results depending on the heap size
>that your application preallocates and on the strategy used by the
>garbage collector.  A runtime system that tries to enlarge the heap
>whenever possible might conceivably be very fast for this benchmark
>but trash the system when it has to do the first collection or even
>when the OS starts paging out old pages to disk.  I could go on, but I
>think you get the idea.

>I can show you a quite amusing example in Common Lisp (which is
>considered to be a "slow" language by many (who don't know what they
>are talking about)).  Your test program looks like this:

>(defclass test-object ()
>  ((member :reader get-member :initarg :member)))

>(defmethod test ()
>  (time
>   (dotimes (i 1000000)
>     (let* ((my-object (make-instance 'test-object :member i))
>     (an-int (get-member my-object)))
>       (declare (ignore an-int))))))

>(define-compiler-macro test ()
>  `(time
>    ,(dotimes (i 1000000)
>       (let* ((my-object (make-instance 'test-object :member i))
>       (an-int (get-member my-object)))
> (declare (ignore an-int))))))

>(defun main ()
>  (test))

>This program is 16 lines long, which is shorter than the Java version,
>even if we don't count lines which contain only braces.  Now what is
>the runtime (using ACL, but should be about the same with every Lisp
>worth its salt)?

>USER(15): (main)
>; cpu time (non-gc) 0 msec user, 0 msec system
>; cpu time (gc)     0 msec user, 0 msec system
>; cpu time (total)  0 msec user, 0 msec system
>; real time  0 msec
>; space allocation:
>;  0 cons cells, 0 symbols, -32 other bytes, 0 static bytes
>NIL
>USER(16):

>Huh? No run-time at all? Can this be possible? Have I shown that Lisp
>is infinitely faster than Java?

>Yes, almost no runtime, yes it is possible and unfortunately not
>(although in my experience the Lisp implementations I use are quite a
>bit faster than Pizza and the kaffe Java VM).

>The run time is below the resolution of the timer because the
>compiler-macro does all the computation performed in the program at
>compile-time.  This means that there is nothing left to do at
>run-time.  Actually this example would have been more convincing if
>your program would have done something at runtime, like summing the
>values of i:

>(defclass test-object ()
>  ((member :reader get-member :initarg :member)))

>(defmethod test ()
>  (let ((value 0))
>    (time
>     (dotimes (i 1000000)
>       (let* ((my-object (make-instance 'test-object :member i))
>       (an-int (get-member my-object)))
> (setf value (+ an-int value)))))
>    value))

>(define-compiler-macro test ()
>  (let ((value 0))
>    `(progn
>       (time
> ,(dotimes (i 1000000)
>    (let* ((my-object (make-instance 'test-object :member i))
>   (an-int (get-member my-object)))
>      (setf value (+ an-int value)))))
>       ,value)))

>(defun main ()
>  (test))

>If you run this version of the program you get

>USER(51): (main)
>; cpu time (non-gc) 0 msec user, 0 msec system
>; cpu time (gc)     0 msec user, 0 msec system
>; cpu time (total)  0 msec user, 0 msec system
>; real time  0 msec
>; space allocation:
>;  0 cons cells, 0 symbols, -32 other bytes, 0 static bytes
>499999500000
>USER(52):

>Still all the work can be done at compile time.  The Java version
>prints:


>Value: 1783293664
>Runtime: 3128

>Oops, of course `int' is too short to hold the result of this
>benchmark...

>Oh, and just another note: The Lisp version of the original benchmark
>without compiler-macro and with full debugging enabled is three times
>faster than the Java code when run under kaffe on my machine:

>USER(55): (test-1)
>; cpu time (non-gc) 740 msec user, 0 msec system
>; cpu time (gc)     270 msec user, 0 msec system
>; cpu time (total)  1,010 msec user, 0 msec system
>; real time  1,006 msec
>; space allocation:
>;  0 cons cells, 0 symbols, 39,999,968 other bytes, 0 static bytes
>NIL
>USER(56):



>(Amazingly enough the Java version that does nothing is slower than
>the one that performs the addition on my machine.)  The second Lisp
>version (again no compiler-macro and full debugging) takes less than
>two seconds:

>USER(53): (test 0)
>; cpu time (non-gc) 1,600 msec user, 20 msec system
>; cpu time (gc)     340 msec user, 10 msec system
>; cpu time (total)  1,940 msec user, 30 msec system
>; real time  1,972 msec
>; space allocation:
>;  0 cons cells, 0 symbols, 63,213,528 other bytes, 0 static bytes
>499999500000

>Regards,

>  Matthias



Tue, 19 Mar 2002 03:00:00 GMT  
 Coding favor
Dear Matthias,
  Well said, sir.  I recall your post from a few months (was it a
year?) ago on comp.lang.Smalltalk about dealing cards.  I enjoy reading
your remarks, and results, when a person wants to compare language
speeds.  Have you commented on the CN2 induction algorithm, developed
by Peter Clark and Tim Niblett?  (see
http://www.infinet.com/~btobin/perf.html).

-----

Are you working primarily with CL?  What other programming languages do
you favor?

Sincerely,
Douglas M. Auclair,
Loving Smalltalk, but also keeping my eyes on the functional world.

Sent via Deja.com http://www.deja.com/
Before you buy.



Sun, 24 Mar 2002 03:00:00 GMT  
 
 [ 12 post ] 

 Relevant Pages 

1. Favor

2. I have a favor to ask of you.

3. In favor of technology in schools

4. In favor of technology in schools

5. A silly favor, please.

6. Ayuda con Crystal Reports,urgente por favor

7. Ayuda con Crystal Reports,urgente por favor

8. Favor

9. Favored Implementations for Linux?

10. OSCON favor

11. Scheme Papers (A huge favor)

12. a favor

 

 
Powered by phpBB® Forum Software