benchmark cut for evaluation of dynamically created function 
Author Message
 benchmark cut for evaluation of dynamically created function

Hi all,

I'd like to make a small projection concerning performance of
the evaluation of dynamically created function in different
dynamic languages.

implementations of following languages are concidered:
Lisp family (including Scheme), Dylan, Python, C# with reflections(?),
<your language here>.

However I'd like to spent my time on mature enough
languages/implementations :)

Initial form of my report you could find here:
http://www.*-*-*.com/

Please send better implemented test of your languages in this
thread or directly at:

   khamenya AT mail DOT ru

some related discussions are:

http://www.*-*-*.com/
http://www.*-*-*.com/

Thank you.
Valery



Wed, 21 Dec 2005 22:14:10 GMT  
 benchmark cut for evaluation of dynamically created function

Quote:

> Hi all,

> I'd like to make a small projection concerning performance of
> the evaluation of dynamically created function in different
> dynamic languages.

[...]

Quote:

> Initial form of my report you could find here:
> http://khamenya.ru/dyn-lang-bench/

> Please send better implemented test of your languages in this
> thread

[...]

I don't have your setup (Windows XP, Pentium IV, 2.4 GHz, 1 GB Ram) but
rather Mac OS X, G3, 600 MHz, 384 MB Ram.

Here are two versions of the test case in Common Lisp, a straightforward
and an optimized version.

(defun run ()
  (time
   (eval
    '(labels ((fib (x)
                (if (< x 2)
                  1
                  (+
                   (fib (- x 2))
                   (fib (1- x))))))
       (fib 41)))))

(defun run-o ()
  (time
   (eval
    '(labels ((fib (x)
                (declare (dynamic-extent x)
                         (fixnum x)
                         (ftype (function (fixnum) fixnum) fib))
                (the fixnum
                  (if (< x 2)
                    1
                    (+ (fib (the fixnum (- x 2)))
                       (fib (the fixnum (1- x))))))))
       (fib 41)))))

I am not an expert in Common Lisp optimization, so I guess this code is
either funny or not optimal or both. But I have got the following
results with Macintosh Common Lisp 5.0:

? (progn (run) (run-o))

(eval '(labels ((fib (x) (if (< x 2) 1 (+ (fib (- x 2)) (fib (1- x))))))
(fib 41))) took 94,399 milliseconds (94.399 seconds) to run.
Of that, 49 milliseconds (0.049 seconds) were spent in The Cooperative
Multitasking Experience.
 7,656 bytes of memory allocated.

(eval '(labels ((fib (x)
                  (declare (dynamic-extent x) (fixnum x) (ftype
(function (fixnum) fixnum) fib))
                  (the fixnum (if (< x 2) 1 (+ (fib (the fixnum (- x
2))) (fib (the fixnum (1- x))))))))
         (fib 41))) took 63,715 milliseconds (63.715 seconds) to run.
Of that, 39 milliseconds (0.039 seconds) were spent in The Cooperative
Multitasking Experience.
 7,648 bytes of memory allocated.

267914296

The code should run on other CL implementations as well.

Pascal



Wed, 21 Dec 2005 23:53:04 GMT  
 benchmark cut for evaluation of dynamically created function

Quote:

> I'd like to make a small projection concerning performance of
> the evaluation of dynamically created function in different
> dynamic languages.

This is *not* what people in the dynamic languages community normally
refer to as creating a function at runtime.  See, for example, _On Lisp_
(http://www.paulgraham.com/onlisp.html), in which Paul Graham is quite
explicit that he's talking about closures, NOT "eval".

I offer the following code in Dylan, which *is* an example of what we
refer to as creating functions at runtime -- in fact it create a total
of 41 funtions at runtime:

-------------------------------------------------------
module: fib

define open generic fib(n);

define method fib(n :: <integer>)
  let result =
    if (n < 2)
      1
    else
      fib(n - 1) + fib(n - 2)
    end;
  add-method(fib, method(n == n) result end);
  result;
end;

format-out("fib 41 = %d\n", fib(41))
-------------------------------------------------------

fib 41 = 267914296

real    0m0.020s
user    0m0.020s
sys     0m0.000s
-------------------------------------------------------

This result obtained with Gwydion Dylan 2.3.10 on a 700 MHz Athlon.

-- Bruce



Thu, 22 Dec 2005 09:14:33 GMT  
 benchmark cut for evaluation of dynamically created function

Quote:



>> Hi all,

>> I'd like to make a small projection concerning performance of
>> the evaluation of dynamically created function in different
>> dynamic languages.
> [...]

>> Initial form of my report you could find here:
>> http://khamenya.ru/dyn-lang-bench/

>> Please send better implemented test of your languages in this
>> thread
> [...]

[...]

> (defun run ()
>   (time
>    (eval
>     '(labels ((fib (x)
>                 (if (< x 2)
>                   1
>                   (+
>                    (fib (- x 2))
>                    (fib (1- x))))))
>        (fib 41)))))

Just because you are constructing a function at runtime, doesn't mean
that you can't compile it...

(time
 (eval '(funcall (compile (defun fib (x)
                            (declare (optimize (speed 3) (safety 0) (debug 0))
                                     (type fixnum x))
                            (if (< x 2)
                                1
                                (+ (fib (- x 2))
                                   (fib (1- x))))))
                 41)))

With CMUCL on an Athlon XP 1700:

Evaluation took:
  11.12 seconds of real time
  11.12 seconds of user run time
  0.0 seconds of system run time
  0 page faults and
  184696 bytes consed.



Thu, 22 Dec 2005 09:25:00 GMT  
 benchmark cut for evaluation of dynamically created function

Quote:


> [...]
> With CMUCL on an Athlon XP 1700:

first of all thank you for your version of test posted.

and secondly, do you have win32 binary of CMUCL?

best regards,
Valery



Fri, 23 Dec 2005 03:32:18 GMT  
 benchmark cut for evaluation of dynamically created function

Quote:


> fib 41 = 267914296
> real    0m0.020s
> user    0m0.020s
> sys     0m0.000s
> -------------------------------------------------------

My first idea was "aha, irrelevant output, probably
because of ...".

Then I have installed dylan in order to run this test
myself.

Nice, the result is really so quick :)

Now I know what do you mean saying:

Quote:
> I offer the following code in Dylan, which *is* an example of
> what we refer to as creating functions at runtime -- in fact
> it create a total of 41 funtions at runtime:

:)

Cool !

Hm, should I take another test where calculated results
can't be reused anymore? otherwise we have a gold
prizer already. Or, maybe, just dylan should be
disqualified for over-performance using cheats?
What a funny weekend :)

best regards,
Valery



Fri, 23 Dec 2005 04:28:40 GMT  
 benchmark cut for evaluation of dynamically created function

Quote:

> I offer the following code in Dylan, which *is* an example of what we
> refer to as creating functions at runtime -- in fact it create a total
> of 41 funtions at runtime:

Common Lisp transcription:

(defgeneric fib (n))

(defmethod fib ((n integer))
  (let ((result
         (if (< n 2)
             1
             (+ (fib (- n 1)) (fib (- n 2))))))
    (defmethod fib ((n (eql n))) result)
    result))

(format t "~&fib 41 = ~D~%" (fib 41))



Fri, 23 Dec 2005 05:58:03 GMT  
 benchmark cut for evaluation of dynamically created function

Quote:

> and secondly, do you have win32 binary of CMUCL?

Sorry, CMUCL is unix-only.  The code should work in other
implementations of Common Lisp, though.

Iain



Fri, 23 Dec 2005 09:37:09 GMT  
 benchmark cut for evaluation of dynamically created function

Quote:

> Hm, should I take another test where calculated results
> can't be reused anymore? otherwise we have a gold
> prizer already.

If you're going to allow caching, then this Scheme version (and similar
versions in other languages) should be allowed:

(define fib
  (cache-unary-integer-function
    (lambda (n)
      (if (< n 2)
         1
         (+ (fib (- n 1)) (fib (- n 2)))))))

It "creates a function at runtime" in the same sense as the Dylan version
does (although it only creates a single function for fib).
'Cache-unary-integer-function' can be defined as:

(define (cache-unary-integer-function f)
  (let ((cache (make-vector 0 #f)))
    (lambda (n)
      (if (>= n (vector-length cache))
          (set! cache (vector-resize cache (+ n 1) #f)))
      (let ((x (vector-ref cache n)))
        (if x
            x
            (let ((x (f n)))
              (vector-set! cache n x)
              x))))))

This executes (fib 1000) in 7 milliseconds on a PIII-750 (and faster on
subsequent runs...)

Of course, none of this says anything about the speed of code that's
compiled and loaded at runtime (e.g. via eval), so perhaps it's time to
rethink exactly what it is you're trying to test.

Anton



Fri, 23 Dec 2005 09:48:15 GMT  
 benchmark cut for evaluation of dynamically created function

Quote:

> > Hm, should I take another test where calculated results
> > can't be reused anymore? otherwise we have a gold
> > prizer already.

> If you're going to allow caching, then this Scheme version (and similar
> versions in other languages) should be allowed:
> [...]

Or this Swindle version which is identical to the Dylan one:

  (defmethod (fib n)
    (let ((r (if (< n 2) 1 (+ (fib (- n 1)) (fib (- n 2))))))
      (add-method fib (method ((n = n)) r))
      r))

And obviously runs in a similar time frame (~50ms on a 400MHz
machine).

Quote:
> Of course, none of this says anything about the speed of code that's
> compiled and loaded at runtime (e.g. via eval), so perhaps it's time
> to rethink exactly what it is you're trying to test.

#t

--
          ((lambda (x) (x x)) (lambda (x) (x x)))          Eli Barzilay:
                  http://www.barzilay.org/                 Maze is Life!



Fri, 23 Dec 2005 10:36:57 GMT  
 benchmark cut for evaluation of dynamically created function

    > It "creates a function at runtime" in the same sense as the Dylan
    > version does (although it only creates a single function for fib).
    > 'Cache-unary-integer-function' can be defined as:

    > (define (cache-unary-integer-function f)
    >   (let ((cache (make-vector 0 #f)))
    >     (lambda (n)
    >       (if (>= n (vector-length cache))
    >           (set! cache (vector-resize cache (+ n 1) #f)))
    >       (let ((x (vector-ref cache n)))
    >         (if x
    >             x (let ((x (f n)))
    >               (vector-set! cache n x) x))))))

Where vector-resize could be defined as the following standard scheme:

(define (vector-resize vec size . args)
  (let ((new (apply make-vector size args)))
    (do ((i (- (vector-length vec) 1) (- i 1)))
        ((< i 0) new)
      (vector-set! new i (vector-ref vec i)))))

... but then you're better off using (vector-resize cache (* n 2) #f) or
similar to avoid excessive recopying.

--
Alex



Fri, 23 Dec 2005 11:11:25 GMT  
 benchmark cut for evaluation of dynamically created function

Quote:
> If you're going to allow caching, then this Scheme version (and similar
> versions in other languages) should be allowed:
> [...]

I guess vector-resize is not present in R5RS, so,
could you post full version of your example?

thank you!
best regards
Valery



Fri, 23 Dec 2005 17:09:04 GMT  
 benchmark cut for evaluation of dynamically created function

Quote:

> I guess vector-resize is not present in R5RS, so,

No, but easy to write with SRFI-43 as mentioned before!

Quote:
> could you post full version of your example?

(define vector-resize (lambda (v n fill)
  (let ((w (make-vector n fill)))
    (vector-copy! w 0 v) ; from SRFI-43
     w)))

(define (vector-copy! target tstart source . sstart+send)
  (let* ((target-len (vector-length target))
         (source-len (vector-length source))
         (sstart (if (pair? sstart+send)
                   (car sstart+send)
                   0))
         (send (if (and (pair? sstart+send)
                        (pair? (cdr sstart+send)))
                 (cadr sstart+send)
                 source-len)))
    (do ((to tstart (+ to 1))
         (from sstart (+ from 1)))
      ((>= from send))
      (vector-set! target to (vector-ref source from)))))

Please note that vector-copy! in the proposed reference implementation of
SRFI-43 contains an inconsistency between documentation and implementation.
I fixed the implementation so that send is taken as an exclusive index.
(Are you reading this, Taylor Campbell?)

Ciao
Sven



Fri, 23 Dec 2005 19:38:15 GMT  
 benchmark cut for evaluation of dynamically created function

+---------------

| > (defun run ()
| >   (time
| >    (eval
| >     '(labels ((fib (x)
| >                 (if (< x 2)
| >                   1
| >                   (+
| >                    (fib (- x 2))
| >                    (fib (1- x))))))
| >        (fib 41)))))
|
| Just because you are constructing a function at runtime, doesn't mean
| that you can't compile it...
|
| (time
|  (eval '(funcall (compile (defun fib (x)
|                           (declare (optimize (speed 3) (safety 0) (debug 0))
|                                    (type fixnum x))
|                           (if (< x 2)
|                               1
|                               (+ (fib (- x 2))
|                                  (fib (1- x))))))
|                41)))
|
| With CMUCL on an Athlon XP 1700:
|
| Evaluation took:
|   11.12 seconds of real time
|   11.12 seconds of user run time
|   0.0 seconds of system run time
|   0 page faults and
|   184696 bytes consed.
+---------------

I didn't see the original test code, but both of those look *awefully*
wasteful to this CL newbie. CMUCL's "time" is a macro, and automatically
compiles top-level forms anyway, so on my Athlon XP 1600+ (1.4 Gz) here's
what I get (with a few more declarations to get rid of the remaining
compiler warnings):

        cmu> (time (labels ((fib (x)
                              (declare (optimize (speed 3) (safety 0)
                                                 (debug 0))
                                       (type fixnum x))
                              (the fixnum
                                (if (< x 2)  
                                  1
                                  (+ (the fixnum (fib (- x 2)))
                                     (the fixnum (fib (1- x))))))))
                     (fib 41)))
        ; Compiling LAMBDA NIL:
        ; Compiling Top-Level Form:

        ; Evaluation took:
[*]==>       ;   9.75 seconds of real time
        ;   9.705533 seconds of user run time
        ;   0.021392 seconds of system run time
        ;   13,708,807,468 CPU cycles
        ;   0 page faults and
        ;   64 bytes consed.
        ;
        267914296
        cmu>

-Rob

-----

627 26th Avenue                 <URL:http://rpw3.org/>
San Mateo, CA 94403             (650)572-2607



Fri, 23 Dec 2005 20:23:11 GMT  
 benchmark cut for evaluation of dynamically created function

Quote:

> I guess vector-resize is not present in R5RS, so,
> could you post full version of your example?

Here's an updated version, with a standalone vector-resize function (doesn't
need SRFI-43) based on Alex Shinn's version.  It also implements Alex's
suggestion to base the cache vector size on (* n 2), to avoid excessive
copying of the vector.

(define (cache-unary-integer-function f)
  (let ((cache (make-vector 0 #f)))
    (lambda (n)
      (if (>= n (vector-length cache))
          (set! cache (vector-resize cache (* n 2) #f)))
      (let ((x (vector-ref cache n)))
        (if x
            x
            (let ((x (f n)))
              (vector-set! cache n x)
              x))))))

;; returns a copy of 'vec' grown or shrunk to the specified
;; size, with any new elements set to the value of 'init'
(define (vector-resize vec size . init)
  (let ((copy-size (min size (vector-length vec)))
        (new-vec (apply make-vector size init)))
    (do ((i 0 (+ i 1)))
        ((= i copy-size) new-vec)
      (vector-set! new-vec i (vector-ref vec i)))))

(define fib
  (cache-unary-integer-function
   (lambda (n)
     (if (< n 2)
         1
         (+ (fib (- n 1)) (fib (- n 2)))))))

;; --Anton



Fri, 23 Dec 2005 22:13:09 GMT  
 
 [ 21 post ]  Go to page: [1] [2]

 Relevant Pages 

1. benchmark cut for evaluation of dynamically created function

2. benchmark cut for evaluation of dynamically created function

3. Performance Evaluation and Benchmarks

4. HELP: creating Edit(Cut,Copy,Paste) PullDown Menu

5. Refering to Objects on a dynamically create window

6. Dragging, dynamically creating controls and window refresh

7. Removing Dynamically Created Controls

8. Dynamically creating controls

9. Deleting dynamically created controls

10. How are instances created dynamically?

11. how to dynamically create a merging MDI menu?

12. how do I create controls dynamically

 

 
Powered by phpBB® Forum Software