BENCHMARKS: Tcl 7.4 -> Tcl 8.4a1 
Author Message
 BENCHMARKS: Tcl 7.4 -> Tcl 8.4a1

People have been interested in seeing benchmarks done for Tcl for
some time now.  After getting the initial kick from Laurent Duperval,
the core team finally sat down and commited to starting these test
suites.  While the code is a total rewrite, we'd like to thank him
for prodding us into gear on this, as the numbers are quite
educational.

The code for the benchmark suite is available in CVS now.  At the
moment, we only have done Tcl benchmarks, but we're hoping that
others will provide bench tests for things that they are interested
in.  There aren't any docs as of yet, but it's a fairly easy source
read (all Tcl).

The numbers should be used for comparative purposes only, but
you're welcome to use the suite for CPU based tests.  The attached
numbers were all done on x86 Solaris, with each interpreter compiled
with -O (no -g), the 7.x interps static, the 8.x interps dynamic.

--
   Jeffrey Hobbs                          The Tcl Guy
   jeffrey.hobbs at scriptics.com         Scriptics Corp.

[ 0509.txt 14K ]
Run benchmarks for 1:8.4a1 /home/hobbs/install/solaris-x86/bin/tclsh8.4
Run benchmarks for 2:8.3.1 /home/hobbs/install/solaris-x86/bin/tclsh8.3
Run benchmarks for 3:8.2.3 /home/hobbs/install/solaris-x86/bin/tclsh8.2
Run benchmarks for 4:8.1.1 /home/hobbs/install/solaris-x86/bin/tclsh8.1
Run benchmarks for 5:8.0.5 /home/hobbs/install/solaris-x86/bin/tclsh8.0
Run benchmarks for 6:7.6p2 /home/hobbs/install/solaris-x86/bin/tclsh7.6
Run benchmarks for 7:7.4 /home/hobbs/install/solaris-x86/bin/tclsh7.4
                                        1:8.4a1 2:8.3.1 3:8.2.3 4:8.1.1 5:8.0.5 6:7.6p2 7:7.4
001) VAR set scalar                           4       5       4       4       4      12      11
002) VAR set array element                   10      10       8      10       8      14      13
003) VAR multiple 'set's                    101     103     105     107     113     320     315
004) VAR single 'array set'                 208     210     208     211     174     354     349
005) CATCH without error condition            4       4       4       4       4      12      11
006) CATCH with error condition              56      51      55      50      43      28      27
007) CATCH command with no catch              4       4       4       4       3       8       8
008) IF if true numeric                       8       8       7       8       7      47      49
009) IF elseif true numeric                  10      11       9      10       9      60      56
010) IF else true numeric                    10      11       9       9       9      60      56
011) IF if true num/num                       9       9       8       8       7      49      48
012) IF if false num/num                      8       9       8       8       7      51      48
013) IF if false al/num                      14      14      12      13      10      90      82
014) IF if true al/num                       16      16      14      15      12      50      47
015) IF if false al/num                      16      16      14      15      12      51      47
016) IF if true al/al                        16      17      14      14      12      54      51
017) IF elseif true al/al                    22      23      20      22      18      70      65
018) IF else true al/al                      22      24      20      22      17      70      67
019) SWITCH first true                       34      30      32      31      22     117     110
020) SWITCH second true                      35      31      33      31      24     117     112
021) SWITCH ninth true                       42      36      40      35      28     119     115
022) SWITCH default true                     36      32      33      31      24     117     111
023) DATA create in a list                 3299    2943    3097    3421    2561    8936    8120
024) DATA create in an array               4023    4621    4397    4538    5621   11766   11090
025) DATA access in a list                 2589    2552    2520    2857    2051   57918   57060
026) DATA access in an array               3169    3084    2799    3292    2461    9677    8867
027) EVAL cmd eval in list obj var           18      17      53      51      42      21      20
028) EVAL cmd eval as list                   15      15      48      49      38      15      14
029) EVAL cmd eval as string                 40      37      50      51      40      17      17
030) EVAL cmd and mixed lists              2524    2341    5604    5175    3953     863     842
031) EVAL list cmd and mixed lists         2302    2321    5619    5163    3980     844     838
032) EVAL list cmd and pure lists           376     377    5668    5222    4026     846     844
033) EXPR unbraced                          139     138     139      17     105      93      96
034) EXPR braced                             17      17      16      15      14      99     100
035) EXPR inline                             19      18      18      18      14      46      44
036) EXPR one operand                         4       5       4       4       3      18      16
037) EXPR ten operands                        9       9       8       8       7      38      35
038) EXPR fifty operands                     28      28      27      27      24     128     119
039) EXPR incr with incr                      6       7       7       6       3      19      12
040) EXPR incr with expr                      4       4       4       4       3      20      19
041) STR/LIST length, obj shimmer           559     584     576     106      10     288     285
042) LIST length, pure list                   8       9       8       9       7     214     216
043) STR length of a LIST                     9       8       9      55       6      93      93
044) LIST exact search, first item           10      10       9       8       7     189     199
045) LIST exact search, middle item          22      18      19      18      15     206     202
046) LIST exact search, last item            47      42      47      44      35     218     217
047) LIST exact search, non-item             75      66      68      66      58     229     224
048) LIST sorted search, first item          13      10       9       8       7     189     191
049) LIST sorted search, middle item         12      18      20      18      15     206     203
050) LIST sorted search, last item           13      42      45      44      36     220     216
051) LIST sorted search, non-item            13      65      68      64      57     227     225
052) LIST exact search, untyped item         47      42      45      43      36     240     216
053) LIST exact search, typed item           33      42      44      42      35     218     218
054) LIST sorted search, typed item          11      42      44      44      34     223     217
055) LIST sort                              653     660     652     660     588     900     898
056) LIST typed sort                        536     525     511     498     382    2085    2053
057) LIST remove first element               39      39      42      41      34     116     111
058) LIST remove middle element              38      40      42      41      32     148     146
059) LIST remove last element                39      39      42      40      32     164     163
060) LIST replace first element              39      40      43      41      32     131     130
061) LIST replace middle element             40      39      42      46      31     153     152
062) LIST replace last element               39      41      42      41      31     169     178
063) LIST remove in mixed list               51     534     524      48      42     149     145
064) LIST replace in mixed list              51     534     528      49      40     154     152
065) LIST index first element                 8       8       7       8       7      84      84
066) LIST index middle element                8       8       7       9       7     105     105
067) LIST index last element                  8       8       7       7       7     126     124
068) LIST insert an item at start            37      41      43      42      25     130     124
069) LIST insert an item at middle           36      40      42      41      25     148     148
070) LIST insert an item at "end"                  36      39      43      42      23     181     181
071) LIST small, early range                 17      18      16      16      11      93      90
072) LIST small, late range                  18      17      16      17      10     125     165
073) LIST large, early range                 24      24      25      24      15     130     129
074) LIST large, late range                  23      23      24      25      15     141     139
075) LIST append to list                    131     125     123     128     115     130     123
076) LIST join list                         269     266     258     194     186     538     533
077) LOOP iterate list with for            1720    1734    1710    1732    1640   13517   12346
078) LOOP iterate list with foreach        1105    1118    1122    1155    1063    1851    1841
079) LOOP for                              2005    2024    1988    2030    1960   15229   14034
080) LOOP while                            2005    2030    1988    2015    1973   15111   14043
081) LOOP iterate string with for          6809    6606    6117   47789    4188   50785   49783
082) LOOP iterate string with for          2762    2791    2833    2633    2397    1876    1895
083) PROC explicit return                     6       6       5       6       5      32      13
084) PROC implicit return                     4       5       4       4       4      13      12
085) PROC heavily commented                   3       4       4       3       3     213     175
086) PROC do-nothing, no args                 3       3      23      22       4       2       2
087) PROC do-nothing, one arg                 3       4       4       3       3       6       5
088) PROC local links with global           978     981     988     983     955    2746    2679
089) PROC local links with upvar            840     861     862     845     858    2720    2658
090) PROC local links with variable         821     836     815     828     469    2726    2648
091) READ 595K file with gets            277914  276726  260758  233567  123450  243516  171742
092) READ 595K file with read             56415   59596   54377   58773  347506  340409   46706
093) READ 595K file with read & size          55546   57984   53553   58531   19888  338366   44302
094) READ 3050b file with gets             1546    1503    1584    1288     740    1341    1003
095) READ 3050b file with read              390     398     378     390     203     204     229
096) READ 3050b file with read & size           419     452     414     428     239     255     254
097) REGEXP literal regexp                   32      32      31      35      19      22      21
098) REGEXP var-based regexp                 34      33      32      35      20      27    
...

read more »



Sat, 26 Oct 2002 03:00:00 GMT  
 BENCHMARKS: Tcl 7.4 -> Tcl 8.4a1
How do you benchmark features not present in a Tcl version?, as for
example [string equal] in Tcl 7.4 among many others ...

(sorry, no CVS setup)

Juan Carlos---



Sun, 27 Oct 2002 03:00:00 GMT  
 BENCHMARKS: Tcl 7.4 -> Tcl 8.4a1

Quote:

> How do you benchmark features not present in a Tcl version?, as for
> example [string equal] in Tcl 7.4 among many others ...

In the version I was working on (before Jeff et al did their work) I'd
return -1. Not necessarily the brightest idea but...

Quote:
> (sorry, no CVS setup)

Me either. Can someone send me a copy? Before you do, though, send me an
email to make sure I haven't received it yet.

Also, the module isn't listed on the CVS page.

L

--
Laurent Duperval                 "Montreal winters are an intelligence test,

                                                               -Doug Camilli
Penguin Power!         ***Nothing I say reflects the views of my employer***



Sun, 27 Oct 2002 03:00:00 GMT  
 BENCHMARKS: Tcl 7.4 -> Tcl 8.4a1

Quote:

>...
>   --------------------------------------------------------------------------------
> Run benchmarks for 1:8.4a1 /home/hobbs/install/solaris-x86/bin/tclsh8.4
> Run benchmarks for 2:8.3.1 /home/hobbs/install/solaris-x86/bin/tclsh8.3
> Run benchmarks for 3:8.2.3 /home/hobbs/install/solaris-x86/bin/tclsh8.2
> Run benchmarks for 4:8.1.1 /home/hobbs/install/solaris-x86/bin/tclsh8.1
> Run benchmarks for 5:8.0.5 /home/hobbs/install/solaris-x86/bin/tclsh8.0
> Run benchmarks for 6:7.6p2 /home/hobbs/install/solaris-x86/bin/tclsh7.6
> Run benchmarks for 7:7.4 /home/hobbs/install/solaris-x86/bin/tclsh7.4
>                                         1:8.4a1 2:8.3.1 3:8.2.3 4:8.1.1 5:8.0.5 6:7.6p2 7:7.4
> ...
> 117) STR map                               5080    7224    7201    6643    1538    3237    3349
> 118) STR map -nocase                       6396   10668   10414   10017    1832    3694    3771
> ...

I'm a little confused.  How did you time [string map] in 7.4 when it was added
at 8.1?

                                              Chris
--
Kids need dads!              http://www.crisny.org/not-for-profit/franys
Sign the petition:    http://childrens-justice.org/petition-4-online.htm



Sun, 27 Oct 2002 03:00:00 GMT  
 BENCHMARKS: Tcl 7.4 -> Tcl 8.4a1

Quote:
> > How do you benchmark features not present in a Tcl version?, as for
> > example [string equal] in Tcl 7.4 among many others ...

I believe Jeff and Eric wrote tcl only equivalents (what you had to do
in
tcl prior to the command being added) to do the same thing so that it
could
be tested how fast certain operations were in all versions going back to
7.4.

For [string equal] that would be ![string compare] I believe.

--Dan



Sun, 27 Oct 2002 03:00:00 GMT  
 BENCHMARKS: Tcl 7.4 -> Tcl 8.4a1

Quote:

> I'm a little confused.  How did you time [string map] in 7.4 when it was added
> at 8.1?

>                                               Chris

Jeff and Eric wrote tcl-only equivalents for commands that were added
between 7.4 and 8.1
so that the benchmarks would work.

For string map I think it is:

if { [catch {string map [list a b] "abc"}] } {
    proc str-map {str {nocase 0}} {
        if {$nocase} {
            regsub -nocase -all A $str at str
            regsub -nocase -all 0123 $str 0 str
        } else {
            regsub -all a $str at str
            regsub -all 0123 $str num str
        }
        return $str
    }

Quote:
} else {

    proc str-map {str {nocase 0}} {
        if {$nocase} {
            set str [string map -nocase {A at 0123 0} $str]
        } else {
            set str [string map {a at 0123 0} $str]
        }
        return $str
    }

Quote:
}

then in the benchmark suite they call str-map instead of [string map]


Sun, 27 Oct 2002 03:00:00 GMT  
 BENCHMARKS: Tcl 7.4 -> Tcl 8.4a1

Quote:
> Jeff and Eric wrote tcl-only equivalents for commands that were added
> between 7.4 and 8.1
> so that the benchmarks would work.

Cool! Can't wait to try'em (hint! hint!).

L

--
Laurent Duperval                 "Montreal winters are an intelligence test,

                                                               -Doug Camilli
Penguin Power!         ***Nothing I say reflects the views of my employer***



Sun, 27 Oct 2002 03:00:00 GMT  
 BENCHMARKS: Tcl 7.4 -> Tcl 8.4a1

Quote:


        ...
> > Run benchmarks for 1:8.4a1 /home/hobbs/install/solaris-x86/bin/tclsh8.4
> > Run benchmarks for 2:8.3.1 /home/hobbs/install/solaris-x86/bin/tclsh8.3
> > Run benchmarks for 3:8.2.3 /home/hobbs/install/solaris-x86/bin/tclsh8.2
> > Run benchmarks for 4:8.1.1 /home/hobbs/install/solaris-x86/bin/tclsh8.1
> > Run benchmarks for 5:8.0.5 /home/hobbs/install/solaris-x86/bin/tclsh8.0
> > Run benchmarks for 6:7.6p2 /home/hobbs/install/solaris-x86/bin/tclsh7.6
> > Run benchmarks for 7:7.4 /home/hobbs/install/solaris-x86/bin/tclsh7.4
> >                      1:8.4a1 2:8.3.1 3:8.2.3 4:8.1.1 5:8.0.5 6:7.6p2 7:7.4
> > ...
> > 117) STR map            5080    7224    7201    6643    1538    3237    3349
> > 118) STR map -nocase    6396   10668   10414   10017    1832    3694    3771
> > ...

> I'm a little confused.  How did you time [string map] in 7.4 when it was added
> at 8.1?

This relates to Laurent's question of how I did [string equal] as well, and
I should have addressed it before.  Dan Kuchler actually answered correctly,
in that Eric and I developed tests that had the same semantic equivalence
across versions.

What was a pure string based equal in 7.4-8.1.0 is:
        proc str-eq {a b} {
                if {![string compare $a $b]} { set i 1 } else { set i 0 }
        }
becomes for 8.1.1-8.3.1:
        proc str-eq {a b} {
                if {[string equal $a $b]} { set i 1 } else { set i 0 }
        }
becomes (hey hey, new magic) in 8.4a1:
        proc str-eq {a b} {
                if {$a eq $b} { set i 1 } else { set i 0 }
        }
(and the last one is 100% byte-compiled).

For string map, it was simulated for 8.1.0- as a regexp.  In fact, it's not
quite fair for the test above, because it seems that string map is slower,
but in fact it's twice as fast the regexp equivalent in 8.1+, but we actually
need a different test for that.

There are other similar algorithmic benchmarks, for example where regexp -start
is used vs. regexp with string range (at least I think that's in there).

Anyway, this is just a start, and has a ways to go.  Many of the significant
speedups you'll see in the 8.4a1 column are there because of the eye-opening
numbers we received from some of the tests.  My goal is to bring 90% of the
common case test times for 8.4 to within the range of, or better than any
previous version.  I don't know that I'll be able to do it, but I've got lots
of things I plan on trying.

Anyone with ideas or a few hours hacking time to spare, please tell me.

--
   Jeffrey Hobbs                          The Tcl Guy
   jeffrey.hobbs at scriptics.com         Scriptics Corp.



Sun, 27 Oct 2002 03:00:00 GMT  
 BENCHMARKS: Tcl 7.4 -> Tcl 8.4a1

Jeffrey,

I am not an expert in the field of benchmarking.

What is the timescale that you use here? Relative or Absolute?

Stefan

--
________________________________________________
|_______________________________________________|

        DI Stefan Weichselbaum
        Software Engineer

        AUSTRIAN AEROSPACE GmbH
        Stachegasse 16
        A-1120 Vienna, AUSTRIA / EUROPE
        Internet: http://www.space.at

        Tel.  : +43-1-80199/5594
        Fax   : +43-1-80199/5577

________________________________________________
        This email is for information only!
________________________________________________
|_______________________________________________|

  stefan.weichselbaum.vcf
< 1K Download


Tue, 29 Oct 2002 03:00:00 GMT  
 BENCHMARKS: Tcl 7.4 -> Tcl 8.4a1

Quote:
> Jeffrey,

> I am not an expert in the field of benchmarking.

> What is the timescale that you use here? Relative or Absolute?

Uhm... He parses the results of the "time" command, which are absolute, I
believe.

L

--
Laurent Duperval                 "Montreal winters are an intelligence test,

                                                               -Doug Camilli
Penguin Power!         ***Nothing I say reflects the views of my employer***



Tue, 29 Oct 2002 03:00:00 GMT  
 BENCHMARKS: Tcl 7.4 -> Tcl 8.4a1

Quote:

> What is the timescale that you use here? Relative or Absolute?

The numbers are specifically the microseconds returned by time,
but it's best to look at them as relative across versions.
Different machines will give different results of course, but
in many cases, when the thing being timed is so quick (like an
'if'), I actually have to repeat the same call several times so
that I get a more accurate time count out of it.

--
   Jeffrey Hobbs                          The Tcl Guy
   jeffrey.hobbs at scriptics.com         Scriptics Corp.



Tue, 29 Oct 2002 03:00:00 GMT  
 BENCHMARKS: Tcl 7.4 -> Tcl 8.4a1

Quote:


> > What is the timescale that you use here? Relative or Absolute?

> The numbers are specifically the microseconds returned by time,
> but it's best to look at them as relative across versions.
> Different machines will give different results of course, but
> in many cases, when the thing being timed is so quick (like an
> 'if'), I actually have to repeat the same call several times so
> that I get a more accurate time count out of it.

It might be very useful to make the benchmarks more-or-less absolute.

I have bit of code from UNIX Review, Feb 1991, Vol 9, No. 2, pp38-48
which benchmarks a variety of C constructs.  It uses preprocessor macros
to run control structures and computations in a tight loop and count the
system ticks to generate an absolute value per operation.  This data is
useful for comparing machines, comparing compiler options, and even
comparing the various constructs (i.e., compare x*2 with x<<1).

The number of iterations for each benchmark is the same for all tests,
so relative comparisons between tests can be made.  You change "N" and
the system clock info at compile time.  You could probably set a similar
constant for all the benchmarks, so that a value of "82" meant the same
thing for all tests.

As it stantds, I *presume* that "CATCH without error condition" and
"CATCH with error condition" are run with the same number of iterations,
but I'm not sure I can compare those with the EVAL tests.

Bob
--

Mayo Foundation                                           (507) 284-2702
Rochester MN, 55905 USA   http://www.mayo.edu/sppdg/sppdg_home_page.html



Tue, 29 Oct 2002 03:00:00 GMT  
 
 [ 14 post ] 

 Relevant Pages 

1. mSQL tcl extension compatible with tcl > 7.4

2. Looking for Tcl 7.3->7.4 help

3. TCL C API 7.4 -> 8.0 problems

4. Tcl 7.4 to Tcl 8.4 porting issues

5. TCL 8.0 vs TCL 7.4

6. Change from TCL/TK(7.4,4.0) to TCL/TK(7.5,4.1)

7. regsub in tcl 7.3 and tcl 7.4

8. C interface changes between Tcl 7.4 and Tcl 7.5

9. Tknt40r1 (Tcl 7.4, Tk 4.0, Tcl-DP 3.3, Itcl 1.5, BLT 1.8, Tix 4.0)

10. ANNOUNCE: Tknt40r1beta2 (Tcl 7.4/Tk 4.0/Tcl-DP 3.2) for NT

11. safe-tcl / Tcl 7.4 / Tk 4.0

12. TCL 7.4 serial port

 

 
Powered by phpBB® Forum Software