Web Toolkit Benchmark 
Author Message
 Web Toolkit Benchmark

Alan Knight gets credit for doing the testing (and for being the
architect of the web toolkit!).  Here's some interesting results:

http://www.*-*-*.com/ :8080/CincomSmalltalkWiki/Web+Toolkit+B...



Wed, 26 May 2004 14:05:02 GMT  
 Web Toolkit Benchmark
After taking a quick look at the numbers I would have to say you are way off
base.  And your statement that VW can serve dynamic content faster than Apache
serves static content should make you realize it right there.  Beyond that I did
some test long ago in weaker equipment that show that Apache can process pearl
relays over 3 times faster than you have it serving static content here.  And of
course it served static content much faster than that.  You do realize that you
are saying that these apps are processing about 20 transactions a second. I think
that you will find that to be extremely low in the web serving world.
Quote:

> Alan Knight gets credit for doing the testing (and for being the
> architect of the web toolkit!).  Here's some interesting results:

> http://www.cincomsmalltalk.com:8080/CincomSmalltalkWiki/Web+Toolkit+B...



Thu, 27 May 2004 00:19:51 GMT  
 Web Toolkit Benchmark
Also that last thing you want happening in a test like this is to have the network
being a bottle neck and with a 10Mb wireless that sounds like a problem.
Quote:

> After taking a quick look at the numbers I would have to say you are way off
> base.  And your statement that VW can serve dynamic content faster than Apache
> serves static content should make you realize it right there.  Beyond that I did
> some test long ago in weaker equipment that show that Apache can process pearl
> relays over 3 times faster than you have it serving static content here.  And of
> course it served static content much faster than that.  You do realize that you
> are saying that these apps are processing about 20 transactions a second. I think
> that you will find that to be extremely low in the web serving world.


> > Alan Knight gets credit for doing the testing (and for being the
> > architect of the web toolkit!).  Here's some interesting results:

> > http://www.cincomsmalltalk.com:8080/CincomSmalltalkWiki/Web+Toolkit+B...



Thu, 27 May 2004 00:28:28 GMT  
 Web Toolkit Benchmark

Quote:
>Also that last thing you want happening in a test like this is to have the network
>being a bottle neck and with a 10Mb wireless that sounds like a problem.

You might have hit the nail on the head here. I did a bit of testing
in August on some heavy-duty servers hooked up with a 100Mbit switch,
and Apache soared way past VW when serving static content. However, VW
did produce extremely respectable results compared to other application
servers (I compared against Zope), and that's what counts most...

Alan, you may want to retry your tests using a 100Mbit network. While you're
at it, use 'ab' (Apache Benchmark) to be nice to Unix people - as it is part
of the Apache distribution, everyone who has Apache installed will have this
basic command-line benchmarking program and so will be able to reproduce your
tests.

--

GnuPG 1024D/E0989E8B 0016 F679 F38D 5946 4ECD  1986 F303 937F E098 9E8B



Thu, 27 May 2004 20:31:45 GMT  
 Web Toolkit Benchmark
Unfortunately, I only get to test with the equipment (and software) that I
have. I expected the network would be a bottleneck, and the load-generating
software that I have, while simple, cheap, and easy to use, probably isn't
the highest-performance out there. That's why I stuck to a purely relative
benchmark. If the network is a bottleneck, it should be equally a
bottleneck for both. I realize that 20 transactions per second is quite
low. For example, removing one factor, I did an earlier benchmark on a
slower VM version that was showing about 50/second with the same hardware
going over a non-wireless 10mb network (I didn't test Apache at that time).



Quote:
> Also that last thing you want happening in a test like this is to have
> the network being a bottle neck and with a 10Mb wireless that sounds
> like a problem.


>> After taking a quick look at the numbers I would have to say you are
>> way off base.  And your statement that VW can serve dynamic content
>> faster than Apache serves static content should make you realize it
>> right there.  Beyond that I did some test long ago in weaker equipment
>> that show that Apache can process pearl relays over 3 times faster
>> than you have it serving static content here.  And of course it served
>> static content much faster than that.  You do realize that you are
>> saying that these apps are processing about 20 transactions a second.
>> I think that you will find that to be extremely low in the web serving
>> world.


>> > Alan Knight gets credit for doing the testing (and for being the
>> > architect of the web toolkit!).  Here's some interesting results:

>> > http://www.cincomsmalltalk.com:8080/CincomSmalltalkWiki/Web+Toolkit+B
>> > enchmarks



Fri, 28 May 2004 00:16:40 GMT  
 Web Toolkit Benchmark
I'm certainly not making any claims that these results generalize to all
situations. They're the only two tests that I happened to run to do some
simple and not-at-all-rigorous benchmarking.

In your scenario I would think that the heavy-duty servers would make more
difference than the network. Most particularly, since Apache has multiple
processes running, it would be able to exploit SMP machines much better
than a single VW image. Using not-the-cheapest-available hardware and
network would probably help too. At the transaction rates I was looking at
I doubt we were saturating the network

I only have one machine running UNIX, so if I use ab I would need to run
the tests on the same machine running the server, which I think would not
be a good thing.


over to other servers which are currently dropping lots of messages. Best
to copy me in e-mail.



Quote:

>>Also that last thing you want happening in a test like this is to have
>>the network being a bottle neck and with a 10Mb wireless that sounds
>>like a problem.

> You might have hit the nail on the head here. I did a bit of testing
> in August on some heavy-duty servers hooked up with a 100Mbit switch,
> and Apache soared way past VW when serving static content. However, VW
> did produce extremely respectable results compared to other application
> servers (I compared against Zope), and that's what counts most...

> Alan, you may want to retry your tests using a 100Mbit network. While
> you're at it, use 'ab' (Apache Benchmark) to be nice to Unix people -
> as it is part of the Apache distribution, everyone who has Apache
> installed will have this basic command-line benchmarking program and so
> will be able to reproduce your tests.



Fri, 28 May 2004 00:25:29 GMT  
 Web Toolkit Benchmark

Quote:

> Unfortunately, I only get to test with the equipment (and software) that I
> have. I expected the network would be a bottleneck, and the load-generating
> software that I have, while simple, cheap, and easy to use, probably isn't
> the highest-performance out there. That's why I stuck to a purely relative
> benchmark. If the network is a bottleneck, it should be equally a
> bottleneck for both.

This is exacly the problem.  Even if Apache can handle a 1000, the network will
limit it to 20.  Network bottle necks are not proportional they are fixed.

I ran a test a while back on a weeker single process machine and apache was
doing 67 per second dynamic content and vw was doing 5 per second.

BTW webBench from zdnet is free and pretty good.

Also keep in mind that when running vw in the real world it will every
transaction will probally have to go though apache first.

Quote:
> I realize that 20 transactions per second is quite
> low. For example, removing one factor, I did an earlier benchmark on a
> slower VM version that was showing about 50/second with the same hardware
> going over a non-wireless 10mb network (I didn't test Apache at that time).



> > Also that last thing you want happening in a test like this is to have
> > the network being a bottle neck and with a 10Mb wireless that sounds
> > like a problem.


> >> After taking a quick look at the numbers I would have to say you are
> >> way off base.  And your statement that VW can serve dynamic content
> >> faster than Apache serves static content should make you realize it
> >> right there.  Beyond that I did some test long ago in weaker equipment
> >> that show that Apache can process pearl relays over 3 times faster
> >> than you have it serving static content here.  And of course it served
> >> static content much faster than that.  You do realize that you are
> >> saying that these apps are processing about 20 transactions a second.
> >> I think that you will find that to be extremely low in the web serving
> >> world.


> >> > Alan Knight gets credit for doing the testing (and for being the
> >> > architect of the web toolkit!).  Here's some interesting results:

> >> > http://www.cincomsmalltalk.com:8080/CincomSmalltalkWiki/Web+Toolkit+B
> >> > enchmarks



Fri, 28 May 2004 01:03:32 GMT  
 Web Toolkit Benchmark

Quote:
>Also keep in mind that when running vw in the real world it will every
>transaction will probally have to go though apache first.

Given my experience with the benchmarks I posted elsewhere, I would advise
against this for a high-performance solution.

--

GnuPG 1024D/E0989E8B 0016 F679 F38D 5946 4ECD  1986 F303 937F E098 9E8B



Fri, 28 May 2004 04:23:08 GMT  
 Web Toolkit Benchmark

Quote:
>I'm certainly not making any claims that these results generalize to all
>situations. They're the only two tests that I happened to run to do some
>simple and not-at-all-rigorous benchmarking.

Oh, you put disclaimers up and stuff, but surely you know that people don't
read these, so by publicizing your stuff (or letting James do it) your tests
are fair game ;-)

Quote:
>In your scenario I would think that the heavy-duty servers would make more
>difference than the network. Most particularly, since Apache has multiple
>processes running, it would be able to exploit SMP machines much better
>than a single VW image.

Let me produce some figures. Client in both cases is a dual 800/PIII connected
through 100Mbit (switched) to the servers. The same document was used,
document size was 247 bytes in both cases.

1. Apache running on an IBM pizzabox (933/PIII, local SCSI drive)

Concurrency Level:      100
Time taken for tests:   9.629 seconds
Complete requests:      5000
Failed requests:        0
Total transferred:      2878752 bytes
HTML transferred:       1240928 bytes
Requests per second:    519.26
Transfer rate:          298.97 kb/s received

---

Concurrency Level:      500
Time taken for tests:   9.119 seconds
Complete requests:      5000
Failed requests:        0
Total transferred:      2877033 bytes
HTML transferred:       1240187 bytes
Requests per second:    548.31
Transfer rate:          315.50 kb/s received

2. VisualWorks running on an IBM server (Dual 733/XeonIII, RAID-5),
   file 'test.html', debugging switched off

Concurrency Level:      100
Time taken for tests:   17.103 seconds
Complete requests:      5000
Failed requests:        65
   (Connect: 0, Length: 65, Exceptions: 0)
Total transferred:      1870365 bytes
HTML transferred:       1218945 bytes
Requests per second:    292.35
Transfer rate:          109.36 kb/s received

----

Concurrency Level:      500
Time taken for tests:   17.062 seconds
Complete requests:      5000
Failed requests:        56
   (Connect: 0, Length: 56, Exceptions: 0)
Total transferred:      1873776 bytes
HTML transferred:       1221168 bytes
Requests per second:    293.05
Transfer rate:          109.82 kb/s received

3. VisualWorks again, but file is now called "test.ssp":

Concurrency Level:      100
Time taken for tests:   21.584 seconds
Complete requests:      5000
Failed requests:        54
   (Connect: 0, Length: 54, Exceptions: 0)
Total transferred:      3103858 bytes
HTML transferred:       1562936 bytes
Requests per second:    231.65
Transfer rate:          143.80 kb/s received

---

Concurrency Level:      500
Time taken for tests:   20.872 seconds
Complete requests:      5000
Failed requests:        449
   (Connect: 0, Length: 449, Exceptions: 0)
Total transferred:      2855985 bytes
HTML transferred:       1438116 bytes
Requests per second:    239.56
Transfer rate:          136.83 kb/s received

===
My conclusion: over the range of these tests, both seem to scale equally well,
although VW seems to have some trouble at high loads (I'd need to see what
these failed requests are). With a performance at 50% of Apache, I think VW
does extremely well as an application server (and quite fine as a general
webserver).

OBTW: these were direct connections. With the CGI adapter in between,
performance goes way down (waaaay down to about 25% of the direct figures, in
my tests)

Hth,

Cees

--

GnuPG 1024D/E0989E8B 0016 F679 F38D 5946 4ECD  1986 F303 937F E098 9E8B



Fri, 28 May 2004 04:20:54 GMT  
 Web Toolkit Benchmark
I am not sure there is a real alternative.  And if there is why is Cincom
spending so much time doing fast cgi relays?  Why have they always had cgi
relay and then perl relay?  And why does every deployement diagram for wave I
have seen show the webserver in front?
Quote:


> >Also keep in mind that when running vw in the real world it will every
> >transaction will probally have to go though apache first.

> Given my experience with the benchmarks I posted elsewhere, I would advise
> against this for a high-performance solution.

> --

> GnuPG 1024D/E0989E8B 0016 F679 F38D 5946 4ECD  1986 F303 937F E098 9E8B



Fri, 28 May 2004 06:40:58 GMT  
 Web Toolkit Benchmark

Quote:

> I am not sure there is a real alternative.  And if there is why is Cincom
> spending so much time doing fast cgi relays?  Why have they always had cgi
> relay and then perl relay?  And why does every deployement diagram for wave I
> have seen show the webserver in front?

The main reason we build this stuff and the main reasons diagrams show
it that way is expectations - that's the way most people expect to
deploy a web application server.
Quote:



>>>Also keep in mind that when running vw in the real world it will every
>>>transaction will probally have to go though apache first.

>>Given my experience with the benchmarks I posted elsewhere, I would advise
>>against this for a high-performance solution.

>>--

>>GnuPG 1024D/E0989E8B 0016 F679 F38D 5946 4ECD  1986 F303 937F E098 9E8B



Fri, 28 May 2004 06:59:56 GMT  
 Web Toolkit Benchmark

Quote:
>I am not sure there is a real alternative.

Well, somehow loads of consultants have been blabbering about 'x-tier' ('x'
being 'any', 'multi', '3', '4', or whatever) and showing a lot of PowerPoints
where the appserver sits behind the webserver - that's why most people seem to
want it. Technically, VW is quite capable of playing webserver itself, it
seems.

--

GnuPG 1024D/E0989E8B 0016 F679 F38D 5946 4ECD  1986 F303 937F E098 9E8B



Fri, 28 May 2004 18:10:06 GMT  
 Web Toolkit Benchmark


Quote:


>> Unfortunately, I only get to test with the equipment (and software)
>> that I have. I expected the network would be a bottleneck, and the
>> load-generating software that I have, while simple, cheap, and easy to
>> use, probably isn't the highest-performance out there. That's why I
>> stuck to a purely relative benchmark. If the network is a bottleneck,
>> it should be equally a bottleneck for both.

> This is exacly the problem.  Even if Apache can handle a 1000, the
> network will limit it to 20.  Network bottle necks are not proportional
> they are fixed.

If we were seeing a network bottleneck I would expect both VW and Apache to
reach a fixed peak value and go no higher. If one of the two is performing
faster in a particular test, then that indicates that (at least for the
slower one) it is not yet hitting a fixed network bottleneck.

Besides, network effects can occur in various different ways. For example,
I would assume that going through encryption is imposing a fixed overhead
on every request, but not affecting the maximum throughput.

It's possible that I can get a 100Mb card running on my Linux box, and if
so I'll give it a try in that configuration and borrow my wife's laptop
(which is better than the one I get from work) to run the tests.

Quote:
> I ran a test a while back on a weeker single process machine and apache
> was doing 67 per second dynamic content and vw was doing 5 per second.

Can you provide some more details. When did you do this test? "Dynamic
content" is a very broad term. I served a one-word static html file and had
a servlet which dynamically served that one word. This is a significantly
different test case than running a non-trivial SSP (which Cees did) and
radically different from running a VisualWave windowSpec. Also, we continue
to do VM performance tuning, and recent releases have included both
significant general performance boosts (5i.2 or 3) and server-specific
boosts (5i.4).

Could you run the test again and provide details and results. I'd be happy
to have them on the Wiki.

Quote:
> BTW webBench from zdnet is free and pretty good.

Thanks. I've downloaded it and I'll check it out.

Quote:
> Also keep in mind that when running vw in the real world it will every
> transaction will probally have to go though apache first.

Clearly. Although VW is capable of acting as the front-end web server, and
Smalltalk-centric shops may well do this (EZBoard does) we expect most
installations to run with another web server in front. Telling the web
administrator that step one of using this new programming language is to
replace Apache is not generally a good idea. If someone has a mixed-
language site, then a front-end server is needed to provide a single point
of coordination.

I am not attempting to suggest that people should replace Apache, or even
that VisualWorks is a better web-server. I simply wanted to do some
benchmarks to ensure that we were, at a first approximation, reasonably
competitive with other web-serving solutions. True benchmarks are quite
difficult to do and require much more detailed knowledge of the software
than I have of Apache. I also did not attempt to test a clustered situation
or a multi-processor situation for either piece of software, both of which
are much more important for scalability.

Quote:
>> I realize that 20 transactions per second is quite
>> low. For example, removing one factor, I did an earlier benchmark on a
>> slower VM version that was showing about 50/second with the same
>> hardware going over a non-wireless 10mb network (I didn't test Apache
>> at that time).



>> > Also that last thing you want happening in a test like this is to
>> > have the network being a bottle neck and with a 10Mb wireless that
>> > sounds like a problem.


>> >> After taking a quick look at the numbers I would have to say you
>> >> are way off base.  And your statement that VW can serve dynamic
>> >> content faster than Apache serves static content should make you
>> >> realize it right there.  Beyond that I did some test long ago in
>> >> weaker equipment that show that Apache can process pearl relays
>> >> over 3 times faster than you have it serving static content here.
>> >> And of course it served static content much faster than that.  You
>> >> do realize that you are saying that these apps are processing about
>> >> 20 transactions a second. I think that you will find that to be
>> >> extremely low in the web serving world.


>> >> > Alan Knight gets credit for doing the testing (and for being the
>> >> > architect of the web toolkit!).  Here's some interesting results:

>> >> > http://www.cincomsmalltalk.com:8080/CincomSmalltalkWiki/Web+Toolki
>> >> > t+B enchmarks



Sat, 29 May 2004 00:46:14 GMT  
 Web Toolkit Benchmark


Quote:
> I am not sure there is a real alternative.  And if there is why is
> Cincom spending so much time doing fast cgi relays?  Why have they
> always had cgi relay and then perl relay?  And why does every
> deployement diagram for wave I have seen show the webserver in front?

What's a perl relay?
Quote:



>> >Also keep in mind that when running vw in the real world it will
>> >every transaction will probally have to go though apache first.

>> Given my experience with the benchmarks I posted elsewhere, I would
>> advise against this for a high-performance solution.

>> --
>> Cees de Groot               http://www.cdegroot.com    

>> F303 937F E098 9E8B



Sat, 29 May 2004 00:48:00 GMT  
 Web Toolkit Benchmark


Quote:

>>I'm certainly not making any claims that these results generalize to
>>all situations. They're the only two tests that I happened to run to do
>>some simple and not-at-all-rigorous benchmarking.

> Oh, you put disclaimers up and stuff, but surely you know that people
> don't read these, so by publicizing your stuff (or letting James do it)
> your tests are fair game ;-)

Fair enough. I think they're quite rigorous by the standards of Java
benchmarks I've seen :-)  (I like the clause in the Java license that says
you're not allowed to publish benchmarks without Sun's permission).

Quote:
>>In your scenario I would think that the heavy-duty servers would make
>>more difference than the network. Most particularly, since Apache has
>>multiple processes running, it would be able to exploit SMP machines
>>much better than a single VW image.

> Let me produce some figures. Client in both cases is a dual 800/PIII
> connected through 100Mbit (switched) to the servers. The same document
> was used, document size was 247 bytes in both cases.

Thanks. Note that this is not at all the same test I was running. In
particular, VW does no file caching of static content, so if you're hitting
the same document thousands of times we're going to fetch it from disk
thousands of times. That's a very different scenario from running a servlet
in which no disk reads ever need to be done (but also interesting). I would
have expected that to mean that running as an SSP (which IS cached) would
go faster than direct file access, but it may be that the operating
system's file cache is circumventing the need for disk access and then the
overhead of executing the SSP (I assume you had this in production mode so
that the SSP was cached) took over.

I'm surprised you were getting dropped requests. In running VW directly
under Linux I almost never had any dropped requests, regardless of
concurrency level (under Windows I had quite a few).

It would be better if we could compare performance running on the same (or
at least the same type) of machine. It looks like there are significant
differences between the Apache machine and the VW machine.

I'm not surprised that CGI overhead was significant. Did you try using the
FastCGI connection?

Quote:
> 1. Apache running on an IBM pizzabox (933/PIII, local SCSI drive)

> Concurrency Level:      100
> Time taken for tests:   9.629 seconds
> Complete requests:      5000
> Failed requests:        0
> Total transferred:      2878752 bytes
> HTML transferred:       1240928 bytes
> Requests per second:    519.26
> Transfer rate:          298.97 kb/s received

> ---

> Concurrency Level:      500
> Time taken for tests:   9.119 seconds
> Complete requests:      5000
> Failed requests:        0
> Total transferred:      2877033 bytes
> HTML transferred:       1240187 bytes
> Requests per second:    548.31
> Transfer rate:          315.50 kb/s received

> 2. VisualWorks running on an IBM server (Dual 733/XeonIII, RAID-5),
>    file 'test.html', debugging switched off

> Concurrency Level:      100
> Time taken for tests:   17.103 seconds
> Complete requests:      5000
> Failed requests:        65
>    (Connect: 0, Length: 65, Exceptions: 0)
> Total transferred:      1870365 bytes
> HTML transferred:       1218945 bytes
> Requests per second:    292.35
> Transfer rate:          109.36 kb/s received

> ----

> Concurrency Level:      500
> Time taken for tests:   17.062 seconds
> Complete requests:      5000
> Failed requests:        56
>    (Connect: 0, Length: 56, Exceptions: 0)
> Total transferred:      1873776 bytes
> HTML transferred:       1221168 bytes
> Requests per second:    293.05
> Transfer rate:          109.82 kb/s received

> 3. VisualWorks again, but file is now called "test.ssp":

> Concurrency Level:      100
> Time taken for tests:   21.584 seconds
> Complete requests:      5000
> Failed requests:        54
>    (Connect: 0, Length: 54, Exceptions: 0)
> Total transferred:      3103858 bytes
> HTML transferred:       1562936 bytes
> Requests per second:    231.65
> Transfer rate:          143.80 kb/s received

> ---

> Concurrency Level:      500
> Time taken for tests:   20.872 seconds
> Complete requests:      5000
> Failed requests:        449
>    (Connect: 0, Length: 449, Exceptions: 0)
> Total transferred:      2855985 bytes
> HTML transferred:       1438116 bytes
> Requests per second:    239.56
> Transfer rate:          136.83 kb/s received

>===
> My conclusion: over the range of these tests, both seem to scale
> equally well, although VW seems to have some trouble at high loads (I'd
> need to see what these failed requests are). With a performance at 50%
> of Apache, I think VW does extremely well as an application server (and
> quite fine as a general webserver).

> OBTW: these were direct connections. With the CGI adapter in between,
> performance goes way down (waaaay down to about 25% of the direct
> figures, in my tests)

> Hth,

> Cees



Sat, 29 May 2004 01:02:26 GMT  
 
 [ 22 post ]  Go to page: [1] [2]

 Relevant Pages 

1. ensure: vs valueOnUnwindDo: (bug in VW7 Web Toolkit?)

2. The sim_agent toolkit - demonstration movies on web page

3. The sim_agent toolkit - demonstration movies on web page

4. WEB WEB WEB

5. AIDA/Web - Web Application Server released as Open Source

6. Trade Delphi 5 Enterprise Edition for Clarion Web Edition with Web Broker

7. To Web or not to Web.

8. Kicking remote web users from run-time application made with web publishing tool

9. ANNOUNCE: New Clipper Web Site on World Wide Web

10. dB Web Builder - Insert xBase Expressions into HTML to Generate Web Pages

11. ACM Symposium CFP: Web and Web Database

12. Web Think: Applying Web Thinking to Large Aerospace Applicaitons

 

 
Powered by phpBB® Forum Software