Slow file access - Slow program startup 
Author Message
 Slow file access - Slow program startup

Hello,

   I have been reading the last post about TCL8.4 file acces system
slowness and we have decided to make our tests.

   We have a large program that when opens, it looks at several files
and directories with TCL.

   The time the program takes to start and the window appear
is the following:

   - With TCL 8.3.4

   First shoot:     15'
   Following shoots:  3'

   - With TCL 8.4.1

   First shoot:     90'
   Following shoots: 11'

  As we see, it takes from 4 to 6 times more to start the program
with TCL 8.4 than with TCL 8.3

  I would appreciate if someone can confirm these numbers. If this is
the case, of course, it is not possible to upgrade the version.

    System: Win98 with FAT32

  Best regards,

   Ramon Rib



Wed, 27 Jul 2005 19:48:30 GMT  
 Slow file access - Slow program startup
jep, same "VERY BIG" win2k problem here

i also have to cancel my plans on jumping over to 8.4, and i HATE it,
since so
many other things are so much better in 8.4

file exists is useless in 8,4 if you ask me, if there are more then 40
files in a loop of "file exists" you can go to bed, if the are on a
network you are in trouble == "me"

please!!!, someone bring the magic!!!

Quote:

> Hello,

>    I have been reading the last post about TCL8.4 file acces system
> slowness and we have decided to make our tests.

>    We have a large program that when opens, it looks at several files
> and directories with TCL.

>    The time the program takes to start and the window appear
> is the following:

>    - With TCL 8.3.4

>    First shoot:     15'
>    Following shoots:  3'

>    - With TCL 8.4.1

>    First shoot:     90'
>    Following shoots: 11'

>   As we see, it takes from 4 to 6 times more to start the program
> with TCL 8.4 than with TCL 8.3

>   I would appreciate if someone can confirm these numbers. If this is
> the case, of course, it is not possible to upgrade the version.

>     System: Win98 with FAT32

>   Best regards,

>    Ramon Rib



Thu, 28 Jul 2005 01:28:24 GMT  
 Slow file access - Slow program startup

Quote:

>   As we see, it takes from 4 to 6 times more to start the program
> with TCL 8.4 than with TCL 8.3

>   I would appreciate if someone can confirm these numbers. If this is
> the case, of course, it is not possible to upgrade the version.

>     System: Win98 with FAT32

FWIW, I've built Tclkit with Vince Darley's latest speedup patch from
SourceForge - feel free to compare:
    http://www.equi4.com/pub/tk/newer/win32.upx.exe
    http://www.equi4.com/pub/tk/newer/win32.exe.gz
Unpack/rename to tclkit.exe (or wish.exe, if you insist) and it should
drop right in as wish replacement.

These changes will of course also make it to Tcl itself, this is just
a way to quickly find out what needs to be done to improve things.

-jcw



Thu, 28 Jul 2005 02:47:53 GMT  
 Slow file access - Slow program startup

Quote:

>    I have been reading the last post about TCL8.4 file acces system
> slowness and we have decided to make our tests.

I would appreciate it if those considering tests for this would look
into the tclbench system noted here:
        http://wiki.tcl.tk/1611
which is specifically designed for this stuff.

Quote:
>   As we see, it takes from 4 to 6 times more to start the program
> with TCL 8.4 than with TCL 8.3

>   I would appreciate if someone can confirm these numbers. If this is
> the case, of course, it is not possible to upgrade the version.

I wonder how much file access you are doing in your programs that 4-6
times for just that portion is a large cost.  The actual interpreter of
8.4 has many improvement, actual movement of data is faster.  It is
just the file commands as singletons that slow down.

--
     Jeff Hobbs                     The Tcl Guy
     Senior Developer               http://www.ActiveState.com/
         Tcl Support and Productivity Solutions



Thu, 28 Jul 2005 03:35:20 GMT  
 Slow file access - Slow program startup


:I wonder how much file access you are doing in your programs that 4-6
:times for just that portion is a large cost.  The actual interpreter of
:8.4 has many improvement, actual movement of data is faster.  It is
:just the file commands as singletons that slow down.
:

In my case (one of the initial thread starters), the apps are pretty small -
so the startup times are what are killing us.  Users need the apps to
start, execute and complete in under 5 seconds and they were taking 15-20
seconds.

--
Tcl - The glue of a new generation.  <URL: http://wiki.tcl.tk/ >
Even if explicitly stated to the contrary, nothing in this posting
should be construed as representing my employer's opinions.



Thu, 28 Jul 2005 19:46:50 GMT  
 Slow file access - Slow program startup
[posted and mailed]


Quote:
>:I wonder how much file access you are doing in your programs that 4-6
>:times for just that portion is a large cost.  The actual interpreter
>:of 8.4 has many improvement, actual movement of data is faster.  It is
>:just the file commands as singletons that slow down.
> In my case (one of the initial thread starters), the apps are pretty
> small - so the startup times are what are killing us.  Users need the
> apps to start, execute and complete in under 5 seconds and they were
> taking 15-20 seconds.

That would be bad, but my own tests just don't bear out such a slowdown.
The following are the results from a new benchmark, run on a PIII running
Linux, accessing the local file system:

000 VERSIONS:                      1:8.4.1.1 2:8.4.1 3:8.3.5
001 FILE exec interp                 12162   12911   11043
002 FILE exec interp: pkg require    62241   66609   47179

That is with pkg require doing:

"lappend auto_path /usr/local/TDK25/lib;
catch {package require bogus-name}; package names"

And with a more extensive pkg require:

"lappend auto_path /usr/local/TDK25/lib;
catch {package require bogus-name}; package require XMLRPC;
package names"

000 VERSIONS:                      1:8.4.1.1 2:8.4.1 3:8.3.5
001 FILE exec interp                 12187   12980   10892
002 FILE exec interp: pkg require   173924  191655  151085

The numbers are in usecs ... that's still a fraction of a second.
The 8.4.1.1 numbers are of an in-development speedup for the VFS
stuff.  The full FILE numbers for that in-dev version are:

000 VERSIONS:                      1:8.4.1.1 2:8.4.1 3:8.3.5
001 FILE exec interp                 12187   12980   10892
002 FILE exec interp: pkg require   173924  191655  151085
003 FILE exists tmpfile (1)             45      46      42
004 FILE exists tmpfile (100)           17      17      14
005 FILE exists! tmpfile (1)           102     133      47
006 FILE exists! tmpfile (100)          75      81      18
007 FILE glob  tmpdir (60 entries)     427     430     344
008 FILE glob / all subcommands      37829   40529   38303
009 FILE glob / atime                 3248    5805    1310
010 FILE glob / attributes           23785   26677   22239
011 FILE glob / dirname               1925    2085     847
012 FILE glob / executable            3237    5767    1258
013 FILE glob / exists                3230    5775    1253
014 FILE glob / extension              999     669     597
015 FILE glob / isdirectory           3269    5851    1319
016 FILE glob / isfile                3268    5847    1309
017 FILE glob / mtime                 3278    5809    1322
018 FILE glob / owned                 3336    5920    1367
019 FILE glob / readable              3197    5768    1250
020 FILE glob / rootname              1019     686     615
021 FILE glob / size                  3271    5846    1297
022 FILE glob / tail                  1410    1599     747
023 FILE glob / writable              3212    5788    1249
024 FILE recurse / -dir              15959   20430   11248
025 FILE recurse / cd                20022   30421   10820
025 BENCHMARKS                     1:8.4.1.1 2:8.4.1 3:8.3.5

Do you have an example of a simple app that really shows such a slowdown
when going from 8.3 -> 8.4?

--
  Jeff Hobbs                     The Tcl Guy
  Senior Developer               http://www.ActiveState.com/
        Tcl Support and Productivity Solutions



Fri, 29 Jul 2005 09:24:23 GMT  
 Slow file access - Slow program startup
Quote:


>> In my case (one of the initial thread starters), the apps are pretty
>> small - so the startup times are what are killing us.  Users need the
>> apps to start, execute and complete in under 5 seconds and they were
>> taking 15-20 seconds.

>That would be bad, but my own tests just don't bear out such a slowdown.
>The following are the results from a new benchmark, run on a PIII running
>Linux, accessing the local file system: [...]

                  ^^^^^^^^^^^^^^^^^^^^^^

In my experience, the slowdown is heavily dependent
on the speed of the filesystem; the problem won't be
as visible on a local drive.  (OTOH, tk_getOpenFile
on an NFS-mounted directory from a slow server can
take minutes...)

Also, on Linux (at least for ext2 and ext3 filesystems)
the OS directory cache makes a big difference.  On a
freshly booted system, the first time any Tcl 8.4 progam
does a [package require] there's a noticeable delay,
but in subsequent runs the slowdown isn't as significant.
(Subjectively, it goes from "too slow" to "fast enough"
on my laptop, a 400Mhz PIII running Debian sarge).

--Joe English




Fri, 29 Jul 2005 11:50:23 GMT  
 Slow file access - Slow program startup

Quote:

>>The following are the results from a new benchmark, run on a PIII running
>>Linux, accessing the local file system: [...]
        ...
> In my experience, the slowdown is heavily dependent
> on the speed of the filesystem; the problem won't be
> as visible on a local drive.  (OTOH, tk_getOpenFile

But the general FS speed should not be a factor to Tcl.  It still
makes the same number of internals calls.  In fact, a really slow
FS would marginalize the extra overhead that VFS introduces.  The
exception is unnecessary stat'ing which wasn't done before, since
that will be even slower on a slow FS.  That will be addressed.

--
     Jeff Hobbs                     The Tcl Guy
     Senior Developer               http://www.ActiveState.com/
         Tcl Support and Productivity Solutions



Fri, 29 Jul 2005 15:31:05 GMT  
 Slow file access - Slow program startup
Hello,

   Have you tried these tests in Win98 with FAT32 or in any other
Windows? As I see from what people say and from my own tests, this
is a Windows specifically problem, and maybe even a FAT32 problem.

   Best regards,  

   Ramon Rib

Quote:

> [posted and mailed]


> >:I wonder how much file access you are doing in your programs that 4-6
> >:times for just that portion is a large cost.  The actual interpreter
> >:of 8.4 has many improvement, actual movement of data is faster.  It is
> >:just the file commands as singletons that slow down.

> > In my case (one of the initial thread starters), the apps are pretty
> > small - so the startup times are what are killing us.  Users need the
> > apps to start, execute and complete in under 5 seconds and they were
> > taking 15-20 seconds.

> That would be bad, but my own tests just don't bear out such a slowdown.
> The following are the results from a new benchmark, run on a PIII running
> Linux, accessing the local file system:

> 000 VERSIONS:                      1:8.4.1.1 2:8.4.1 3:8.3.5
> 001 FILE exec interp                 12162   12911   11043
> 002 FILE exec interp: pkg require    62241   66609   47179

> That is with pkg require doing:

> "lappend auto_path /usr/local/TDK25/lib;
> catch {package require bogus-name}; package names"

> And with a more extensive pkg require:

> "lappend auto_path /usr/local/TDK25/lib;
> catch {package require bogus-name}; package require XMLRPC;
> package names"

> 000 VERSIONS:                      1:8.4.1.1 2:8.4.1 3:8.3.5
> 001 FILE exec interp                 12187   12980   10892
> 002 FILE exec interp: pkg require   173924  191655  151085

> The numbers are in usecs ... that's still a fraction of a second.
> The 8.4.1.1 numbers are of an in-development speedup for the VFS
> stuff.  The full FILE numbers for that in-dev version are:

> 000 VERSIONS:                      1:8.4.1.1 2:8.4.1 3:8.3.5
> 001 FILE exec interp                 12187   12980   10892
> 002 FILE exec interp: pkg require   173924  191655  151085
> 003 FILE exists tmpfile (1)             45      46      42
> 004 FILE exists tmpfile (100)           17      17      14
> 005 FILE exists! tmpfile (1)           102     133      47
> 006 FILE exists! tmpfile (100)          75      81      18
> 007 FILE glob  tmpdir (60 entries)     427     430     344
> 008 FILE glob / all subcommands      37829   40529   38303
> 009 FILE glob / atime                 3248    5805    1310
> 010 FILE glob / attributes           23785   26677   22239
> 011 FILE glob / dirname               1925    2085     847
> 012 FILE glob / executable            3237    5767    1258
> 013 FILE glob / exists                3230    5775    1253
> 014 FILE glob / extension              999     669     597
> 015 FILE glob / isdirectory           3269    5851    1319
> 016 FILE glob / isfile                3268    5847    1309
> 017 FILE glob / mtime                 3278    5809    1322
> 018 FILE glob / owned                 3336    5920    1367
> 019 FILE glob / readable              3197    5768    1250
> 020 FILE glob / rootname              1019     686     615
> 021 FILE glob / size                  3271    5846    1297
> 022 FILE glob / tail                  1410    1599     747
> 023 FILE glob / writable              3212    5788    1249
> 024 FILE recurse / -dir              15959   20430   11248
> 025 FILE recurse / cd                20022   30421   10820
> 025 BENCHMARKS                     1:8.4.1.1 2:8.4.1 3:8.3.5

> Do you have an example of a simple app that really shows such a slowdown
> when going from 8.3 -> 8.4?



Fri, 29 Jul 2005 17:07:41 GMT  
 Slow file access - Slow program startup
Hello,

   More measurements:

   In Windows2000 with NTFS:

  Tcl 8.4.1 -- 3 seg
  Tcl 8.3.3 -- 1 seg

  So in this case it is a factor 3.

   Regards,

   Ramon Rib

Quote:

> Hello,

>    I have been reading the last post about TCL8.4 file acces system
> slowness and we have decided to make our tests.

>    We have a large program that when opens, it looks at several files
> and directories with TCL.

>    The time the program takes to start and the window appear
> is the following:

>    - With TCL 8.3.4

>    First shoot:     15'
>    Following shoots:  3'

>    - With TCL 8.4.1

>    First shoot:     90'
>    Following shoots: 11'

>   As we see, it takes from 4 to 6 times more to start the program
> with TCL 8.4 than with TCL 8.3

>   I would appreciate if someone can confirm these numbers. If this is
> the case, of course, it is not possible to upgrade the version.

>     System: Win98 with FAT32

>   Best regards,

>    Ramon Rib



Fri, 29 Jul 2005 18:05:50 GMT  
 Slow file access - Slow program startup


:Do you have an example of a simple app that really shows such a slowdown
:when going from 8.3 -> 8.4?

Yep - it was posted earlier in the thread:

#! /usr/tcl83/bin/tclsh
package require Tclx
puts "done"

vs

#! /usr/tcl84/bin/tclsh
package require Tclx
puts "done"

I resolved my problem by no longer doing a package require, but doing
a load directly.

The number of subdirectories for the two versions were pretty much
the same (211 subdirs in the Tcl 8.3 and 201 in the Tcl 8.4 tree).

--
Tcl - The glue of a new generation.  <URL: http://wiki.tcl.tk/ >
Even if explicitly stated to the contrary, nothing in this posting
should be construed as representing my employer's opinions.



Fri, 29 Jul 2005 20:37:52 GMT  
 Slow file access - Slow program startup
Hi all
Run a test script over a mapped netowrk directory:

898717 microsec. ---> new patch

5979288 microsec. ---> 8.4.1

platform w2k.

Xavier

the script:

############################

console show

cd "X:/MP3/Xavier/Rock & Pop"

proc test_cmd {} {
        foreach file [glob -nocomplain *] {
                if {[file exists $file]} {set a hmmm}
        }

Quote:
}

puts [time test_cmd]


Sat, 30 Jul 2005 01:41:55 GMT  
 Slow file access - Slow program startup

Quote:

>[ ... ]
> Also, on Linux (at least for ext2 and ext3 filesystems)
> the OS directory cache makes a big difference.  On a
> freshly booted system, the first time any Tcl 8.4 progam
> does a [package require] there's a noticeable delay,
> but in subsequent runs the slowdown isn't as significant.
> (Subjectively, it goes from "too slow" to "fast enough"
> on my laptop, a 400Mhz PIII running Debian sarge).

You can always flush out the buffer cache with a little memory hog:

    (ulimit -v MMMMMM; perl -e 'while(1) {$x.= "x$x";}')

where MMMMMM is near your physical memory in kilobytes.  Your test should
hit the disk after that.

Er, sorry for the perl example in this newsgroup :-)  It's the only
one-liner memory hog I have handy.  I'd be happy with
"set x {}; while 1 {append x x$x}" but how do I run it from the
command line?



Sat, 30 Jul 2005 09:35:46 GMT  
 Slow file access - Slow program startup

Quote:

> Er, sorry for the perl example in this newsgroup :-)  It's the only
> one-liner memory hog I have handy.  I'd be happy with
> "set x {}; while 1 {append x x$x}" but how do I run it from the
> command line?

Easy: echo 'for {set x x} 1 {append x $x} {}' | tclsh

Donal.
--

-- I could even declare myself a religion, if that'd help.



Sat, 30 Jul 2005 22:29:03 GMT  
 
 [ 14 post ] 

 Relevant Pages 

1. Slow File Access

2. Too slow file access with CW5.0

3. Awesome slowing in concurrent file access on NT WS

4. Files access on NT are slow!..

5. Slow multiuser access to Clarion 4.0 .tps files

6. Slow multiuser access to Clarion 4.0 tps files on Win2000 terminal server

7. slow direct file access in g77

8. Painfully slow startup

9. Python startup seems slow... (NT)

10. Python startup seems slow... (NT)

11. python startup on XP very slow

12. Slow startup of application

 

 
Powered by phpBB® Forum Software