Tcl getting slower and slower on HP-UX 
Author Message
 Tcl getting slower and slower on HP-UX

Hello!

I have a problem with tcl 8.3 on a HP-UX B.10.20. The program is running
in about
1 min, if I add an if-statement is running about 1 h, getting slower and
slower
while running. On other systems (AIX, Linux) the problem doesn't exist.

The problem in detail:
The program reads a large file (about 10 MB, 200000 lines) containing
ascii data
using 'read' command and thereafter scans the data (code see below this
text).
The first 10000 lines are scanned in about 2 seconds, the next in 3-4
seconds,
line 100000 to 110000 are scanned in 1 min and the last 10000 lines
are scanned in about 2 minutes, although all lines are equal (in this
test
example)!
In other words: Tcl is getting slower and slower on a HP-UX.

While trying to understand the problem I removed much of the code inside
the
loop. Now it is running without problems, each 10000 lines are scanned
in
2-3 seconds. Then I added the code line by line. If I add one of the
(harmless concerning run time) if-statements the problem occurs again,
but only on HP-UX.

I wanted to figure out the problem by compiling tclsh with profiling
support.
I'm not sure about the interpretation of the result which I have
obtained.
The program is running about 1 h (system time), but only 65s are
reported
by the profiler. In the 'top' program the process always uses 100% of
the CPU power.
If I remove the if-statement the program runs in about 1 min and 50s are
reported by
the profiler.

Does anyone have an idea to solve this problem? I don't know wether it
is
tcl specific (if I change the tcl program the run time changes
tremendously)
or a hp-ux problem (what about the 59 min, missing in the profilers
summary)?

Thank you in advance

Carsten

----------------------------------------------------------------------------
Carsten Tr"abing                    

IMK - Uni Gh Kassel - M"onchebergstr.3 - D 34125 Kassel - Tel.
0561/804-3654

--------------------------------------------------------------------------------
If you want to reproduce my results, to the following steps:

1. Create a file 'kw.dat' containing about 200000 lines like this one:

CTRIA3   200735  204     201139  200385  200386  0.      0.

(its about 11 MB).

2. Create a tcl-script test.tcl:
- test.tcl
---------------------------------------------------------------------
proc ReadNastran {file field} {
    upvar $field WF

    set fi [open $file r]
    fconfigure $fi -blocksize 1000000
    set lines [read $fi]
    set lines [split $lines "\n"]
    set nolines [llength $lines]
    puts "  scanning $nolines lines..."
    close $fi

    set lnr 0
    set lnrpr 0
    set contl 0
    foreach line $lines {
        # incr line number
        incr lnrpr
        if {"[expr $lnrpr/10000]"=="[expr $lnrpr/10000.]"} {puts "
$lnrpr/$nolines [exec date +\"%M:%S\"]"}

        # skip empty lines
        if {[string length $line]==0} continue
        if {"[string tolower [string range $line 0 6]]"=="enddata"} break

        # comment?
        if {"[string range $line 0 0]"=="$"} {
            incr lnr
            set WF([format %07i ${lnr}].COMMENT) [string range $line 1 end]
            continue
        }

        # free format?
        if {[string first "," $line]>-1} {
            set data [split $line ","]

            # change format to fixed width (8)
            set line ""
            foreach d $data {
                append line [format %-8s $d]
            }
        }

# remove this block and it will run without problems
        # long format?
        if {$contl} {
            set field1 [string range $line 0 0]
        } {
            # Zeilennummer hochzaehlen, ausgeben
            incr lnr
            set field1 [string range $line 0 7]
        }
# end of block

        # save the data
        set idx [format %07i ${lnr}].GRID
        lappend WF($idx) "$line"
    }

Quote:
}

ReadNastran test.dat WF
exit
--------------------------------------------------------------------------------

3. run tclsh:
time tclsh8.3 test.tcl

-> time: about 1 h system time

4. remove the last if statement (between the two comment lines) and run
the
program again:
time tclsh8.3 test.tcl

-> time: about 1 min system time

--------------------------------------------------------------------------------
Some infos from the profiler:

Fast run without if statement, runs about 1min:
 %Time Seconds Cumsecs  #Calls   msec/call  Name

  10.1    5.43    5.43                      _mcount
   7.3    3.93    9.36  397159        0.01  TclExecuteByteCode
   7.0    3.76   13.1232205778        0.00  Tcl_UtfToUniChar
   4.3    2.30   15.42                      _monstartup
   2.3    1.21   16.63       1     1207.50  Tcl_SplitObjCmd
   2.2    1.16   17.7911764033        0.00  Tcl_UniCharToUtf
   2.1    1.13   18.92 4968388        0.00  malloc
...
sum:     50 sec

Slow run with if statement, runs about 1 h:
 %Time Seconds Cumsecs  #Calls   msec/call  Name

   0.5   11.79   11.79                      _mcount
   0.4    8.60   20.39  397159        0.02  TclExecuteByteCode
   0.2    4.89   25.28                      _monstartup
   0.2    3.94   29.2232205803        0.00  Tcl_UtfToUniChar
   0.1    2.31   31.53 6558001        0.00  malloc
   0.1    1.85   33.39 5362111        0.00  free
   0.1    1.50   34.89       1     1497.49  Tcl_SplitObjCmd
   0.1    1.42   36.31  397109        0.00  TclLookupVar
   0.1    1.30   37.6111764211        0.00  Tcl_UniCharToUtf
   0.1    1.27   38.88 8538533        0.00  Tcl_GetStringFromObj
...
sum:     65 sec.



Mon, 05 Aug 2002 03:00:00 GMT  
 Tcl getting slower and slower on HP-UX

Quote:

>Does anyone have an idea to solve this problem?

Just an inital double-check of the code:

Quote:
># remove this block and it will run without problems
> # long format?
> if {$contl} {
>     set field1 [string range $line 0 0]
> } {

^^^^^^^^^^

I'm assuming the actual code has an "else" between
the braces?

Also, are you using 8.3 final, or one of the betas?

-Bill



Mon, 05 Aug 2002 03:00:00 GMT  
 Tcl getting slower and slower on HP-UX

Quote:
> Hello!

> I have a problem with tcl 8.3 on a HP-UX B.10.20. The program is running
> in about
> 1 min, if I add an if-statement is running about 1 h, getting slower and
> slower
> while running. On other systems (AIX, Linux) the problem doesn't exist.

Hmmm... Well, this may or may not help but...

Quote:
> 2. Create a tcl-script test.tcl:
> - test.tcl
> ---------------------------------------------------------------------
> proc ReadNastran {file field} {
>     upvar $field WF

>     set fi [open $file r]
>     fconfigure $fi -blocksize 1000000
>     set lines [read $fi]
>     set lines [split $lines "\n"]
>     set nolines [llength $lines]
>     puts "  scanning $nolines lines..."
>     close $fi

>     set lnr 0
>     set lnrpr 0
>     set contl 0
>     foreach line $lines {
>    # incr line number
>    incr lnrpr
>    if {"[expr $lnrpr/10000]"=="[expr $lnrpr/10000.]"} {puts "
> $lnrpr/$nolines [exec date +\"%M:%S\"]"}

Put your expr in {}:

[expr {$lnpr/10000}] == [expr {$lnrpr/10000}]]

If you're trying to output the date every 10000 lines, using:

if {[expr {$lnrpr % 10000}] == 0}

should be faster.

Also, the exec call is a no-no. Use:

puts "$lnrpr/nolines [clock format [clock seconds] -format "%M:%S"]"

instead.

- Show quoted text -

Quote:
>    # skip empty lines
>    if {[string length $line]==0} continue
>    if {"[string tolower [string range $line 0 6]]"=="enddata"} break

>    # comment?
>    if {"[string range $line 0 0]"=="$"} {
>        incr lnr
>        set WF([format %07i ${lnr}].COMMENT) [string range $line 1 end]
>        continue
>    }

>    # free format?
>    if {[string first "," $line]>-1} {
>        set data [split $line ","]

>        # change format to fixed width (8)
>        set line ""
>        foreach d $data {
>            append line [format %-8s $d]
>        }
>    }

> # remove this block and it will run without problems
>    # long format?
>    if {$contl} {
>        set field1 [string range $line 0 0]
>    } {
>        # Zeilennummer hochzaehlen, ausgeben
>        incr lnr
>        set field1 [string range $line 0 7]
>    }
> # end of block

That's weird sinc contl is always 0 according to your code. Can't help you
there. Unless the code is different on the other platforms and string range ..
0 0 is a particularly slow call.

L

Quote:
>    # save the data
>    set idx [format %07i ${lnr}].GRID
>    lappend WF($idx) "$line"
>     }
> }
> ReadNastran test.dat WF
> exit
> --------------------------------------------------------------------------------

--
Laurent Duperval                 "Montreal winters are an intelligence test,

                                                                -Don Camilli
Penguin Power!         ***Nothing I say reflects the views of my employer***


Mon, 05 Aug 2002 03:00:00 GMT  
 Tcl getting slower and slower on HP-UX
Quote:

> Hello!

> I have a problem with tcl 8.3 on a HP-UX B.10.20. The program is running
> in about
> 1 min, if I add an if-statement is running about 1 h, getting slower and
> slower
> while running. On other systems (AIX, Linux) the problem doesn't exist.

> The problem in detail:

  [snip]

Are you hitting a memory wall?

Your detailed description indicates that the program slows dramatically
for the "last part" of the data.  Your profiler output indicates that
adding your "if" block causes about a million more calls to malloc().
These might indicate that your process is running out of physical memory
and starting to page fault.

I believe you can increase per-process memory on HP/UX using the csh
'limit' command.

Bob
--

Mayo Foundation                                           (507) 284-2702
Rochester MN, 55905 USA   http://www.mayo.edu/sppdg/sppdg_home_page.html



Mon, 05 Aug 2002 03:00:00 GMT  
 Tcl getting slower and slower on HP-UX

Quote:

>> Hello!

>> I have a problem with tcl 8.3 on a HP-UX B.10.20. The program is running
>> in about
>> 1 min, if I add an if-statement is running about 1 h, getting slower and
>> slower
>> while running. On other systems (AIX, Linux) the problem doesn't exist.

> Hmmm... Well, this may or may not help but...

It does on my HPUX:

Your code:

  scanning 200001 lines...

10000/200001 "20:12"

20000/200001 "21:18"

30000/200001 "23:53"

40000/200001 "28:42"

50000/200001 "38:48"

My fixes:

10000/200001 41:07
20000/200001 41:18
30000/200001 41:26
40000/200001 41:34
50000/200001 43:01
60000/200001 43:11
70000/200001 43:32
80000/200001 43:43
90000/200001 43:50
100000/200001 43:59
110000/200001 44:08
120000/200001 44:17
130000/200001 44:26
140000/200001 44:35
150000/200001 44:43
160000/200001 44:52
170000/200001 45:02
180000/200001 45:11
190000/200001 45:20

As you can see, there is a major speed improvement although there were some
hiccups, which I'm not sure how to explain.

L

--
Laurent Duperval                 "Montreal winters are an intelligence test,

                                                                -Don Camilli
Penguin Power!         ***Nothing I say reflects the views of my employer***



Mon, 05 Aug 2002 03:00:00 GMT  
 
 [ 5 post ] 

 Relevant Pages 

1. SysCreateObject gets slower and slower

2. Are Threads or Pthreads supported in HP-UX and Python on HP-UX :)

3. slow, very slow

4. Loop with Write key.vi goes slower and slower

5. Slow file access - Slow program startup

6. Printing to HP II LaserJet SLOW !

7. hp fortran is very slow

8. BUG?: HP-UX 10.20 and IO#gets followed by IO#puts

9. Form is slow getting up to speed

10. GETTING TO C5 EMBEDS IS VERY SLOW

11. canvas with >5000 items gets slow?

12. The Smalltalk Store: Why we've been slow, and why we're getting better

 

 
Powered by phpBB® Forum Software