Making code faster 
Author Message
 Making code faster

How can I make the following code faster? I open a (text)
file and read the lines in the file. The lines of the file
have the form

[<string>,<string>]

After the lines have been loaded, I set up a new list via
the function tmp1.

    i = -1
    print 'loading file'
    def tmp1(s):
      global i
      i += 1
      return QSimpleViewItem(i,self.sv.lv.newColor,eval(s))

    try:
      myfile = open(filename)
    except:
      pass
    else:
      tmp = myfile.readlines()
      print 'lines loaded'
      datei.close()
      self.sv.lv.rows = map(tmp1,tmp)
      self.sv.lv.visible = range(len(self.sv.lv.rows))
      print 'lines decoded',self.sv.lv.rows[0].col[0]

Any ideas? It is astonishing that loading the lines from
external storage takes much less time than the mapping
itself!

--
Janos Blazi

-----------== Posted via Newsfeed.Com - Uncensored Usenet News ==----------
    http://www.*-*-*.com/       The #1 Newsgroup Service in the World!
-----= Over 100,000 Newsgroups - Unlimited Fast Downloads - 19 Servers =-----



Sun, 02 Jan 2005 00:20:23 GMT  
 Making code faster
JB wrote in comp.lang.python:

Quote:
> How can I make the following code faster? I open a (text)
> file and read the lines in the file. The lines of the file
> have the form

> [<string>,<string>]

> After the lines have been loaded, I set up a new list via
> the function tmp1.

>     i = -1
>     print 'loading file'
>     def tmp1(s):
>       global i
>       i += 1
>       return QSimpleViewItem(i,self.sv.lv.newColor,eval(s))

>     try:
>       myfile = open(filename)
>     except:
>       pass
>     else:
>       tmp = myfile.readlines()
>       print 'lines loaded'
>       datei.close()
>       self.sv.lv.rows = map(tmp1,tmp)
>       self.sv.lv.visible = range(len(self.sv.lv.rows))
>       print 'lines decoded',self.sv.lv.rows[0].col[0]

> Any ideas? It is astonishing that loading the lines from
> external storage takes much less time than the mapping
> itself!

The line "self.sv.lv.rows = map(tmp1,tmp)" makes some call into an
external library (PyQt?) that's likely taking up the whole time.

Gerhard
--
mail:   gerhard <at> bigfoot <dot> de       registered Linux user #64239
web:    http://www.cs.fhm.edu/~ifw00065/    OpenPGP public key id AD24C930
public key fingerprint: 3FCC 8700 3012 0A9E B0C9  3667 814B 9CAA AD24 C930
reduce(lambda x,y:x+y,map(lambda x:chr(ord(x)^42),tuple('zS^BED\nX_FOY\x0b')))



Sun, 02 Jan 2005 00:31:14 GMT  
 Making code faster

Quote:

> The line "self.sv.lv.rows = map(tmp1,tmp)" makes some call
> into an external library (PyQt?) that's likely taking up
> the whole time.

No, not really. PyQt is not used here and no other external
library is used either.
--
Janos Blazi

-----------== Posted via Newsfeed.Com - Uncensored Usenet News ==----------
   http://www.newsfeed.com       The #1 Newsgroup Service in the World!
-----= Over 100,000 Newsgroups - Unlimited Fast Downloads - 19 Servers =-----



Sun, 02 Jan 2005 01:04:26 GMT  
 Making code faster

Quote:

> JB wrote in comp.lang.python:
>> How can I make the following code faster? I open a (text)
>> file and read the lines in the file. The lines of the
>> file have the form

>> [<string>,<string>]

>> After the lines have been loaded, I set up a new list via
>> the function tmp1.

>>     i = -1
>>     print 'loading file'
>>     def tmp1(s):
>>       global i
>>       i += 1
>>       return
>>       QSimpleViewItem(i,self.sv.lv.newColor,eval(s))

>>     try:
>>       myfile = open(filename)
>>     except:
>>       pass
>>     else:
>>       tmp = myfile.readlines()
>>       print 'lines loaded'
>>       datei.close()
>>       self.sv.lv.rows = map(tmp1,tmp)
>>       self.sv.lv.visible = range(len(self.sv.lv.rows))
>>       print 'lines decoded',self.sv.lv.rows[0].col[0]

>> Any ideas? It is astonishing that loading the lines from
>> external storage takes much less time than the mapping
>> itself!

> The line "self.sv.lv.rows = map(tmp1,tmp)" makes some call
> into an external library (PyQt?) that's likely taking up
> the whole time.

Yes, sorry. You mean the QListSimpleViewItem constructor:

class QSimpleViewItem:
  def __init__(self,id,color,tuple):
    self.id    = id
    self.col   = tuple
    self.sel   = 0
    self.color = color

It is not very time consuming.

--
Janos Blazi

-----------== Posted via Newsfeed.Com - Uncensored Usenet News ==----------
   http://www.newsfeed.com       The #1 Newsgroup Service in the World!
-----= Over 100,000 Newsgroups - Unlimited Fast Downloads - 19 Servers =-----



Sun, 02 Jan 2005 01:07:13 GMT  
 Making code faster

Quote:

>>>     i = -1
>>>     print 'loading file'
>>>     def tmp1(s):
>>>       global i
>>>       i += 1
>>>       return QSimpleViewItem(i,self.sv.lv.newColor,eval(s))

you use a global --> time hit

Quote:
>>>       tmp = myfile.readlines()

readlines is highly optimized

Quote:
>>>       print 'lines loaded'
>>>       datei.close()
>>>       self.sv.lv.rows = map(tmp1,tmp)
>>>       self.sv.lv.visible = range(len(self.sv.lv.rows))
>>>       print 'lines decoded',self.sv.lv.rows[0].col[0]

>>> Any ideas? It is astonishing that loading the lines from
>>> external storage takes much less time than the mapping
>>> itself!

Have you actually used a profiler on the code to see where the time was going?
Could your call to eval() not be where the time is going?


Sun, 02 Jan 2005 01:17:06 GMT  
 Making code faster



Quote:
> Have you actually used a profiler on the code to see where the time was

going?

What's a profiler...and how is it used to see where the time is going?

Larry



Sun, 02 Jan 2005 02:51:52 GMT  
 Making code faster

Quote:
> What's a profiler...and how is it used to see where the time is going?

> Larry

Larry,

Section 10 in the Library Reference appears to cover it.

http://www.python.org/doc/current/lib/profile.html

If you're in a hurry, you can skip into 10.3, which jumps right into the
usage.

--

When we attempt the impossible, we can experience true growth.



Sun, 02 Jan 2005 03:14:44 GMT  
 Making code faster

Quote:

>How can I make the following code faster? I open a (text)
>file and read the lines in the file. The lines of the file
>have the form

>[<string>,<string>]

- Do you mean that literally? I.e., would
    "['abc','def']\n"
  be a valid example?
- Why do you have to eval() it? That is  probably timeconsuming compared to alternatives.
- Were the lines created by repr(someList)?
- Are all the <string>s actually quoted character sequences, or
  are there numbers too?
- Are there always two elements in the list?
Quote:

>After the lines have been loaded, I set up a new list via
>the function tmp1.

>    i = -1
>    print 'loading file'
>    def tmp1(s):
>      global i
>      i += 1
>      return QSimpleViewItem(i,self.sv.lv.newColor,eval(s))

I can't tell what the enclosing context is that defines 'self',
but you are repeating a lot of lookup doing self.sv.lv.newColor
on every call instead if doing it once at def time.
Likewise the global counter is not as fast as a local one.

To look up once, you could use a default arg, e.g.,

     # untested!
     def tmp1(s, color=self.sv.lv.newColor):
       global i
       i += 1
       return QSimpleViewItem(i,color,eval(s))

To get the counter into a local context as well, you could try
a factory function to return a tmp1 function with
local lookups from a closure.

     # untested!
     def get_tmp1(color):
         i = -1
         def tmp1(evald_s): # expect arg equivalent of eval(s) already done
             i += 1
             return QSimpleViewItem(i,color,evald_s)
         return tmp1

Quote:

>    try:
>      myfile = open(filename)
>    except:
>      pass
>    else:
>      tmp = myfile.readlines()
>      print 'lines loaded'
>      datei.close()

       ^^^^^-- not myfile?

if you must eval the file lines (which I doubt, and which typically carries security dangers),
I'd try mapping eval by itself first, like
       tmp = map(eval,tmp)
then get the function from the factory
       tmp1 = get_tmp1(self.sv.lv.newColor) # gets the color once

Quote:
>      self.sv.lv.rows = map(tmp1,tmp)
>      self.sv.lv.visible = range(len(self.sv.lv.rows))
>      print 'lines decoded',self.sv.lv.rows[0].col[0]

>Any ideas? It is astonishing that loading the lines from
>external storage takes much less time than the mapping
>itself!

I suspect eval() is costing you a big part, unless QSimpleViewItem() has a
complicated constructor/init function. Spec the file line format in detail,
so we can eliminate eval.

Regards,
Bengt Richter



Sun, 02 Jan 2005 15:42:19 GMT  
 Making code faster

Quote:

>>[<string>,<string>]
> - Do you mean that literally? I.e., would
>     "['abc','def']\n"
>   be a valid example?
> - Why do you have to eval() it? That is  probably
> timeconsuming compared to alternatives.

I thought, that would be fast. I can choose the format that
is best. (But probably, I 'd like to have a text format.)
So I could shoose
abc def
too.

Quote:
> - Are there always two elements in the list?

Yes.

Now I have changed my program to

    i = -1
    print 'loading file'
    def tmp1(s,color=self.sv.lv.newColor):
      global i
      i += 1
      j = string.find(s,' ')
      return QSimpleViewItem(i,color,[s[:j],s[j+1:-1]])

    try:
      myfile = open(filename)
    except:
      pass
    else:
      tmp = myfile.readlines()
      print 'lines loaded'
      myfile.close()
      self.sv.lv.rows = map(tmp1,tmp)
      self.sv.lv.visible = range(len(self.sv.lv.rows))
      print 'lines decoded',self.sv.lv.rows[0].col[0]

This is really faster. I did not undertand, how to get rid
of the global variable <i>, but even if I do not use i at
all and replace it by 0, the time I need is the same.

--
Janos Blazi

-----------== Posted via Newsfeed.Com - Uncensored Usenet News ==----------
   http://www.newsfeed.com       The #1 Newsgroup Service in the World!
-----= Over 100,000 Newsgroups - Unlimited Fast Downloads - 19 Servers =-----



Sun, 02 Jan 2005 18:06:39 GMT  
 Making code faster

Quote:

> How can I make the following code faster? I open a (text)
> file and read the lines in the file. The lines of the file
> have the form

> [<string>,<string>]

> After the lines have been loaded, I set up a new list via
> the function tmp1.

>     i = -1
>     print 'loading file'
>     def tmp1(s):
>       global i
>       i += 1
>       return QSimpleViewItem(i,self.sv.lv.newColor,eval(s))

>     try:
>       myfile = open(filename)
>     except:
>       pass
>     else:
>       tmp = myfile.readlines()
>       print 'lines loaded'
>       datei.close()
>       self.sv.lv.rows = map(tmp1,tmp)
>       self.sv.lv.visible = range(len(self.sv.lv.rows))
>       print 'lines decoded',self.sv.lv.rows[0].col[0]

> Any ideas? It is astonishing that loading the lines

 ... how man lines? How long are the lines?

Quote:
> from

 external storage takes much less time than the mapping

Quote:
> itself!

Have you considered to use Pyrex in order to convert it into a C module??

Armin



Sun, 02 Jan 2005 19:05:19 GMT  
 Making code faster

Quote:

>  ... how man lines? How long are the lines?

The lines are approximately 69 chars long. There are
approximately 100000 lines.

Quote:
> Have you considered to use Pyrex in order to convert it
> into a C module??

No, as I do not knoaw what Pyrex is.

--
Janos Blazi

-----------== Posted via Newsfeed.Com - Uncensored Usenet News ==----------
   http://www.newsfeed.com       The #1 Newsgroup Service in the World!
-----= Over 100,000 Newsgroups - Unlimited Fast Downloads - 19 Servers =-----



Sun, 02 Jan 2005 19:36:02 GMT  
 Making code faster

Quote:


>>>[<string>,<string>]
>> - Do you mean that literally? I.e., would
>>     "['abc','def']\n"
>>   be a valid example?
>> - Why do you have to eval() it? That is  probably
>> timeconsuming compared to alternatives.

>I thought, that would be fast. I can choose the format that
>is best. (But probably, I 'd like to have a text format.)
>So I could shoose
>abc def

Simple is generally good ;-)
Quote:
>too.

>> - Are there always two elements in the list?
>Yes.

>Now I have changed my program to

>    i = -1
>    print 'loading file'
>    def tmp1(s,color=self.sv.lv.newColor):
>      global i
>      i += 1
>      j = string.find(s,' ')
>      return QSimpleViewItem(i,color,[s[:j],s[j+1:-1]])

Replacing the above 2 lines with the following one should be faster:
       return QSimpleViewItem(i,color,s.split())

It assumes you have an s from your file that looks like

<opt ws><item1 with no embedded whitespace><ws><item2 with no embedded whitespace><opt ws>

Where <opt ws> is one or more optional white space characters [ \t\r\n\f]
and <ws> is at least one white space character.

Note how split() works:
 >>> 'abc def'.split()
 ['abc', 'def']
 >>> 'abc def\n'.split()
 ['abc', 'def']
 >>> '  abc    def  \n'.split()
 ['abc', 'def']
 >>> '  abc    def  \n  '.split()
 ['abc', 'def']
 >>>

Quote:

>    try:
>      myfile = open(filename)
>    except:
>      pass
>    else:
>      tmp = myfile.readlines()
>      print 'lines loaded'
>      myfile.close()
>      self.sv.lv.rows = map(tmp1,tmp)
>      self.sv.lv.visible = range(len(self.sv.lv.rows))
>      print 'lines decoded',self.sv.lv.rows[0].col[0]

>This is really faster. I did not undertand, how to get rid
>of the global variable <i>, but even if I do not use i at
>all and replace it by 0, the time I need is the same.

That was a good experiment, showing that global/local is
insignificant compared to other overheads in this particular
loop. But if you have a loop where the other stuff is really fast,
you will notice a difference when you execute enough loops.

Regards,
Bengt Richter



Sun, 02 Jan 2005 23:56:49 GMT  
 
 [ 12 post ] 

 Relevant Pages 

1. Making Lisp code run faster

2. fast, fast, fast sin, cos, sqrt - OSI open source code

3. Making std asm code into Turbo Asm code?

4. Making Smalltalk fast.

5. Making Apps Faster?

6. Making Applications Load Faster

7. Making the Synopsys VSS faster

8. Making a program fast and small

9. popen2 made >7 times faster!

10. Start Making Money Fast!

11. Free Software--EasyType 1.2 Basic makes you type as fast you think

12. making CMU + PCL + CLX + GINA run faster?

 

 
Powered by phpBB® Forum Software