memory leak and leak-fixing 'patterns' 
Author Message
 memory leak and leak-fixing 'patterns'

I'm in the process of plugging memory leaks in a fairly complex system
which leaks quite a bit.  It's GUI-based and uses lots of extensions, both
python and C-based (Pmw, Tkinter, PyOpenGL, Acquisition, others). Clearly,
I have my work cut out for me.

I was wondering if anyone had developed any tools to aid in finding such
leaks.  Alternatively, if anyone wishes to share their expertise as to
'common problems w/ memory leaks' in Python, I (and, I suspect, others)
would learn a thing or two.  Has anyone found Purify and the like to be
helpful in debugging Python code (as opposed to the Python interpreter)?

On a related note, would it be at all possible to have a
memory_used(object) function which returned the amount of memory used by a
Python object?  That function (using knowledge of the implementation, and,
of course, implementation-specific) could then be used by other functions
such as memory_in_contents(object, deep=0) (for container objects like
lists and dicts) or memory_in_referencees(object, deep=0) (with the latter
following all references, and thereby giving the total amount of memory
used by Python objects in the process)?

yes, I know that

        memory_used(ob1) + memory_used(ob2) + memory_used([])

and

        memory_used([ob1,ob2])

may or may not be the same...

I'm more interested in memory_used(ob) at time1 vs. memory_used(ob) at
time2...  Purely as a debugging aid...

Cheers,

--david



Sat, 03 Mar 2001 03:00:00 GMT  
 memory leak and leak-fixing 'patterns'

Quote:

> I'm in the process of plugging memory leaks in a fairly complex system
> which leaks quite a bit.  It's GUI-based and uses lots of extensions, both
> Python and C-based (Pmw, Tkinter, PyOpenGL, Acquisition, others). Clearly,
> I have my work cut out for me.

> I was wondering if anyone had developed any tools to aid in finding such
> leaks.  Alternatively, if anyone wishes to share their expertise as to
> 'common problems w/ memory leaks' in Python, I (and, I suspect, others)
> would learn a thing or two.  Has anyone found Purify and the like to be
> helpful in debugging Python code (as opposed to the Python interpreter)?

I've had some trouble with this too lately. The solution was usually
slowly narrowing down the responsible calls to extensions by carefully
looking for invariants. For leaks in Python instances I had to go
a little further and instrument the interpreter (i.e. patching it).

There's are also a few other ways (apart from using Purify) to
identify leaks: one is compiling Python with symbol Py_DEBUG defined
and then querying sys.getobjects(), the other is using Vladimir
Marangozov's pymalloc:

        http://sirac.inrialpes.fr/~marangoz

Quote:
> On a related note, would it be at all possible to have a
> memory_used(object) function which returned the amount of memory used by a
> Python object?

There a sizeof() function in mxTools. It returns much the same
as the C macro of the same name, e.g.

Quote:
>>> sizeof([])
16
>>> sizeof(())
12
>>> sizeof([1,2])
16
>>> sizeof((1,2))
20
>>> sizeof(1)
12
>>> sizeof(1.5)
16
>>> sizeof(1.5j)

24

Note that dynamically allocated memory that is only referenced
by the object (using a pointer) does not count. Only the size
of the pointer goes into the value.

--
Marc-Andre Lemburg                               Y2000: 471 days left
---------------------------------------------------------------------
          : Python Pages >>> http://starship.skyport.net/~lemburg/  :
           ---------------------------------------------------------



Sun, 04 Mar 2001 03:00:00 GMT  
 memory leak and leak-fixing 'patterns'

Quote:


> > I was wondering if anyone had developed any tools to aid in finding such
> > leaks.  Alternatively, if anyone wishes to share their expertise as to
> > 'common problems w/ memory leaks' in Python, I (and, I suspect, others)
> > would learn a thing or two.  Has anyone found Purify and the like to be
> > helpful in debugging Python code (as opposed to the Python interpreter)?
> ...
> There's are also a few other ways (apart from using Purify) to
> identify leaks: one is compiling Python with symbol Py_DEBUG defined
> and then querying sys.getobjects(), the other is using Vladimir
> Marangozov's pymalloc:

>         http://sirac.inrialpes.fr/~marangoz

The correct url is http://starship.skyport.net/~vlad/pymalloc/

The most common reasons for leaking memory, on the top of my head, are:

1. The core interpreter leaks memory => make sure you use the most recent
     version of it (currently 1.5.1)
2. Circular references from Python code => search dejanews for hints on
     how to detect/avoid them
3. Bad memory management in extension modules, which may be:
     a) wrong refcounting => recompiling w/ Py_DEBUG defined might help
     b) missing frees on chunks allocated by the module
     c) mixing memory allocators (the one used by Python -> usually the
          malloc in libc vs. different one(s) used by the extension(s) ).
          Example: the 1st malloc allocates a chunk, then the 2nd is
          asked to free it. In this case, you're *not* lucky if you
          don't dump core soon.

Point 3 requires some (time +) more powerful & low-level tools for
memory management and analysis, like the ones listed at:
http://www.cs.colorado.edu/~zorn/MallocDebug.html

pymalloc can be considered as such a tool adapted to Python, although
you can't do fine grain (per object) management with it. I'll probably
add such facility some day, but don't ask me when. See the url for more
info.

Anyway, a good strategy to deal with the problem is to try to catch &
isolate the piece(s) of code that causes the leak by cutting some parts
of the program. Thus it would be possible to focus subsequently on
potential solutions.

--

http://sirac.inrialpes.fr/~marangoz | tel:(+33-4)76615277 fax:76615252



Sun, 04 Mar 2001 03:00:00 GMT  
 memory leak and leak-fixing 'patterns'

Quote:

> From python the caller populates an object like this:

>     in.info["radius"] = 2.5
>     in.info["nodata"] = 255

> Then I want to extract this data from C using:
>     PyObject* in_info = PyObject_GetAttrString(in, "info");
>     PyObject* myradius = Py_BuildValue("s", "radius");
>     PyObject* po = PyObject_GetItem(in_info, myradius);
>     double radius = PyFloat_AsDouble(po);

> Obvously I have a few if-statements to verify that I receive proper PyObjects
> etc, but that's not the issue here.

> What I experience is that occasionally PyObject_GetItem() returns NULL even
> though I KNOW there is an item! Even more strange: if I insert the line:

>     n=PyObject_Length(in_info);

> ... in order to dump all existing objects the object magically reappears! It
> is all very repeatable.

This behavior is most probably due to some erroneous logic in your code.
As you don't provide it in its details, I hardly can guess what's going
wrong. (Commenting all Py_DECREF's is certainly a bad idea, so you'd
better restore them).

Quote:
> - Do you consider these parts of python rock solid?

yes, absolutely.

Quote:
> - Am I using Py*-calls that I shouldn't use?

I really can't answer that question because everything is in the details
which are missing in your msg. The Python/C API by itself is okay.

Quote:
> - Any other tips on how the track this?

Try some alternatives for the code sequence above and see what happens, eg:

    PyObject* myradius = PyString_FromString("radius");
    PyObject* po = PyObject_GetItem(in_info, myradius);

or

    PyObject* po = PyMapping_GetItemString(in_info, "radius");

--

http://sirac.inrialpes.fr/~marangoz | tel:(+33-4)76615277 fax:76615252



Mon, 02 Apr 2001 03:00:00 GMT  
 
 [ 5 post ] 

 Relevant Pages 

1. What are 'Memory Leaks'?

2. Memory leak with 'fileevent' command

3. memory leak in Tkinter's Canvas widget

4. BLT memory leak: Tree objects don't release key value pairwise data

5. Patch to tcl 8.3.0 to fix memory leaks and deadlocks in exec

6. BUG+FIX: Memory leak in Tk8.0

7. 'exec' mem leak on NT

8. python startup memory size and memory leak

9. Uninitialized memory errors and memory leaks in Tk

10. memory usage (how to debug a memory leak?)

11. User's Personal PL/I Neural Network Leaks

12. Patch with mem leak fixes for Python 1.5.2

 

 
Powered by phpBB® Forum Software