Freeing reallocated memory 
Author Message
 Freeing reallocated memory

Hi all

A colleague of mine is using realloc to create large arrays (up to 200Mb in
size). Free doesn't seem to give up the allocated memory on the heap unless
the allocated memory is fairly small. The way we know it's not being given
up is by using the 'ps' command through Digital Unix to look at currently
allocated memory. Obviously such a large chunk of memory is easily seen and
will cause major problems if the realloc is used too much without being
freed properly. We are currently using Compaq C++ compiler ver 6.2-033. We
have found that the C++ route isn't any better but we're guessing that is
because under the STL templates we were using it's basically doing reallocs
and frees anyway because it has the same problem.

Does anyone know of any bugs or limits associated with reallocs and frees
under this version of the compiler or Unix? Are there any workarounds for
this type of problem?

Any help would greatly be appreciated.

Mel Hamer
--



Sun, 02 Oct 2005 08:26:13 GMT  
 Freeing reallocated memory

Quote:

> A colleague of mine is using realloc to create large arrays (up to 200Mb in
> size). Free doesn't seem to give up the allocated memory on the heap unless
> the allocated memory is fairly small. The way we know it's not being given
> up is by using the 'ps' command through Digital Unix to look at currently
> allocated memory. Obviously such a large chunk of memory is easily seen and
> will cause major problems if the realloc is used too much without being
> freed properly.

On Unix systems, the "arena" (heap) is managed by adjusting
the "program break" (top of bss) address via brk() or sbrk().
It would be unsafe for the allocator to reduce the amount of
storage in the arena *unless* the freed block abuts the end
of the bss area.  Most allocators use a "boundary tag" scheme
and thus have a small control node allocated for internal use
abutting the break, in which case reducing the break requires
that the block right under that control block is freed, and
the allocator would have to detect this, change the break,
move the control block, and fix up any links that pointed to
that control block.  While this is doable, it involves too
much overhead for the most typical applications, so it is
seldom implemented.  If you want to add break reduction to
your own copy of a malloc/realloc/free implementation, it
might be simplest to start with a "buddy system" scheme.

Generally speaking, it might be more productive to change the
application so that it doesn't require so much RAM.
--



Mon, 03 Oct 2005 14:52:35 GMT  
 Freeing reallocated memory
On 16 Apr 2003 00:26:13 GMT, "Mel Hamer"

Quote:

>Hi all

>A colleague of mine is using realloc to create large arrays (up to 200Mb in
>size). Free doesn't seem to give up the allocated memory on the heap unless
>the allocated memory is fairly small. The way we know it's not being given
>up is by using the 'ps' command through Digital Unix to look at currently
>allocated memory. Obviously such a large chunk of memory is easily seen and
>will cause major problems if the realloc is used too much without being
>freed properly. We are currently using Compaq C++ compiler ver 6.2-033. We
>have found that the C++ route isn't any better but we're guessing that is
>because under the STL templates we were using it's basically doing reallocs
>and frees anyway because it has the same problem.

>Does anyone know of any bugs or limits associated with reallocs and frees
>under this version of the compiler or Unix? Are there any workarounds for
>this type of problem?

>Any help would greatly be appreciated.

This is not a language issue.  You need to ask in a newgroup related
to your system.

<<Remove the del for email>>
--



Mon, 03 Oct 2005 14:52:47 GMT  
 Freeing reallocated memory

Quote:
> Hi all

> A colleague of mine is using realloc to create large arrays (up to 200Mb
in
> size). Free doesn't seem to give up the allocated memory on the heap
unless
> the allocated memory is fairly small. The way we know it's not being given
> up is by using the 'ps' command through Digital Unix to look at currently
> allocated memory. Obviously such a large chunk of memory is easily seen
and
> will cause major problems if the realloc is used too much without being
> freed properly. We are currently using Compaq C++ compiler ver 6.2-033. We
> have found that the C++ route isn't any better but we're guessing that is
> because under the STL templates we were using it's basically doing
reallocs
> and frees anyway because it has the same problem.

> Does anyone know of any bugs or limits associated with reallocs and frees
> under this version of the compiler or Unix? Are there any workarounds for
> this type of problem?

What you're seeing is the fact that malloc/realloc/free are defined for
managing memory *as seen by your program*.   A common implementation
strategy for these functions is that releasing memory back to the operating
system is typically deferred.   This is a performance optimisation, because
it avoids the problems of giving memory back simply to reclaim it
again:  it is quicker to simply return a pointer to memory that has been
free()d, but not actually handed back to the operating system.   This gives
a side effect that your program appears not to be releasing memory (eg when
using the Unix ps command).

I would suggest you not worry about this, unless it actually shows up as
a problem that prevents your software from running.
--



Mon, 03 Oct 2005 14:52:50 GMT  
 Freeing reallocated memory

Quote:

> up is by using the 'ps' command through Digital Unix to look at currently
> allocated memory.

[...]

You're at least partly mistaken: with all probability, that memory is
free()d properly.  What doesn't happen, though, is this freed part of
the memory being given back to the operating system.  This is typical
behaviour of malloc()/free() implementations on almost all Unix-like
platforms, not a problem of your particular C program.  Read the
platform documentation --- it might even mention this.

If you don't want this to happen, you'll have to do you allocations in
a different way.  E.g. about the worst thing you could do for a
dynamic array approach, is to realloc exactly to the needed new size
each time you run out of space.  This can leave you with O(s^2)
virtual memory requirements for an actual memory consumption of size
s.
--

Even if all the snow were burnt, ashes would remain.
--



Mon, 03 Oct 2005 14:53:04 GMT  
 Freeing reallocated memory

Quote:
>From the FAQ:

7.25:   I have a program which mallocs and later frees a lot of memory,
        but I can see from the operating system that memory usage
        doesn't actually go back down.

A:      Most implementations of malloc/free do not return freed memory
        to the operating system, but merely make it available for future
        malloc() calls within the same program.



Quote:
> Hi all

> A colleague of mine is using realloc to create large arrays (up to 200Mb in
> size). Free doesn't seem to give up the allocated memory on the heap unless
> the allocated memory is fairly small. The way we know it's not being given
> up is by using the 'ps' command through Digital Unix to look at currently
> allocated memory. Obviously such a large chunk of memory is easily seen and
> will cause major problems if the realloc is used too much without being
> freed properly. We are currently using Compaq C++ compiler ver 6.2-033. We
> have found that the C++ route isn't any better but we're guessing that is
> because under the STL templates we were using it's basically doing reallocs
> and frees anyway because it has the same problem.

> Does anyone know of any bugs or limits associated with reallocs and frees
> under this version of the compiler or Unix? Are there any workarounds for
> this type of problem?

> Any help would greatly be appreciated.

> Mel Hamer
> --


--
--

--



Mon, 03 Oct 2005 14:53:39 GMT  
 Freeing reallocated memory

Quote:

> A colleague of mine is using realloc to create large arrays (up
> to 200Mb in size). Free doesn't seem to give up the allocated
> memory on the heap unless the allocated memory is fairly small.
> The way we know it's not being given up is by using the 'ps'
> command through Digital Unix to look at currently allocated
> memory.

        Unless your system is displaying something different to other unix
variants I would not expect the ps command to show the the
effect of memory being freed.

        malloc/realloc/calloc generally obtain memory from the OS
and NEVER return it to the operating system. Freed memory is usually
added to the free memory  kept within the process heap segment, not
the operating system. Thus the memory of a process doing frequent
reallocs will usually increase steadily until there is enough memory
taken from then OS to supply new allocations as they are requested,
when memory usage as shown by ps levels off.

Quote:
> Obviously such a large chunk of memory is easily seen and
> will cause major problems if the realloc is used too much without
> being freed properly.

        There should not be a problem provided unused memory
is eventually freed. There is a remote possibility of a sequence
of small and large allocations resulting in memory fragmentation
where there is lots of free memory on the heap, but all the
available blocks are too small for the large allocation, causing
the heap manager to continually request more memory from the
OS. The probability of this happening with any decent memory
manager should be vanishingly small.

Charles
--



Mon, 03 Oct 2005 14:53:43 GMT  
 Freeing reallocated memory

Quote:


> > A colleague of mine is using realloc to create large arrays
> > (up to 200Mb in size). Free doesn't seem to give up the
> > allocated memory on the heap unless the allocated memory is
> > fairly small. The way we know it's not being given up is by
> > using the 'ps' command through Digital Unix to look at
> > currently allocated memory. Obviously such a large chunk of
> > memory is easily seen and will cause major problems if the
> > realloc is used too much without being freed properly.

> On Unix systems, the "arena" (heap) is managed by adjusting
> the "program break" (top of bss) address via brk() or sbrk().
> It would be unsafe for the allocator to reduce the amount of
> storage in the arena *unless* the freed block abuts the end
> of the bss area.  Most allocators use a "boundary tag" scheme
> and thus have a small control node allocated for internal use
> abutting the break, in which case reducing the break requires
> that the block right under that control block is freed, and
> the allocator would have to detect this, change the break,
> move the control block, and fix up any links that pointed to
> that control block.  While this is doable, it involves too
> much overhead for the most typical applications, so it is
> seldom implemented.  If you want to add break reduction to
> your own copy of a malloc/realloc/free implementation, it
> might be simplest to start with a "buddy system" scheme.

I find that that is not really necessary.  It suffices to build
things out of linked lists, and allows for efficient recombination
and efficient realloc (meaning a minimum of data movement).  For
an example see:

     <http://cbfalconer.home.att/download/nmalloc.zip>

That system also can easily detect when the empty top of the heap
is large enough to make return to the system worthwhile.  It
doesn't do it.  All operations are O(1).

--

   Available for consulting/temporary embedded and systems.
   <http://cbfalconer.home.att.net>  USE worldnet address!
--



Thu, 06 Oct 2005 01:24:32 GMT  
 Freeing reallocated memory

Quote:

>A colleague of mine is using realloc to create large arrays (up to 200Mb in
>size). Free doesn't seem to give up the allocated memory on the heap unless
>the allocated memory is fairly small. The way we know it's not being given
>up is by using the 'ps' command through Digital Unix to look at currently
>allocated memory.

There are two cases of large free() that are of interest.

1. You are going to reallocate a similar amount of memory in the same
program, in the near future.

2. This program is never going to need that memory again (or at least, not
for a long time, several minutes or hours).

In case 1, the Digital UNIX libc you're using is doing precisely the right
thing.  The memory is still allocated to your program, is on libc's free
list, and will be quick to allocate the next time you call malloc().  Don't
worry about it.

In case 2, there are two things to consider.  First, is the system you're
using actually short of virtual memory?  If you have a large amount of
space on the freelist, and the system runs low of physical memory,  the
kernel will simply write the pages you're not using out to swap.  They
will not consume physical memory.  They will be somewhat slower to access
the next time around, though.
Second, is your program going to be running on the system for a long time
(minutes to hours or days) without accessing the 200 MB allocation?  If
so, *AND* you feel that you need to optimize the VM consumption of your
program, you should investigate other methods to allocate memory.  For
example, the mmap(2) MAP_ANON flag or mapping /dev/zero can allow you to
control how your allocations are returned to the system.

As an example of a library that handles this case explicitly, the GNU libc
(glibc-2.3.1 is what I'm using now) has a value, M_MMAP_THRESHOLD, which
is documented thusly:

    `M_MMAP_THRESHOLD'
          All chunks larger than this value are allocated outside the
          normal heap, using the `mmap' system call.  This way it is
          guaranteed that the memory for these chunks can be returned
          to the system on `free'.

This value appears to be set to 128 kB on the system I'm using now.

If you want to discuss this problem in more depth, I would suggest that
you visit comp.unix.programmer or a newsgroup dedicated to Digital UNIX.

-andy
--



Sat, 08 Oct 2005 00:33:11 GMT  
 Freeing reallocated memory
One thing I forgot to mention is that on Unix, not only the
malloc subsystem moves the program break, but also the
application can glom onto a separate piece of storage by
moving the program break (I think "ed" does this, and "sh"
certainly does).  Therefore the malloc implementation must
not assume that the program break is where it left it..
Therefore it takes an extra system call just to be sure
that it's safe to reduce the break, even more overhead than
one might have thought.
--



Sat, 08 Oct 2005 00:33:43 GMT  
 
 [ 10 post ] 

 Relevant Pages 

1. Reallocating memory without disturbing the original address

2. reallocating memory

3. Reallocate memory

4. dynamically reallocate memory for object derived from class without destroying the data

5. How to Reallocate memory ?

6. can I know the amount of memory freed by call to free

7. how free realy frees memory ?

8. Problems freeing memory using "free"

9. Free memory, memory usage

10. freeing memory of memory CDC

11. reallocating space for pointer to structs?

12. getting a reallocated string back

 

 
Powered by phpBB® Forum Software