perl memory efficiency 
Author Message
 perl memory efficiency

I've always known perl was sloppy with memory, but this is ridiculous.
The following program should generate 12.5 million integers for
storage.  Depending on your architecture, this may be 4-8 bytes per
int in an ideal situation.  However, on solaris 5.8 using perl64
5.6.1, it uses 863 megs!!!
If I use the 32-bit version of perl (5.004) on the same system, it
drops to 560 megs or so (I'm sure due to change in size of pointers
and perhaps default integer size).  Even on a linux(32 bit, of course)
box using perl 5.6.1, the usage is 550 megs.  Does anyone know how to
get around this?  an 8-fold blowup is unacceptible in my real program
(not this simple case) as I aim to have an array of at least 2 million
elements where each element is a 12-element array containing 3 strings
(short) and 9 integers (eventually 100 million of these 12-element
arrays).

anyone know any tricks to get around this?  I know if I had only ints
in the
array I could use vec, but that's not the case.  This is a general
problem
I have with perl (it's excessively sloppy memory usage--in this case,
using 70 bytes per int).

help?

offending code:

#!/usr/local/bin/perl64 -w

use strict;


my $numElem = 50;
my $numArr = 250000;

my $i;
for($i=0; $i <= $numArr; $i++)
    {
    my $j;
    for($j=0; $j < $numElem; $j++)
        {

        }
    }
print "wait\n";
<>;



Sun, 16 May 2004 04:51:54 GMT  
 perl memory efficiency
: I've always known perl was sloppy with memory, but this is ridiculous.
: The following program should generate 12.5 million integers for
: storage.  Depending on your architecture, this may be 4-8 bytes per
: int in an ideal situation.  However, on solaris 5.8 using perl64
: 5.6.1, it uses 863 megs!!!
: If I use the 32-bit version of perl (5.004) on the same system, it
: drops to 560 megs or so (I'm sure due to change in size of pointers
: and perhaps default integer size).  Even on a linux(32 bit, of course)
: box using perl 5.6.1, the usage is 550 megs.  Does anyone know how to
: get around this?  an 8-fold blowup is unacceptible in my real program
: (not this simple case) as I aim to have an array of at least 2 million
: elements where each element is a 12-element array containing 3 strings
: (short) and 9 integers (eventually 100 million of these 12-element
: arrays).

: anyone know any tricks to get around this?  I know if I had only ints
: in the
: array I could use vec, but that's not the case.  This is a general
: problem
: I have with perl (it's excessively sloppy memory usage--in this case,
: using 70 bytes per int).

That's not 70 bytes per integer - it's 70 bytes per dynamic polymorphic
data structure, all help within a larger dynamic data structure that can
be efficiently manipulated in a variety of useful ways.

70 bytes doesn't sound too bad really.

If you want ints to take 4 bytes then program in C, or get more memory, or
possibly tie the data to something else.



Sun, 16 May 2004 05:34:14 GMT  
 perl memory efficiency

Quote:

>That's not 70 bytes per integer - it's 70 bytes per dynamic polymorphic
>data structure, all help within a larger dynamic data structure that can
>be efficiently manipulated in a variety of useful ways.

>70 bytes doesn't sound too bad really.

>If you want ints to take 4 bytes then program in C, or get more memory, or
>possibly tie the data to something else.

Or store it in a big string and use vec() to access it.  It's not
clean, but it is efficient with memory.

  - Logan
--
"In order to be prepared to hope in what does not deceive,
 we must first lose hope in everything that deceives."

                                          Georges Bernanos



Sun, 16 May 2004 07:38:06 GMT  
 
 [ 3 post ] 

 Relevant Pages 

1. memory efficiency question

2. Another question about memory efficiency

3. perl memory management - does @array = () free the memory?

4. Interpreted or compiled -- Perl efficiency

5. sorting efficiency in perl

6. efficiency in data-processing perl script ...

7. help assessing efficiency of perl script, and testing file content size

8. references for perl efficiency

9. perl efficiency tip

10. efficiency, perl vs C

11. Perl efficiency question

12. Efficiency/New Perl Book: return @array vs \@array

 

 
Powered by phpBB® Forum Software