memory usage question 
Author Message
 memory usage question

I need to write a script that will use a hash with 4 million strings of 16
chars each.  As a test I wrote this quick script:

  h = {}; s = 'aaaaaaaaaaaaaaaa'
  for i in 1..4_000_000
    h[i] = (s = s.succ)
  end

This ran relatively fast, but it consumed 265 megs of windows memory and
failed with out of memory error at around 3 million elements.

Why did Ruby fail at 265 megs and is there another way for me to do this?



Sun, 04 Jul 2004 01:38:58 GMT  
 memory usage question

Quote:
----- Original Message -----


Sent: Tuesday, January 15, 2002 11:37 AM
Subject: memory usage question

> I need to write a script that will use a hash with 4 million strings of 16
> chars each.  As a test I wrote this quick script:

>   h = {}; s = 'aaaaaaaaaaaaaaaa'
>   for i in 1..4_000_000
>     h[i] = (s = s.succ)
>   end

> This ran relatively fast, but it consumed 265 megs of windows memory and
> failed with out of memory error at around 3 million elements.

> Why did Ruby fail at 265 megs and is there another way for me to do this?

Four million strings, eh?  :)

<oldtimer_rant>
Buncha young kids wanna allocate data structures with four million
items... in my day, we had 64K of RAM and part of that was the OS
and the runtime. The compiler wouldn't all fit in memory at once. It
was swapped out to this clunky external Shugart floppy disk drive
that would walk across the table if you didn't tape it down. It made a
noise like a boa constrictor gagging on a weasel. And the compiler
couldn't handle arrays bigger than a thousand elements.
And we LIKED IT...
</oldtimer_rant>

But seriously...  :)

Even today when RAM approaches a gigabyte in daily life (and your PC
may have a terabyte drive soon), memory ain't always the way to go.

Eventually you should consider using a database or something instead.
If you want something that will have a low impact on your code, try the
dbm interface. Very easy, not as much overhead as a "real" database.

There's also gdbm and maybe one or two other flavors. Not sure what
the differences are. Last time I tried, gdbm worked better for me on
Windows.

I think it's part of the standard distribution.

Hope this helps.

Hal



Sun, 04 Jul 2004 02:11:28 GMT  
 memory usage question


Quote:
>----- Original Message -----


>Sent: Tuesday, January 15, 2002 11:37 AM
>Subject: memory usage question

>> I need to write a script that will use a hash with 4 million strings of 16
>> chars each.  As a test I wrote this quick script:

>>   h = {}; s = 'aaaaaaaaaaaaaaaa'
>>   for i in 1..4_000_000
>>     h[i] = (s = s.succ)
>>   end

>> This ran relatively fast, but it consumed 265 megs of windows memory and
>> failed with out of memory error at around 3 million elements.

>> Why did Ruby fail at 265 megs and is there another way for me to do this?

>Four million strings, eh?  :)

><oldtimer_rant>
>Buncha young kids wanna allocate data structures with four million
>items... in my day, we had 64K of RAM and part of that was the OS
>and the runtime. The compiler wouldn't all fit in memory at once. It
>was swapped out to this clunky external Shugart floppy disk drive
>that would walk across the table if you didn't tape it down. It made a
>noise like a boa constrictor gagging on a weasel. And the compiler
>couldn't handle arrays bigger than a thousand elements.
>And we LIKED IT...
></oldtimer_rant>

And in my day we had 16K of RAM and no Shugart floppy, we saved our
programs and data on casette tapes, believe it or not, and your program
loaded from tape at about 600 baud... But the worst part was that the
programs we wrote were in BASIC or machine code (not assembly, we're
talking typing in the Hex for he opcodes - we considered ourselves lucky
to not have to toggle our programs in with switches :).  Of course we
walked several miles to school in the snow, too.  Uphill both ways ;-)

Phil



Sun, 04 Jul 2004 02:21:51 GMT  
 memory usage question

Quote:
> you should consider using a database

Doing 200 million lookups in a db would take way too long for my script.  If
there is no other work-around than using a DB, then I'll have to go back to
Java, which can handle this kind of memory usage.  I'm dissapointed that in
one of the first quick scripts I went to Ruby for, Ruby couldn't do the job.

Quote:
> <oldtimer_rant>

Don't try to compete with me on oldtimer stories.  I spent two summers
programming an IBM 650 which had NO core memory.  It was a 1-bit processor
with only drum memory.  Even the registers were on the drum.  It used about
100 vacumn tubes to implement the cpu.

Quote:
> Even today when RAM approaches a gigabyte in daily life

The server I'm running this on has 4 gigabytes of ram.

Quote:
-----Original Message-----

Sent: Tuesday, January 15, 2002 10:10 AM
To: ruby-talk ML
Subject: Re: memory usage question

----- Original Message -----


Sent: Tuesday, January 15, 2002 11:37 AM
Subject: memory usage question

> I need to write a script that will use a hash with 4 million strings of 16
> chars each.  As a test I wrote this quick script:

>   h = {}; s = 'aaaaaaaaaaaaaaaa'
>   for i in 1..4_000_000
>     h[i] = (s = s.succ)
>   end

> This ran relatively fast, but it consumed 265 megs of windows memory and
> failed with out of memory error at around 3 million elements.

> Why did Ruby fail at 265 megs and is there another way for me to do this?

Four million strings, eh?  :)

<oldtimer_rant>
Buncha young kids wanna allocate data structures with four million
items... in my day, we had 64K of RAM and part of that was the OS
and the runtime. The compiler wouldn't all fit in memory at once. It
was swapped out to this clunky external Shugart floppy disk drive
that would walk across the table if you didn't tape it down. It made a
noise like a boa constrictor gagging on a weasel. And the compiler
couldn't handle arrays bigger than a thousand elements.
And we LIKED IT...
</oldtimer_rant>

But seriously...  :)

Even today when RAM approaches a gigabyte in daily life (and your PC
may have a terabyte drive soon), memory ain't always the way to go.

Eventually you should consider using a database or something instead.
If you want something that will have a low impact on your code, try the
dbm interface. Very easy, not as much overhead as a "real" database.

There's also gdbm and maybe one or two other flavors. Not sure what
the differences are. Last time I tried, gdbm worked better for me on
Windows.

I think it's part of the standard distribution.

Hope this helps.

Hal



Sun, 04 Jul 2004 02:27:29 GMT  
 memory usage question

Quote:

>> you should consider using a database

>Doing 200 million lookups in a db would take way too long for my script.  If
>there is no other work-around than using a DB, then I'll have to go back to
>Java, which can handle this kind of memory usage.  I'm dissapointed that in
>one of the first quick scripts I went to Ruby for, Ruby couldn't do the job.

How does java acheive this feat?
(or how do you do it in java?)

Quote:

>> <oldtimer_rant>

>Don't try to compete with me on oldtimer stories.  I spent two summers
>programming an IBM 650 which had NO core memory.  It was a 1-bit processor
>with only drum memory.  Even the registers were on the drum.  It used about
>100 vacumn tubes to implement the cpu.

I think you win the prize unless there's a ENIAC user out there.

Phil



Sun, 04 Jul 2004 02:26:13 GMT  
 memory usage question

Quote:



>>> you should consider using a database

>>Doing 200 million lookups in a db would take way too long for my
>>script.  If there is no other work-around than using a DB, then I'll
>>have to go back to Java, which can handle this kind of memory usage.
>>I'm dissapointed that in one of the first quick scripts I went to
>>Ruby for, Ruby couldn't do the job.

> How does java acheive this feat?
> (or how do you do it in java?)

4 million 16 character strings is only 64 megs.  Even if the overhead
of a String object itself and the entry in the hash table is 4 times
that, you're still only at about 300 megs.  On a machine with 4 gigs
of RAM, that should be possible.

--
matt



Sun, 04 Jul 2004 02:54:19 GMT  
 memory usage question

Quote:

>>you should consider using a database

> Doing 200 million lookups in a db would take way too long for my script.  If
> there is no other work-around than using a DB, then I'll have to go back to
> Java, which can handle this kind of memory usage.  I'm dissapointed that in
> one of the first quick scripts I went to Ruby for, Ruby couldn't do the job.

Have you considered using a btree?  There is a btree class in Aruna (at
www.arunadb.org) that will allow you to set the page size such that only
  one I/O is needed to find the value you are looking for.

--
Michael Davis
Database Architect and Senior Software Engineer, Seva Inc.



Sun, 04 Jul 2004 02:54:19 GMT  
 memory usage question

Quote:



> >----- Original Message -----


> >Sent: Tuesday, January 15, 2002 11:37 AM
> >Subject: memory usage question

> >> I need to write a script that will use a hash with 4 million strings of 16
> >> chars each.  As a test I wrote this quick script:

> >>   h = {}; s = 'aaaaaaaaaaaaaaaa'
> >>   for i in 1..4_000_000
> >>     h[i] = (s = s.succ)
> >>   end

> >> This ran relatively fast, but it consumed 265 megs of windows memory and
> >> failed with out of memory error at around 3 million elements.

> >> Why did Ruby fail at 265 megs and is there another way for me to do this?

> >Four million strings, eh?  :)

> ><oldtimer_rant>
> >Buncha young kids wanna allocate data structures with four million
> >items... in my day, we had 64K of RAM and part of that was the OS
> >and the runtime. The compiler wouldn't all fit in memory at once. It
> >was swapped out to this clunky external Shugart floppy disk drive
> >that would walk across the table if you didn't tape it down. It made a
> >noise like a boa constrictor gagging on a weasel. And the compiler
> >couldn't handle arrays bigger than a thousand elements.
> >And we LIKED IT...
> ></oldtimer_rant>

> And in my day we had 16K of RAM and no Shugart floppy, we saved our
> programs and data on casette tapes, believe it or not, and your program
> loaded from tape at about 600 baud... But the worst part was that the
> programs we wrote were in BASIC or machine code (not assembly, we're
> talking typing in the Hex for he opcodes - we considered ourselves lucky
> to not have to toggle our programs in with switches :).  Of course we
> walked several miles to school in the snow, too.  Uphill both ways ;-)

> Phil

Ahhh, That brings back the senior project for my Aerospace Engineering
degree.  I was manually swapping out computer fluid dynamics grids and
coeffecients to files as I completed calculations on them on my dos
based x86 with 4MB.  IIRC, the data for my entire grid was 800MB in
total...  I think running that for three or four days straight led to
the premature death of my hard drive in that machine...

-- alan

--
Alan Chen
Digikata LLC
http://digikata.com



Sun, 04 Jul 2004 02:59:40 GMT  
 memory usage question

Quote:

> > you should consider using a database

> Doing 200 million lookups in a db would take way too long for my script.  If
> there is no other work-around than using a DB, then I'll have to go back to
> Java, which can handle this kind of memory usage.  I'm dissapointed that in
> one of the first quick scripts I went to Ruby for, Ruby couldn't do the job.

> > <oldtimer_rant>

It might be worth a try.  A quick ruby script connecting to a
PostgreSQL or some similiar DB may suprise you. I would think that the
DB has a more sophisticated method of managing memory and lookups than
I would ever have the time to research and implement.

--
Alan Chen
Digikata LLC
http://digikata.com



Sun, 04 Jul 2004 02:59:41 GMT  
 memory usage question

Quote:




> >>> you should consider using a database

> >>Doing 200 million lookups in a db would take way too long for my
> >>script.  If there is no other work-around than using a DB, then I'll
> >>have to go back to Java, which can handle this kind of memory usage.
> >>I'm dissapointed that in one of the first quick scripts I went to
> >>Ruby for, Ruby couldn't do the job.

> > How does java acheive this feat?
> > (or how do you do it in java?)

> 4 million 16 character strings is only 64 megs.

They're not wide characters in java?

So you're saying that Ruby strings take up more space? (probably, since
doesn't ruby use wide chars?)

What do these 16 character strings represent?

Quote:
> Even if the overhead
> of a String object itself and the entry in the hash table is 4 times
> that, you're still only at about 300 megs.  On a machine with 4 gigs
> of RAM, that should be possible.

Yes, but I think that's a lot more available RAM than you mentioned in
your original post.  From what you said earlier if you 4gigs of RAM then
the Ruby version should work without any problem (am I missing something?)

Phil



Sun, 04 Jul 2004 03:05:02 GMT  
 memory usage question
Quote:
----- Original Message -----


Sent: Tuesday, January 15, 2002 12:25 PM
Subject: RE: memory usage question

> Doing 200 million lookups in a db would take way too long for my script.
If
> there is no other work-around than using a DB, then I'll have to go back
to
> Java, which can handle this kind of memory usage.  I'm dissapointed that
in
> one of the first quick scripts I went to Ruby for, Ruby couldn't do the

job.

I can see where you would be disappointed.

On the other hand, my "quick scripts" in the past have never involved
four million of anything in memory. Obviously your mileage varies.

I hope, by the way, that my flippant tone did not annoy you. That
wasn't the intention.

Quote:
> Don't try to compete with me on oldtimer stories.  I spent two summers
> programming an IBM 650 which had NO core memory.  It was a 1-bit processor
> with only drum memory.  Even the registers were on the drum.  It used
about
> 100 vacumn tubes to implement the cpu.

I graciously concede defeat.

Quote:
> > Even today when RAM approaches a gigabyte in daily life

> The server I'm running this on has 4 gigabytes of ram.

By daily life, I meant on the average desktop.

Still, I'm impressed with the 4 gig... I guess I am out of the loop,
because I didn't know that Windows boxes anywhere were
running with that much.

Hal



Sun, 04 Jul 2004 03:10:30 GMT  
 memory usage question


Quote:

> I need to write a script that will use a hash with 4 million strings of 16
> chars each.  As a test I wrote this quick script:

>   h = {}; s = 'aaaaaaaaaaaaaaaa'
>   for i in 1..4_000_000
>     h[i] = (s = s.succ)
>   end

> This ran relatively fast, but it consumed 265 megs of windows memory and
> failed with out of memory error at around 3 million elements.

> Why did Ruby fail at 265 megs and is there another way for me to do this?

Interesting, I tried it under ruby 1.6.2 (2000-12-18) [i386-cygwin]
and got a failure at the same point (~265 megs), but instead of an
out-of-memory error, I got:

  /cygdrive/t/test.rb:5: [BUG] Segmentation fault
  ruby 1.6.2 (2000-12-18) [i386-cygwin]
        0 [sig] ruby 1336 open_stackdumpfile: Dumping stack trace to
  ruby.exe.stackdump

Whereas under ruby 1.6.6 (2001-12-26) [i586-mswin32], the script
worked, allocating well over 300M, and completing successfully.

That [i586-mswin32] version also completed the task about 20x faster
than the cygwin version. . . . My cygwin1.dll is 1003.2.0.0.

It would seem cygwin's memory allocation may be subtly broken?

Regards,

Bill



Sun, 04 Jul 2004 03:15:48 GMT  
 memory usage question

Quote:





>> >>> you should consider using a database

>> >>Doing 200 million lookups in a db would take way too long for my
>> >>script.  If there is no other work-around than using a DB, then I'll
>> >>have to go back to Java, which can handle this kind of memory usage.
>> >>I'm dissapointed that in one of the first quick scripts I went to
>> >>Ruby for, Ruby couldn't do the job.

>> > How does java acheive this feat?
>> > (or how do you do it in java?)

>> 4 million 16 character strings is only 64 megs.

> They're not wide characters in java?

> So you're saying that Ruby strings take up more space? (probably,
> since doesn't ruby use wide chars?)

> What do these 16 character strings represent?

You're right, think Java does us 16 bit Unicode chars.  Ruby doesn't.
It uses UTF-8 or SJIS, etc.  These all use one byte for an ASCII
character and multiple byte encodings for others.

Quote:
>> Even if the overhead of a String object itself and the entry in the
>> hash table is 4 times that, you're still only at about 300 megs.
>> On a machine with 4 gigs of RAM, that should be possible.

> Yes, but I think that's a lot more available RAM than you mentioned
> in your original post.  From what you said earlier if you 4gigs of
> RAM then the Ruby version should work without any problem (am I
> missing something?)

Exactly, but the original poster says it doesn't work -- that is the
whole question.

--
matt



Sun, 04 Jul 2004 03:15:49 GMT  
 memory usage question

Quote:
> How does java acheive this feat?

I only know I've done this before in Java without running into trouble.  My
guess is that Java uses less memory per string in the hash and it allows
large heap allocations.  I've never run into a heap size limit, but I assume
it is around a gig or two.  A 32 bit cpu could theoretically address 4 gigs.
Quote:
-----Original Message-----

Sent: Tuesday, January 15, 2002 10:34 AM
To: ruby-talk ML; undisclosed-recipients:
Subject: Re: memory usage question



>> you should consider using a database

>Doing 200 million lookups in a db would take way too long for my script.
If
>there is no other work-around than using a DB, then I'll have to go back to
>Java, which can handle this kind of memory usage.  I'm dissapointed that in
>one of the first quick scripts I went to Ruby for, Ruby couldn't do the
job.

How does java acheive this feat?
(or how do you do it in java?)

>> <oldtimer_rant>

>Don't try to compete with me on oldtimer stories.  I spent two summers
>programming an IBM 650 which had NO core memory.  It was a 1-bit processor
>with only drum memory.  Even the registers were on the drum.  It used about
>100 vacumn tubes to implement the cpu.

I think you win the prize unless there's a ENIAC user out there.

Phil



Sun, 04 Jul 2004 03:26:31 GMT  
 memory usage question

Quote:
> From what you said earlier if you 4gigs of RAM then the Ruby version

should work without any problem

I included the code in my original post.  Feel free to try it yourself.  It
should run on any PC with virtual memory over 300 megs.  I ran it on my
development PC, which has 500 megs of ram and 1.3 gigs of virtual memory.

Quote:
-----Original Message-----

Sent: Tuesday, January 15, 2002 11:01 AM
To: ruby-talk ML; Matt Armstrong

Cc: ruby-talk ML
Subject: Re: memory usage question





> >>> you should consider using a database

> >>Doing 200 million lookups in a db would take way too long for my
> >>script.  If there is no other work-around than using a DB, then I'll
> >>have to go back to Java, which can handle this kind of memory usage.
> >>I'm dissapointed that in one of the first quick scripts I went to
> >>Ruby for, Ruby couldn't do the job.

> > How does java acheive this feat?
> > (or how do you do it in java?)

> 4 million 16 character strings is only 64 megs.

They're not wide characters in java?

So you're saying that Ruby strings take up more space? (probably, since
doesn't ruby use wide chars?)

What do these 16 character strings represent?

> Even if the overhead
> of a String object itself and the entry in the hash table is 4 times
> that, you're still only at about 300 megs.  On a machine with 4 gigs
> of RAM, that should be possible.

Yes, but I think that's a lot more available RAM than you mentioned in
your original post.  From what you said earlier if you 4gigs of RAM then
the Ruby version should work without any problem (am I missing something?)

Phil



Sun, 04 Jul 2004 03:31:51 GMT  
 
 [ 30 post ]  Go to page: [1] [2] [3]

 Relevant Pages 

1. Memory usage questions

2. tcl Memory Usage Question

3. win32 memory usage question

4. question about memory usage of a script

5. Memory usage of PHP script/arrays and memory restriction/error

6. optimizing memory usage of large in memory structures

7. memory usage (how to debug a memory leak?)

8. Memory usage under J 5.1 /386 (PC)

9. RISC vs. CISC memory usage

10. SYNCDSM and paging/memory usage monitoring

11. Reducing memory usage

12. VW2.0 & Real Memory Usage

 

 
Powered by phpBB® Forum Software