Large arrays in Unix 
Author Message
 Large arrays in Unix

        Note: if you are reading this after Midnight EDT, August 14,
1993, then please don't respond; by that time I'll have probably
gotten two dozen replies and I will have posted a summary of that info
to the net for general perusal.  (Unless, of course, you know of some
technique that works especially well and that nobody else is likely to
have thought of...in that case by all means respond! :-) )

        I have an efficiency question.  I'm using an SGI Iris Indigo
running Irix 4.0.5H and compiling with SGI's cc.  Later this afternoon
or this evening I'll be writing a program that does an statistical
analysis on two sets of data, each containing over 5,000,000
datapoints (all floats), which, in the simplest of terms, means I'll
need to allocate over 40 megs of memory to hold the data.  Since my
machine has only 32 megs of RAM, this means that the machine is going
to need to swap.  (If it'll let me allocate that much space at all!)

        Swapping takes lots of time, and I want to avoid it as much as
possible, since proccessing that many datapoints is going to take a
while as it is.  My question is this: How can I allocate the space for
that data while requiring a minimum of swapping?

        My first though was to allocate 10 distinct arrays of
1,000,000 points, (ex: *aa, *ab, *ac ... *aj) and access them one at a
time.  That way the entire array could be held in RAM while the other
nine arrays were swapped out to disk.

        There must be other ways of controlling how and when swapping
takes place, but they aren't described in the ANSI standard, so I'm
sure they're not portable.  That really isn't an issue; I just want to
know how to do it, if it's possible.  Should I let malloc() deal with
it and not worry about it?  Are there other issues that I should worry
about?

Thanks in advance for all your help!
Email replies are appreciated; I'll summarize for the group.
--
Lee Silverman, Brown GeoPhysics ScB '94, ScM '95
Currently Summer Student Fellow at WHOI


"Nonsense - you only say it's impossible because nobody's ever done it."



Mon, 29 Jan 1996 20:04:57 GMT  
 
 [ 1 post ] 

 Relevant Pages 

1. Help: Large 3-d Arrays, work in unix, not in dos

2. Help: Large 3-d Arrays, work in unix, not in dos

3. Help !!! Concat small array into large array

4. Arrays [256][256] --- error Array size too large!

5. Declaring Large Arrays in C#

6. problem with large arrays

7. problem with large arrays

8. large array problem?

9. Creating large C# Null Arrays

10. Passing large byte-array from C# to Delphi DLL

11. Why is dealing with large arrays so slow?

12. very large arrays/pointer

 

 
Powered by phpBB® Forum Software