memory-mapped Numeric arrays: arrayfrombuffer version 2 
Author Message
 memory-mapped Numeric arrays: arrayfrombuffer version 2

The 'arrayfrombuffer' package features support for Numerical python
arrays whose contents are stored in buffer objects, including
memory-mapped files.  This has the following advantages:

- loading your array from a file is easy --- a module import and a
  single function call --- and doesn't use excessive amounts of
  memory.
- loading your array is quick; it doesn't need to be copied from one
  part of memory to another in order to be loaded.
- your array gets demand-loaded; parts you aren't using don't need to
  be in memory or in swap.
- under memory-pressure conditions, your array doesn't use up swap,
  and parts of it you haven't modified can be evicted from RAM without
  the need for a disk write
- your arrays can be bigger than your physical memory
- when you modify your array, only the parts you modify get written
  back out to disk

This is something that's been requested on the Numpy list a few times
a year since 1999.

arrayfrombuffer lives at
http://www.*-*-*.com/ ~kragen/sw/arrayfrombuffer/
The current version is version 2; it is released under the X11 license
(the BSD license without the advertising clause).


<P><A
HREF=" http://www.*-*-*.com/ ~kragen/sw/arrayfrombuffer/">arrayfrombuffer
2</A> - creates Numeric arrays from memory-mapped files.  (23-Jan-02)



Mon, 12 Jul 2004 13:39:35 GMT  
 
 [ 1 post ] 

 Relevant Pages 

1. memory-mapped Numeric arrays: arrayfrombuffer version 2

2. Convert 1D string Array to Numeric Array

3. ARRAY[NUMERIC] := ARRAY[DOUBLE] -- where?

4. Reserved memory and memory mapped I/O

5. Numeric 2d Array to 3d Array

6. Tcl array - an array or a map?

7. Memory usage of PHP script/arrays and memory restriction/error

8. Version 6 Waveform versus Version 5 2D array in the AI READ vi

9. Memory address of a numeric command

10. list math for version 1.5.2 without Numeric

11. Get the version of IMSL/Fortran Mathematics and Statistics and NAG/Fortran Numeric Algorithms Library Library

12. ELS95: NUMERIC, REAL, DOUBLE, ARRAY

 

 
Powered by phpBB® Forum Software