how to read a huge file line by line without loading it into memory 
Author Message
 how to read a huge file line by line without loading it into memory

Quote:

> I have two very large files (each greater than the amount of memory on my
> box) -  I need to perform a two file match, but the only way I know in perl
> is to load the files up into memory with Open.

open() does not load the file into memory.  It initializes a filehandle
and associates it with a file on disk.  What you do with it after that
point will effect your memory usage.

Quote:
> How can I read from each file one line at a time, or even better a block of N
> lines at a time?

while(<FILEHANDLE>){
        # WooHoo!!

Quote:
}

For all the exciting detail on I/O operators,  open(), angle brackets,
and other file oriented functions, read the perlop and perlfunc
documents that came with your perl distribution.

Good Luck!

--
Jerome O'Neil, Operations and Information Services
Atrieva Corporation, 600 University St., Ste. 911, Seattle, WA 98101

The Atrieva Service: Safe and Easy Online Backup   http://www.*-*-*.com/



Sat, 13 Oct 2001 03:00:00 GMT  
 how to read a huge file line by line without loading it into memory

Quote:
> I have two very large files (each greater than the amount of memory on my
> box) -     I need to perform a two file match, but the only way I know in perl
> is to load the files up into memory with Open.

> How can I read from each file one line at a time, or even better a block of N
> lines at a time?

Sorry, something other than reading the file a line at a time like:

while (<FILEHANDLE> )
  {
    # Blah
  }

Or am I misunderstanding something here ?

/J\
--

Some of your questions answered:
<URL:http://www.btinternet.com/~gellyfish/resources/wwwfaq.htm>
Hastings: <URL:http://www.newhoo.com/Regional/UK/England/East_Sussex/Hastings>



Sat, 13 Oct 2001 03:00:00 GMT  
 
 [ 3 post ] 

 Relevant Pages 

1. read last line without reading previous lines, how?

2. How do I read line by line of data from a file using perl

3. reading lines from one file and pulling matching lines in another

4. newbie perl question - reading a file line by line

5. Line by line file reading

6. How to read a file line by line w/loop

7. Read remote text file line by line

8. reading a file line by line backwards

9. single line regex and multi-line regex without resetting $/

10. Print lines without new-line

11. how to read in line by line?

12. read line by line

 

 
Powered by phpBB® Forum Software