best way to read last line?
Author |
Message |
rog #1 / 31
|
 best way to read last line?
Hello, not sure on this one - what to do? yes yes, ive read the faq, camel, cookbook and searched deja, but am still not sure on the best way to do this. What is the most efficient & fastest way of reading the last line of a flat file (small or large)? Ive heard these solutions: 1) Read the whole file into an array, then process. (how well would this work for large files???) 2) Open the file, seek to the end, seek back a little, use sysread to read the part you seeked back over. If it contains a newline, you now know the part you can read, else you seek back a bit more, rinse and repeat. 3) using tell() and truncate(): open (FH, "+< $file"); while ( <FH> ) { $addr = tell(FH) unless eof(FH) } truncate(FH, $addr); This deletes the last line of a file without making a copy or reading the whole file into memory, but how can it be used to return the last line? (then i guess you could process it then append it back on - quick?) 4) is there a tail function for perl? as in unix? where the end is read only? (perhaps I could then tell/truncate??) I want to try and avoid reading the whole file just to find the end, as presumably this could take ages with a large file. Any opinons on this?? much appreciated, rog
|
Wed, 18 Jun 1902 08:00:00 GMT |
|
 |
Bill Mosele #2 / 31
|
 best way to read last line?
remarked... Quote: > What is the most efficient & fastest way of reading the last line of a flat > file (small or large)?
is print `tail -1 $file`; not efficient for your use? --
pls note the one line sig, not counting this one.
|
Wed, 18 Jun 1902 08:00:00 GMT |
|
 |
Martien Verbrugg #3 / 31
|
 best way to read last line?
On Sat, 1 Jan 2000 20:34:34 -0000,
Quote: > Hello, not sure on this one - what to do? > yes yes, ive read the faq, camel, cookbook and searched deja, but am still > not sure on the best way to do this.
Not all of them, but that is possible. Quote: > 4) is there a tail function for perl? as in unix? where the end is read > only? (perhaps I could then tell/truncate??)
You had the right question though: # perldoc -q tail =head1 Found in /opt/perl5.00503/lib/5.00503/pod/perlfaq5.pod =head2 How do I do a C<tail -f> in perl? As far as 'which of the methods is best?' goes. Probably depends. I would just code for the general case, which disallows slurping in the whole file (what if it's 3 GB large?). The Perl Power Tools also come with a tail. Have a look at that. http://language.perl.com/ Martien -- Martien Verbruggen | Interactive Media Division | The gene pool could use a little Commercial Dynamics Pty. Ltd. | chlorine. NSW, Australia |
|
Wed, 18 Jun 1902 08:00:00 GMT |
|
 |
Martien Verbrugg #4 / 31
|
 best way to read last line?
Quote:
> remarked... > > What is the most efficient & fastest way of reading the last line of > > a flat file (small or large)? > is print `tail -1 $file`; not efficient for your use?
Well.. it isn't very portable, and probably it's slower than doing some seeks yourself. No benchmarks, but spawning processes can be expensive. Martien -- Martien Verbruggen | Interactive Media Division | For heaven's sake, don't TRY to be Commercial Dynamics Pty. Ltd. | cynical. It's perfectly easy to be NSW, Australia | cynical.
|
Wed, 18 Jun 1902 08:00:00 GMT |
|
 |
Cameron Lai #5 / 31
|
 best way to read last line?
Quote:
>Hello, not sure on this one - what to do? >yes yes, ive read the faq, camel, cookbook and searched deja, but am still >not sure on the best way to do this. >What is the most efficient & fastest way of reading the last line of a flat >file (small or large)?
. . . Quote: >I want to try and avoid reading the whole file just to find the end, as >presumably this could take ages with a large file.
... but it might be necessary, right? What if it's a large file, but all within a single line? I recommend running your own tests. It's good to profit from the experience of others. However, cod- ing each of the possibilities you're already mentioned is so quick in Perl that, if the differ- ences in performance truly matter to you, your time will be will spent on measuring their performance yourself. Please do consider the `tail -1 ...` alternative. I suspect that Mr. Verbruggen's reservations about it might not apply in your situation. --
Business: http://www.Phaseit.net Personal: http://starbase.neosoft.com/~claird/home.html
|
Wed, 18 Jun 1902 08:00:00 GMT |
|
 |
sublym.. #6 / 31
|
 best way to read last line?
Quote: > Hello, not sure on this one - what to do? > yes yes, ive read the faq, camel, cookbook and searched deja, but am still > not sure on the best way to do this. > What is the most efficient & fastest way of reading the last line of a flat > file (small or large)? > Ive heard these solutions: > 1) Read the whole file into an array, then process. (how well would this > work for large files???) > 2) Open the file, seek to the end, seek back a little, use sysread to read > the part you seeked back over. If it contains a newline, you now know the > part you can read, else you seek back a bit more, rinse and repeat. > 3) using tell() and truncate(): > open (FH, "+< $file"); > while ( <FH> ) { $addr = tell(FH) unless eof(FH) } > truncate(FH, $addr); > This deletes the last line of a file without making a copy or reading the > whole file into memory, but how can it be used to return the last line? > (then i guess you could process it then append it back on - quick?) > 4) is there a tail function for perl? as in unix? where the end is read > only? (perhaps I could then tell/truncate??) > I want to try and avoid reading the whole file just to find the end, as > presumably this could take ages with a large file. > Any opinons on this?? > much appreciated, > rog
I've used this method with reasonable success: open the FH with a pipe to/from tail -f. in a loop, call seek (FH,0,1) --this causes the EOF to be cleared, still enabling you to "tail" the file. Sent via Deja.com http://www.deja.com/ Before you buy.
|
Wed, 18 Jun 1902 08:00:00 GMT |
|
 |
David Cassel #7 / 31
|
 best way to read last line?
[huge snip] Quote: > I've used this method with reasonable success: > open the FH with a pipe to/from tail -f. > in a loop, call seek (FH,0,1) --this causes the EOF to be cleared, > still enabling you to "tail" the file.
Errm, is there a good reason why you: [1] quoted the *entire* original post with no t{*filter*}; [2] gave a sub-optimal answer; and also [3] totally ignored what the FAQ has to say about this? Try this at a command prompt: perldoc -q tail David --
Senior Computing Specialist mathematical statistician
|
Wed, 18 Jun 1902 08:00:00 GMT |
|
 |
#8 / 31
|
 best way to read last line?
|
Wed, 18 Jun 1902 08:00:00 GMT |
|
 |
David Cassel #9 / 31
|
 best way to read last line?
Quote:
> So here is a burning question on reading the last line of a file that > has been bugging me for months.... > This WILL work > open(FILE, '/dir/file.file') or die "Can't read file.file, $!\n"; > $line = (<FILE>)[-1]; > close FILE or warn "Can't close file.file, $!\n"; > print "$line\n";
Checks on open(0 and close() . Good. Have you already declared $line to have a lexical scope, or are you making it a global? Quote: > # prints last line or (<FILE>)[5] prints... well, you know.. > BUT! > Is the whole file being read into memory so it can be treated as an > array, or is Perl counting $/ and then pulling the proper line or??? > I haven't found any docs that tell me what's going on here. I may be > looking in the wrong places of course.
Think context. Then you'll see that you are looking in the wrong places. What does the (<FILE>) do? You end up slurping in the entire file, then discarding all but one line of it. Now then, you have reached the end of the file. What should happen if you try to read another line? Quote: > Interestingly, this code will ONLY grab one line from the file, attempts > to run a loop and get more than one line seem doomed to failure.
Exactly! Perl knows you're at EOF and can't fetch another line for you. If you want to start over at the beginning, you have to seek() to the start of the file. Or close() and open() the file again, which is less efficient. [snip of code showing this] Quote: > will ONLY grab the first element, subsequent arguments are ignored. > 'Course I could be doing something wrong. :-)
I'll please the Fifth here. :-) Quote: > I've used this once or twice on small files on (shudder) win32 where I
Hey! Some of my best friends know people who have used win32 !! Actually, I find that ActiveState Perl on win32 and the Perl Power Tools can make working on win32 platforms much nicer. There's a tail program in there, IIRC... Quote: > didn't have tail handy and I needed a specific line, but have always > been hesitant to use it on bigger files. And a bit hesitant to use it in > production code since I don't really understand exactly what's going on.
Good, and good. If your files are big enough, the slurp will hurt.. right about the time you have to start swapping virtual memory. And if there's something about Perl you don't exactly, precisely understand.. then join the crowd. :-) Quote: > Interested in comments.
You may get a few! David --
Senior Computing Specialist mathematical statistician
|
Wed, 18 Jun 1902 08:00:00 GMT |
|
 |
#10 / 31
|
 best way to read last line?
|
Wed, 18 Jun 1902 08:00:00 GMT |
|
 |
Abiga #11 / 31
|
 best way to read last line?
MCMXCIII in <URL::">
{} > {} > What is the most efficient & fastest way of reading the last line of {} > a flat file (small or large)? {} > {} > 2) Open the file, seek to the end, seek back a little, use sysread {} > to read the part you seeked back over. If it contains a newline, {} > you now know the part you can read, else you seek back a bit more, {} > rinse and repeat. {} {} Here is an example implementing approach 2). It is ugly but it seems {} to be working and it is efficient enough for my purposes: {} {} {} [80+ lines snipped] {} {} {} my $filename = $ARGV[0]; {} my $lines = $ARGV[1]; {}
And in considerably less lines:
Abigail --
"\150\145\162\040\120\145\162\154\040\110\141\143\153\145\162".
-----------== Posted via Newsfeeds.Com, Uncensored Usenet News ==---------- http://www.newsfeeds.com The Largest Usenet Servers in the World! ------== Over 73,000 Newsgroups - Including Dedicated Binaries Servers ==-----
|
Wed, 18 Jun 1902 08:00:00 GMT |
|
 |
Bill Mosele #12 / 31
|
 best way to read last line?
Quote: > And in considerably less lines:
This thread has gone full circle now. Does that mean the thread is not done? --
pls note the one line sig, not counting this one.
|
Wed, 18 Jun 1902 08:00:00 GMT |
|
 |
#13 / 31
|
 best way to read last line?
|
Wed, 18 Jun 1902 08:00:00 GMT |
|
 |
Steven Ma #14 / 31
|
 best way to read last line?
So here is a burning question on reading the last line of a file that has been bugging me for months.... This WILL work open(FILE, '/dir/file.file') or die "Can't read file.file, $!\n"; $line = (<FILE>)[-1]; close FILE or warn "Can't close file.file, $!\n"; print "$line\n"; # prints last line or (<FILE>)[5] prints... well, you know.. BUT! Is the whole file being read into memory so it can be treated as an array, or is Perl counting $/ and then pulling the proper line or??? I haven't found any docs that tell me what's going on here. I may be looking in the wrong places of course. Interestingly, this code will ONLY grab one line from the file, attempts to run a loop and get more than one line seem doomed to failure. Something like:
open(FILE, '/dir/file.file') or die "Can't read file.file, $!\n";
close FILE or warn "Can't close file.file, $!\n"; or open(FILE, '/dir/file.file') or die "Can't read file.file, $!\n"; $string .= (<FILE>)[1],"\n"; $string .= (<FILE>)[2],"\n"; close FILE or warn "Can't close file.file, $!\n"; will ONLY grab the first element, subsequent arguments are ignored. 'Course I could be doing something wrong. :-) I've used this once or twice on small files on (shudder) win32 where I didn't have tail handy and I needed a specific line, but have always been hesitant to use it on bigger files. And a bit hesitant to use it in production code since I don't really understand exactly what's going on. Interested in comments. Steve
|
Wed, 18 Jun 1902 08:00:00 GMT |
|
 |
Martien Verbrugg #15 / 31
|
 best way to read last line?
On Tue, 04 Jan 2000 00:33:12 GMT,
Quote: > So here is a burning question on reading the last line of a file that > has been bugging me for months.... > This WILL work > open(FILE, '/dir/file.file') or die "Can't read file.file, $!\n"; > $line = (<FILE>)[-1]; > close FILE or warn "Can't close file.file, $!\n"; > print "$line\n"; > # prints last line or (<FILE>)[5] prints... well, you know.. > BUT! > Is the whole file being read into memory so it can be treated as an > array, or is Perl counting $/ and then pulling the proper line or???
Yes. No. But: my $line; $line = $_ while(<FILE>); only reads in one line at a time. Of course, it still does an assignment for each line in the file, and it has to scan the whole file. Something like this avoids the assignments: while(<FILE>) { next unless eof FILE; $line = $_; Quote: }
but it has to do a test on each iteration. (A benchmark might show what's best. but I don't feel like doing it). And it still scans the whole file. Probably it depends on the size of the file you want to open what's most efficient (a seek to the end and jumping backwards, slurping in the file, scanning each line until the last). You could have a look to see how this is implemented in the tail that comes with the Perl Power Tools: http://language.perl.com/ppt. (It seems that that tail always scans the file if it's text, so maybe it was judged to be the best general case). Martien PS. The sig is a fluke. Randomly but appropriately picked. -- Martien Verbruggen | Interactive Media Division | Begin at the beginning and go on till Commercial Dynamics Pty. Ltd. | you come to the end; then stop. NSW, Australia |
|
Wed, 18 Jun 1902 08:00:00 GMT |
|
|
Page 1 of 3
|
[ 31 post ] |
|
Go to page:
[1]
[2] [3] |
|