best way to read last line? 
Author Message
 best way to read last line?

Hello, not sure on this one - what to do?

yes yes, ive read the faq, camel, cookbook and searched deja, but am still
not sure on the best way to do this.

What is the most efficient & fastest way of reading the last line of a flat
file (small or large)?

Ive heard these solutions:
1) Read the whole file into an array, then process. (how well would this
work for large files???)
2) Open the file, seek to the end, seek back a little, use sysread to read
the part you seeked back over. If it contains a newline, you now know the
part you can read, else you seek back a bit more, rinse and repeat.
3) using tell() and truncate():
        open (FH, "+< $file");
        while ( <FH> ) { $addr = tell(FH) unless eof(FH) }
        truncate(FH, $addr);
This deletes the last line of a file without making a copy or reading the
whole file into memory, but how can it be used to return the last line?
(then i guess you could process it then append it back on - quick?)
4) is there a tail function for perl? as in unix? where the end is read
only? (perhaps I could then tell/truncate??)

I want to try and avoid reading the whole file just to find the end, as
presumably this could take ages with a large file.

Any opinons on this??

much appreciated,
rog



Wed, 18 Jun 1902 08:00:00 GMT  
 best way to read last line?

remarked...

Quote:
> What is the most efficient & fastest way of reading the last line of a flat
> file (small or large)?

is print `tail -1 $file`; not efficient for your use?

--

pls note the one line sig, not counting this one.



Wed, 18 Jun 1902 08:00:00 GMT  
 best way to read last line?
On Sat, 1 Jan 2000 20:34:34 -0000,

Quote:
> Hello, not sure on this one - what to do?

> yes yes, ive read the faq, camel, cookbook and searched deja, but am still
> not sure on the best way to do this.

Not all of them, but that is possible.

Quote:
> 4) is there a tail function for perl? as in unix? where the end is read
> only? (perhaps I could then tell/truncate??)

You had the right question though:

# perldoc -q tail
=head1 Found in /opt/perl5.00503/lib/5.00503/pod/perlfaq5.pod

=head2 How do I do a C<tail -f> in perl?

As far as 'which of the methods is best?' goes. Probably depends. I
would just code for the general case, which disallows slurping in the
whole file (what if it's 3 GB large?).

The Perl Power Tools also come with a tail. Have a look at that.
http://language.perl.com/

Martien
--
Martien Verbruggen              |
Interactive Media Division      | The gene pool could use a little
Commercial Dynamics Pty. Ltd.   | chlorine.
NSW, Australia                  |



Wed, 18 Jun 1902 08:00:00 GMT  
 best way to read last line?


Quote:

> remarked...
> > What is the most efficient & fastest way of reading the last line of
> > a flat file (small or large)?

> is print `tail -1 $file`; not efficient for your use?

Well.. it isn't very portable, and probably it's slower than doing some
seeks yourself. No benchmarks, but spawning processes can be expensive.

Martien
--
Martien Verbruggen              |
Interactive Media Division      | For heaven's sake, don't TRY to be
Commercial Dynamics Pty. Ltd.   | cynical. It's perfectly easy to be
NSW, Australia                  | cynical.



Wed, 18 Jun 1902 08:00:00 GMT  
 best way to read last line?
Quote:

>Hello, not sure on this one - what to do?

>yes yes, ive read the faq, camel, cookbook and searched deja, but am still
>not sure on the best way to do this.

>What is the most efficient & fastest way of reading the last line of a flat
>file (small or large)?

                        .
                        .
                        .
Quote:
>I want to try and avoid reading the whole file just to find the end, as
>presumably this could take ages with a large file.

... but it might be necessary, right?  What if it's
a large file, but all within a single line?

I recommend running your own tests.  It's good to
profit from the experience of others.  However, cod-
ing each of the possibilities you're already
mentioned is so quick in Perl that, if the differ-
ences in performance truly matter to you, your time
will be will spent on measuring their performance
yourself.

Please do consider the `tail -1 ...` alternative.
I suspect that Mr. Verbruggen's reservations about
it might not apply in your situation.
--


Business:  http://www.Phaseit.net
Personal:  http://starbase.neosoft.com/~claird/home.html



Wed, 18 Jun 1902 08:00:00 GMT  
 best way to read last line?


Quote:
> Hello, not sure on this one - what to do?

> yes yes, ive read the faq, camel, cookbook and searched deja, but am
still
> not sure on the best way to do this.

> What is the most efficient & fastest way of reading the last line of a
flat
> file (small or large)?

> Ive heard these solutions:
> 1) Read the whole file into an array, then process. (how well would
this
> work for large files???)
> 2) Open the file, seek to the end, seek back a little, use sysread to
read
> the part you seeked back over. If it contains a newline, you now know
the
> part you can read, else you seek back a bit more, rinse and repeat.
> 3) using tell() and truncate():
>         open (FH, "+< $file");
>         while ( <FH> ) { $addr = tell(FH) unless eof(FH) }
>         truncate(FH, $addr);
> This deletes the last line of a file without making a copy or reading
the
> whole file into memory, but how can it be used to return the last
line?
> (then i guess you could process it then append it back on - quick?)
> 4) is there a tail function for perl? as in unix? where the end is
read
> only? (perhaps I could then tell/truncate??)

> I want to try and avoid reading the whole file just to find the end,
as
> presumably this could take ages with a large file.

> Any opinons on this??

> much appreciated,
> rog

I've used this method with reasonable success:
open the FH with a pipe to/from tail -f.
in a loop, call seek (FH,0,1)     --this causes the EOF to be cleared,
still enabling you to "tail" the file.

Sent via Deja.com http://www.deja.com/
Before you buy.



Wed, 18 Jun 1902 08:00:00 GMT  
 best way to read last line?
[huge snip]

Quote:
> I've used this method with reasonable success:
> open the FH with a pipe to/from tail -f.
> in a loop, call seek (FH,0,1)     --this causes the EOF to be cleared,
> still enabling you to "tail" the file.

Errm, is there a good reason why you:
[1] quoted the *entire* original post with no t{*filter*};
[2] gave a sub-optimal answer; and also
[3] totally ignored what the FAQ has to say about this?

Try this at a command prompt:

perldoc -q tail

David
--

Senior Computing Specialist
mathematical statistician



Wed, 18 Jun 1902 08:00:00 GMT  
 best way to read last line?


Wed, 18 Jun 1902 08:00:00 GMT  
 best way to read last line?

Quote:

> So here is a burning question on reading the last line of a file that
> has been bugging me for months....

> This WILL work

> open(FILE, '/dir/file.file') or die "Can't read file.file, $!\n";
>  $line = (<FILE>)[-1];
> close FILE  or warn "Can't close file.file, $!\n";

> print "$line\n";

Checks on open(0 and close() .  Good.  Have you already declared $line
to have a lexical scope, or are you making it a global?

Quote:
> # prints last line or (<FILE>)[5] prints... well, you know..

> BUT!

> Is the whole file being read into memory so it can be treated as an
> array, or is Perl counting $/ and then pulling the proper line or???

> I haven't found any docs that tell me what's going on here.  I may be
> looking in the wrong places of course.

Think context.  Then you'll see that you are looking in the wrong
places.  What does the (<FILE>) do?  You end up slurping in the entire
file, then discarding all but one line of it.  Now then, you have
reached the end of the file.  What should happen if you try to read
another line?

Quote:
> Interestingly, this code will ONLY grab one line from the file, attempts
> to run a loop and get more than one line seem doomed to failure.

Exactly!  Perl knows you're at EOF and can't fetch another line for
you.  If you want to start over at the beginning, you have to seek()
to the start of the file.  Or close() and open() the file again,
which is less efficient.

[snip of code showing this]

Quote:
> will ONLY grab the first element, subsequent arguments are ignored.
> 'Course I could be doing something wrong. :-)

I'll please the Fifth here.  :-)

Quote:
> I've used this once or twice on small files on (shudder) win32 where I

Hey!  Some of my best friends know people who have used win32 !!

Actually, I find that ActiveState Perl on win32 and the Perl Power Tools
can make working on win32 platforms much nicer.  There's a tail program
in there, IIRC...

Quote:
> didn't have tail handy and I needed a specific line, but have always
> been hesitant to use it on bigger files. And a bit hesitant to use it in
> production code since I don't really understand exactly what's going on.

Good, and good.  If your files are big enough, the slurp will hurt..
right about the time you have to start swapping virtual memory.  And if
there's something about Perl you don't exactly, precisely understand..
then join the crowd.  :-)

Quote:
> Interested in comments.

You may get a few!

David
--

Senior Computing Specialist
mathematical statistician



Wed, 18 Jun 1902 08:00:00 GMT  
 best way to read last line?


Wed, 18 Jun 1902 08:00:00 GMT  
 best way to read last line?

MCMXCIII in <URL::">

{} >
{} > What is the most efficient & fastest way of reading the last line of
{} > a flat file (small or large)?
{} >
{} > 2) Open the file, seek to the end, seek back a little, use sysread
{} > to read the part you seeked back over. If it contains a newline,
{} > you now know the part you can read, else you seek back a bit more,
{} > rinse and repeat.
{}
{} Here is an example implementing approach 2). It is ugly but it seems
{} to be working and it is efficient enough for my purposes:
{}
{}
{} [80+ lines snipped]
{}
{}
{} my $filename = $ARGV[0];
{} my $lines    = $ARGV[1];
{}

And in considerably less lines:



Abigail
--

             "\150\145\162\040\120\145\162\154\040\110\141\143\153\145\162".

  -----------== Posted via Newsfeeds.Com, Uncensored Usenet News ==----------
   http://www.newsfeeds.com       The Largest Usenet Servers in the World!
------== Over 73,000 Newsgroups - Including  Dedicated  Binaries Servers ==-----



Wed, 18 Jun 1902 08:00:00 GMT  
 best way to read last line?

Quote:
> And in considerably less lines:




This thread has gone full circle now.  Does that mean the thread is not
done?

--

pls note the one line sig, not counting this one.



Wed, 18 Jun 1902 08:00:00 GMT  
 best way to read last line?


Wed, 18 Jun 1902 08:00:00 GMT  
 best way to read last line?

So here is a burning question on reading the last line of a file that
has been bugging me for months....

This WILL work

open(FILE, '/dir/file.file') or die "Can't read file.file, $!\n";
 $line = (<FILE>)[-1];
close FILE  or warn "Can't close file.file, $!\n";

print "$line\n";

# prints last line or (<FILE>)[5] prints... well, you know..

BUT!

Is the whole file being read into memory so it can be treated as an
array, or is Perl counting $/ and then pulling the proper line or???

I haven't found any docs that tell me what's going on here.  I may be
looking in the wrong places of course.

Interestingly, this code will ONLY grab one line from the file, attempts
to run a loop and get more than one line seem doomed to failure.

Something like:


open(FILE, '/dir/file.file') or die "Can't read file.file, $!\n";

close FILE  or warn "Can't close file.file, $!\n";

or

open(FILE, '/dir/file.file') or die "Can't read file.file, $!\n";
$string .= (<FILE>)[1],"\n";
$string .= (<FILE>)[2],"\n";
close FILE  or warn "Can't close file.file, $!\n";

will ONLY grab the first element, subsequent arguments are ignored.  
'Course I could be doing something wrong. :-)

I've used this once or twice on small files on (shudder) win32 where I
didn't have tail handy and I needed a specific line, but have always
been hesitant to use it on bigger files. And a bit hesitant to use it in
production code since I don't really understand exactly what's going on.

Interested in comments.

Steve



Wed, 18 Jun 1902 08:00:00 GMT  
 best way to read last line?
On Tue, 04 Jan 2000 00:33:12 GMT,

Quote:

> So here is a burning question on reading the last line of a file that
> has been bugging me for months....

> This WILL work

> open(FILE, '/dir/file.file') or die "Can't read file.file, $!\n";
>  $line = (<FILE>)[-1];
> close FILE  or warn "Can't close file.file, $!\n";

> print "$line\n";

> # prints last line or (<FILE>)[5] prints... well, you know..

> BUT!

> Is the whole file being read into memory so it can be treated as an
> array, or is Perl counting $/ and then pulling the proper line or???

Yes. No.

But:

my $line;
$line = $_ while(<FILE>);

only reads in one line at a time.  Of course, it still does an
assignment for each line in the file, and it has to scan the whole
file.

Something like this avoids the assignments:

while(<FILE>) {
        next unless eof FILE;
        $line = $_;

Quote:
}

but it has to do a test on each iteration. (A benchmark might show
what's best. but I don't feel like doing it). And it still scans the
whole file.

Probably it depends on the size of the file you want to open what's
most efficient (a seek to the end and jumping backwards, slurping in
the file, scanning each line until the last).

You could have a look to see how this is implemented in the tail that comes
with the Perl Power Tools: http://language.perl.com/ppt. (It seems
that  that tail always scans the file if it's text, so maybe it was
judged to be the best general case).

Martien

PS. The sig is a fluke. Randomly but appropriately picked.
--
Martien Verbruggen              |
Interactive Media Division      | Begin at the beginning and go on till
Commercial Dynamics Pty. Ltd.   | you come to the end; then stop.
NSW, Australia                  |



Wed, 18 Jun 1902 08:00:00 GMT  
 
 [ 31 post ]  Go to page: [1] [2] [3]

 Relevant Pages 

1. read last line without reading previous lines, how?

2. Reading Last X Lines of File?

3. Quickly read the last 10 lines of a very large logfile

4. Reading last 3 lines from the file

5. read last line only...

6. Reading the last line of a file

7. Seeking advice on semantics (pros / cons and best ways of) using constants across modules

8. Better ways to deal with yppasswd?

9. Better ways to do this?

10. Better ways to...

11. Any better ways of doing this ?

12. EDITING DATAFILES - What is best of 3 ways?

 

 
Powered by phpBB® Forum Software