Line by line file reading 
Author Message
 Line by line file reading

Hi there,

 I wonder if anyone can help me with a small
problem

I need to be able to open a file line by line as
the file is about 10,000 lines long by the time I
need to run this script.

what i need to be able to do (in pseudo Code) is

Open file
  read line to EOF
  compair this line to user input
    if TRUE then counter++
close file
output results

simple eh!

I DON'T want to do something like:

open (FILEHANDLE, "$fileName");

{
  DO PROCESSING

Quote:
}

close (FILEHANDLE);

as this will take up TOO much memory and lock the
file out for too long (as this is a log file)

clear ?

Could you please e-mail me any responcces as well
as leaving them in this group

Thanks in advance

Colin Bell

Sent via Deja.com http://www.*-*-*.com/
Before you buy.



Wed, 18 Jun 1902 08:00:00 GMT  
 Line by line file reading
Hi there,

 I wonder if anyone can help me with a small
problem

I need to be able to open a file line by line as
the file is about 10,000 lines long by the time I
need to run this script.

what i need to be able to do (in pseudo Code) is

Open file
  read line to EOF
  compair this line to user input
    if TRUE then counter++
close file
output results

simple eh!

I DON'T want to do something like:

open (FILEHANDLE, "$fileName");

{
  DO PROCESSING

Quote:
}

close (FILEHANDLE);

as this will take up TOO much memory and lock the
file out for too long (as this is a log file)

clear ?

Could you please e-mail me any responcces as well
as leaving them in this group

Thanks in advance

Colin Bell

Sent via Deja.com http://www.deja.com/
Before you buy.



Wed, 18 Jun 1902 08:00:00 GMT  
 Line by line file reading

Quote:
> Hi there,

>  I wonder if anyone can help me with a small
> problem

> I need to be able to open a file line by line as
> the file is about 10,000 lines long by the time I
> need to run this script.

> what i need to be able to do (in pseudo Code) is

> Open file
>   read line to EOF
>   compair this line to user input
>     if TRUE then counter++
> close file
> output results

> simple eh!

> I DON'T want to do something like:

> open (FILEHANDLE, "$fileName");
> flock FILEHANDLE, 2;

> {
>   DO PROCESSING
> }
> close (FILEHANDLE);

> as this will take up TOO much memory and lock the
> file out for too long (as this is a log file)

Well you _are_ going to have to something *like* that.

If you have already read the contents of the file into the array
you can close the filehandle straight away :

open (FILE,$filename) || die "Cant open $filename - $!\n";

close FILE;


{
  # whatever

Quote:
}

This is expensive in terms of memory for a large file.

Or you can read from the file a line at a time :

open (FILE,$filename) || die "Cant open $filename - $!\n";

while (<FILE>)
{
  # whatever

Quote:
}

close FILE

This might be expensive if it keeps the file open too long.

Thems are your choices - you will have to do one or the other and work
around the drawbacks.

/J\
--

<http://www.gellyfish.com>
** Uri Guttman - Have You CPANed Backward.pm Yet ? **



Wed, 18 Jun 1902 08:00:00 GMT  
 Line by line file reading
Could you maybe duplicate the file (it is a log file so it shouldn't be THAT
big) to a tmp directory then you would be without any problems when it comes
to locking the file, then you could use the most efficient method of reading
the file, not that I know the answer to THAT question as I want to find out
myself..

ANy ideas??

Stuart Low


Quote:

> > Hi there,

> >  I wonder if anyone can help me with a small
> > problem

> > I need to be able to open a file line by line as
> > the file is about 10,000 lines long by the time I
> > need to run this script.

> > what i need to be able to do (in pseudo Code) is

> > Open file
> >   read line to EOF
> >   compair this line to user input
> >     if TRUE then counter++
> > close file
> > output results

> > simple eh!

> > I DON'T want to do something like:

> > open (FILEHANDLE, "$fileName");
> > flock FILEHANDLE, 2;

> > {
> >   DO PROCESSING
> > }
> > close (FILEHANDLE);

> > as this will take up TOO much memory and lock the
> > file out for too long (as this is a log file)

> Well you _are_ going to have to something *like* that.

> If you have already read the contents of the file into the array
> you can close the filehandle straight away :

> open (FILE,$filename) || die "Cant open $filename - $!\n";


> close FILE;


> {
>   # whatever
> }

> This is expensive in terms of memory for a large file.

> Or you can read from the file a line at a time :

> open (FILE,$filename) || die "Cant open $filename - $!\n";

> while (<FILE>)
> {
>   # whatever
> }

> close FILE

> This might be expensive if it keeps the file open too long.

> Thems are your choices - you will have to do one or the other and work
> around the drawbacks.

> /J\
> --

> <http://www.gellyfish.com>
> ** Uri Guttman - Have You CPANed Backward.pm Yet ? **



Wed, 18 Jun 1902 08:00:00 GMT  
 Line by line file reading
[Your newsreader has chronology problems. Your reply appears before the
text you reply to. Please, next time respect the arrow of time]

On Mon, 24 Jan 2000 04:42:40 +1000,

[reordered post]

Quote:



> > > I need to be able to open a file line by line as
> > > the file is about 10,000 lines long by the time I
> > > need to run this script.

> > while (<FILE>)
> > {
> >   # whatever
> > }

> Could you maybe duplicate the file (it is a log file so it shouldn't
> be THAT big) to a tmp directory then you would be without any problems
> when it comes to locking the file, then you could use the most
> efficient method of reading the file, not that I know the answer to
> THAT question as I want to find out myself..

I find the statement 'it is a log dile, so it shouldn't be THAT big' a
bit odd... Have you ever worked with FTP or Web servers where even
rotating log files on an hourly basis would leave you with files that
were hundreds of megabytes or more? Of course, you could have a
different idea about what 'large' means, but 100 MB  for a text file to
me is quite considerable :) 800 MB comes into the realm where almost no
one would call it small anymore. Most /tmp partitions on web servers
aren't big enough to hold them anyway.

Of course, 10,000 lines is indeed small, and could probably just be
taken care of with a copy of some sorts.

Martien
--
Martien Verbruggen              | My friend has a baby. I'm writing
Interactive Media Division      | down all the noises the baby makes so
Commercial Dynamics Pty. Ltd.   | later I can ask him what he meant -
NSW, Australia                  | Steven Wright



Wed, 18 Jun 1902 08:00:00 GMT  
 Line by line file reading
[Your newsreader has chronology problems. Your reply appears before the
text you reply to. Please, next time respect the arrow of time]

On Mon, 24 Jan 2000 04:42:40 +1000,

[reordered post]

Quote:



> > > I need to be able to open a file line by line as
> > > the file is about 10,000 lines long by the time I
> > > need to run this script.

> > while (<FILE>)
> > {
> >   # whatever
> > }

> Could you maybe duplicate the file (it is a log file so it shouldn't
> be THAT big) to a tmp directory then you would be without any problems
> when it comes to locking the file, then you could use the most
> efficient method of reading the file, not that I know the answer to
> THAT question as I want to find out myself..

I find the statement 'it is a log file, so it shouldn't be THAT big' a
bit odd... Have you ever worked with FTP or Web servers where even
rotating log files on an hourly basis would leave you with files that
were hundreds of megabytes or more? Of course, you could have a
different idea about what 'large' means, but 100 MB  for a text file to
me is quite considerable :) 800 MB comes into the realm where almost no
one would call it small anymore. Most /tmp partitions on web servers
aren't big enough to hold them anyway.

Of course, 10,000 lines is indeed small, and could probably just be
taken care of with a copy of some sorts.

Martien
--
Martien Verbruggen              | My friend has a baby. I'm writing
Interactive Media Division      | down all the noises the baby makes so
Commercial Dynamics Pty. Ltd.   | later I can ask him what he meant -
NSW, Australia                  | Steven Wright



Wed, 18 Jun 1902 08:00:00 GMT  
 Line by line file reading
Hello fellow learner
If I was to compare a file to <STDIN> I would do something
like:
open(NAME, /home/filename); #open the file
print "please enter input\n";
$input=<STDIN>; #create scalar for STDIN
chomp $input;
while <NAME>;
print $_; #the magic operator
if ($input !eq $_){
then
print $input does not equal this line;
Quote:
}

This is a rushed answer but it should not be too difficult
to carry out a compare like this - the !eq is meant to mean
"not equal to" I can't remember the syntax for this.

Regards..

* Sent from AltaVista http://www.altavista.com Where you can also find related Web Pages, Images, Audios, Videos, News, and Shopping.  Smart is Beautiful



Wed, 18 Jun 1902 08:00:00 GMT  
 Line by line file reading


Quote:
> Hello fellow learner
> If I was to compare a file to <STDIN> I would do something
> like:
> open(NAME, /home/filename); #open the file
> print "please enter input\n";
> $input=<STDIN>; #create scalar for STDIN
> chomp $input;
> while <NAME>;
> print $_; #the magic operator
> if ($input !eq $_){
> then
> print $input does not equal this line;
> }
> This is a rushed answer but it should not be too difficult
> to carry out a compare like this - the !eq is meant to mean
> "not equal to" I can't remember the syntax for this.

Whoa, hold on there dude! Your example doesn't even compile, and
it's considered bad form to post something like that, so you might
want to duck :).

In addition, the style is not really how such things would be
typically done in Perl. If you hang around here for a while and
read the examples in the faqs and elswhere, you'll see there are
some definite Perl idioms that are commonly used.

Just a few remarks on your "example":

Quote:
> open(NAME, /home/filename); #open the file

At a minimum, you need quotes around the file name. Also, ALWAYS
check the return from open() or other system calls. What if the
file isn't there?

Quote:
> print "please enter input\n";
> $input=<STDIN>; #create scalar for STDIN
> chomp $input;

You should always 'use strict' and have -w enabled. They keep
you out of all kinds of trouble.

To pass a single argument into a program, you would typically
pass it on the command line, and retrieve it with something
like:

    defined($input = shift)
        or die "Expected search pattern\n";

Quote:
> while <NAME>;

This is invalid syntax all the way around. The condition to
while must be enclosed in parentheses, and must be followed
by a block.

Quote:
> print $_; #the magic operator

No "magic operator" here. $_ is a global variable which is
assigned by the while(<FOO>) construct, if you get your
syntax right above.

Quote:
> if ($input !eq $_){

The ! is a unary operator, which logically negates the value
following. 'eq' is a binary string comparison operator. So
'!eq' is a sequence of two operators, which is gibberish.
You could write this as

    if ($input eq $_) { ... }

or

    unless ($input eq $_) { ... }

However, since the OP was wanting to search through a log
file, he probably wants a regular expression match.

Quote:
> then

Huh? No such keyword in Perl.

Quote:
> print $input does not equal this line;

Need some quotes there!

Quote:
> }

It's OK to post, and posting is a great way to learn (I am
just a learner too). But hang around and watch the group for
a while and you'll quickly pick up on the things to avoid.
And posting code that doesn't compile is definitely a no-no.
Getting critiques on style is very common, as well, and
should be taken in stride. I expect that my commentary above
will yield some responses challenging my style or analysis.

--
Bob

Sent via Deja.com http://www.deja.com/
Before you buy.



Wed, 18 Jun 1902 08:00:00 GMT  
 
 [ 8 post ] 

 Relevant Pages 

1. How do I read line by line of data from a file using perl

2. how to read a huge file line by line without loading it into memory

3. reading lines from one file and pulling matching lines in another

4. newbie perl question - reading a file line by line

5. How to read a file line by line w/loop

6. Read remote text file line by line

7. reading a file line by line backwards

8. read last line without reading previous lines, how?

9. how to read in line by line?

10. read line by line

11. Limiting maximum line length during line based read

12. read line by line in perl

 

 
Powered by phpBB® Forum Software