Line by line file reading
Author |
Message |
vulcan_saa.. #1 / 8
|
 Line by line file reading
Hi there, I wonder if anyone can help me with a small problem I need to be able to open a file line by line as the file is about 10,000 lines long by the time I need to run this script. what i need to be able to do (in pseudo Code) is Open file read line to EOF compair this line to user input if TRUE then counter++ close file output results simple eh! I DON'T want to do something like: open (FILEHANDLE, "$fileName");
{ DO PROCESSING Quote: }
close (FILEHANDLE); as this will take up TOO much memory and lock the file out for too long (as this is a log file) clear ? Could you please e-mail me any responcces as well as leaving them in this group Thanks in advance Colin Bell
Sent via Deja.com http://www.*-*-*.com/ Before you buy.
|
Wed, 18 Jun 1902 08:00:00 GMT |
|
 |
vulcan_saa.. #2 / 8
|
 Line by line file reading
Hi there, I wonder if anyone can help me with a small problem I need to be able to open a file line by line as the file is about 10,000 lines long by the time I need to run this script. what i need to be able to do (in pseudo Code) is Open file read line to EOF compair this line to user input if TRUE then counter++ close file output results simple eh! I DON'T want to do something like: open (FILEHANDLE, "$fileName");
{ DO PROCESSING Quote: }
close (FILEHANDLE); as this will take up TOO much memory and lock the file out for too long (as this is a log file) clear ? Could you please e-mail me any responcces as well as leaving them in this group Thanks in advance Colin Bell
Sent via Deja.com http://www.deja.com/ Before you buy.
|
Wed, 18 Jun 1902 08:00:00 GMT |
|
 |
Jonathan Stow #3 / 8
|
 Line by line file reading
Quote: > Hi there, > I wonder if anyone can help me with a small > problem > I need to be able to open a file line by line as > the file is about 10,000 lines long by the time I > need to run this script. > what i need to be able to do (in pseudo Code) is > Open file > read line to EOF > compair this line to user input > if TRUE then counter++ > close file > output results > simple eh! > I DON'T want to do something like: > open (FILEHANDLE, "$fileName"); > flock FILEHANDLE, 2;
> { > DO PROCESSING > } > close (FILEHANDLE); > as this will take up TOO much memory and lock the > file out for too long (as this is a log file)
Well you _are_ going to have to something *like* that. If you have already read the contents of the file into the array you can close the filehandle straight away : open (FILE,$filename) || die "Cant open $filename - $!\n";
close FILE;
{ # whatever Quote: }
This is expensive in terms of memory for a large file. Or you can read from the file a line at a time : open (FILE,$filename) || die "Cant open $filename - $!\n"; while (<FILE>) { # whatever Quote: }
close FILE This might be expensive if it keeps the file open too long. Thems are your choices - you will have to do one or the other and work around the drawbacks. /J\ --
<http://www.gellyfish.com> ** Uri Guttman - Have You CPANed Backward.pm Yet ? **
|
Wed, 18 Jun 1902 08:00:00 GMT |
|
 |
Stuart Lo #4 / 8
|
 Line by line file reading
Could you maybe duplicate the file (it is a log file so it shouldn't be THAT big) to a tmp directory then you would be without any problems when it comes to locking the file, then you could use the most efficient method of reading the file, not that I know the answer to THAT question as I want to find out myself.. ANy ideas?? Stuart Low
Quote:
> > Hi there, > > I wonder if anyone can help me with a small > > problem > > I need to be able to open a file line by line as > > the file is about 10,000 lines long by the time I > > need to run this script. > > what i need to be able to do (in pseudo Code) is > > Open file > > read line to EOF > > compair this line to user input > > if TRUE then counter++ > > close file > > output results > > simple eh! > > I DON'T want to do something like: > > open (FILEHANDLE, "$fileName"); > > flock FILEHANDLE, 2;
> > { > > DO PROCESSING > > } > > close (FILEHANDLE); > > as this will take up TOO much memory and lock the > > file out for too long (as this is a log file) > Well you _are_ going to have to something *like* that. > If you have already read the contents of the file into the array > you can close the filehandle straight away : > open (FILE,$filename) || die "Cant open $filename - $!\n";
> close FILE;
> { > # whatever > } > This is expensive in terms of memory for a large file. > Or you can read from the file a line at a time : > open (FILE,$filename) || die "Cant open $filename - $!\n"; > while (<FILE>) > { > # whatever > } > close FILE > This might be expensive if it keeps the file open too long. > Thems are your choices - you will have to do one or the other and work > around the drawbacks. > /J\ > --
> <http://www.gellyfish.com> > ** Uri Guttman - Have You CPANed Backward.pm Yet ? **
|
Wed, 18 Jun 1902 08:00:00 GMT |
|
 |
Martien Verbrugg #5 / 8
|
 Line by line file reading
[Your newsreader has chronology problems. Your reply appears before the text you reply to. Please, next time respect the arrow of time] On Mon, 24 Jan 2000 04:42:40 +1000,
[reordered post] Quote:
> > > I need to be able to open a file line by line as > > > the file is about 10,000 lines long by the time I > > > need to run this script. > > while (<FILE>) > > { > > # whatever > > } > Could you maybe duplicate the file (it is a log file so it shouldn't > be THAT big) to a tmp directory then you would be without any problems > when it comes to locking the file, then you could use the most > efficient method of reading the file, not that I know the answer to > THAT question as I want to find out myself..
I find the statement 'it is a log dile, so it shouldn't be THAT big' a bit odd... Have you ever worked with FTP or Web servers where even rotating log files on an hourly basis would leave you with files that were hundreds of megabytes or more? Of course, you could have a different idea about what 'large' means, but 100 MB for a text file to me is quite considerable :) 800 MB comes into the realm where almost no one would call it small anymore. Most /tmp partitions on web servers aren't big enough to hold them anyway. Of course, 10,000 lines is indeed small, and could probably just be taken care of with a copy of some sorts. Martien -- Martien Verbruggen | My friend has a baby. I'm writing Interactive Media Division | down all the noises the baby makes so Commercial Dynamics Pty. Ltd. | later I can ask him what he meant - NSW, Australia | Steven Wright
|
Wed, 18 Jun 1902 08:00:00 GMT |
|
 |
Martien Verbrugg #6 / 8
|
 Line by line file reading
[Your newsreader has chronology problems. Your reply appears before the text you reply to. Please, next time respect the arrow of time] On Mon, 24 Jan 2000 04:42:40 +1000,
[reordered post] Quote:
> > > I need to be able to open a file line by line as > > > the file is about 10,000 lines long by the time I > > > need to run this script. > > while (<FILE>) > > { > > # whatever > > } > Could you maybe duplicate the file (it is a log file so it shouldn't > be THAT big) to a tmp directory then you would be without any problems > when it comes to locking the file, then you could use the most > efficient method of reading the file, not that I know the answer to > THAT question as I want to find out myself..
I find the statement 'it is a log file, so it shouldn't be THAT big' a bit odd... Have you ever worked with FTP or Web servers where even rotating log files on an hourly basis would leave you with files that were hundreds of megabytes or more? Of course, you could have a different idea about what 'large' means, but 100 MB for a text file to me is quite considerable :) 800 MB comes into the realm where almost no one would call it small anymore. Most /tmp partitions on web servers aren't big enough to hold them anyway. Of course, 10,000 lines is indeed small, and could probably just be taken care of with a copy of some sorts. Martien -- Martien Verbruggen | My friend has a baby. I'm writing Interactive Media Division | down all the noises the baby makes so Commercial Dynamics Pty. Ltd. | later I can ask him what he meant - NSW, Australia | Steven Wright
|
Wed, 18 Jun 1902 08:00:00 GMT |
|
 |
franki #7 / 8
|
 Line by line file reading
Hello fellow learner If I was to compare a file to <STDIN> I would do something like: open(NAME, /home/filename); #open the file print "please enter input\n"; $input=<STDIN>; #create scalar for STDIN chomp $input; while <NAME>; print $_; #the magic operator if ($input !eq $_){ then print $input does not equal this line; Quote: }
This is a rushed answer but it should not be too difficult to carry out a compare like this - the !eq is meant to mean "not equal to" I can't remember the syntax for this. Regards.. * Sent from AltaVista http://www.altavista.com Where you can also find related Web Pages, Images, Audios, Videos, News, and Shopping. Smart is Beautiful
|
Wed, 18 Jun 1902 08:00:00 GMT |
|
 |
Bob Showalte #8 / 8
|
 Line by line file reading
Quote: > Hello fellow learner > If I was to compare a file to <STDIN> I would do something > like: > open(NAME, /home/filename); #open the file > print "please enter input\n"; > $input=<STDIN>; #create scalar for STDIN > chomp $input; > while <NAME>; > print $_; #the magic operator > if ($input !eq $_){ > then > print $input does not equal this line; > } > This is a rushed answer but it should not be too difficult > to carry out a compare like this - the !eq is meant to mean > "not equal to" I can't remember the syntax for this.
Whoa, hold on there dude! Your example doesn't even compile, and it's considered bad form to post something like that, so you might want to duck :). In addition, the style is not really how such things would be typically done in Perl. If you hang around here for a while and read the examples in the faqs and elswhere, you'll see there are some definite Perl idioms that are commonly used. Just a few remarks on your "example": Quote: > open(NAME, /home/filename); #open the file
At a minimum, you need quotes around the file name. Also, ALWAYS check the return from open() or other system calls. What if the file isn't there? Quote: > print "please enter input\n"; > $input=<STDIN>; #create scalar for STDIN > chomp $input;
You should always 'use strict' and have -w enabled. They keep you out of all kinds of trouble. To pass a single argument into a program, you would typically pass it on the command line, and retrieve it with something like: defined($input = shift) or die "Expected search pattern\n"; Quote: > while <NAME>;
This is invalid syntax all the way around. The condition to while must be enclosed in parentheses, and must be followed by a block. Quote: > print $_; #the magic operator
No "magic operator" here. $_ is a global variable which is assigned by the while(<FOO>) construct, if you get your syntax right above. Quote: > if ($input !eq $_){
The ! is a unary operator, which logically negates the value following. 'eq' is a binary string comparison operator. So '!eq' is a sequence of two operators, which is gibberish. You could write this as if ($input eq $_) { ... } or unless ($input eq $_) { ... } However, since the OP was wanting to search through a log file, he probably wants a regular expression match. Quote: > then
Huh? No such keyword in Perl. Quote: > print $input does not equal this line;
Need some quotes there! Quote: > }
It's OK to post, and posting is a great way to learn (I am just a learner too). But hang around and watch the group for a while and you'll quickly pick up on the things to avoid. And posting code that doesn't compile is definitely a no-no. Getting critiques on style is very common, as well, and should be taken in stride. I expect that my commentary above will yield some responses challenging my style or analysis. -- Bob Sent via Deja.com http://www.deja.com/ Before you buy.
|
Wed, 18 Jun 1902 08:00:00 GMT |
|
|
|