Input File Size Limitations - Newbie question 
Author Message
 Input File Size Limitations - Newbie question

I use awk to format fairly large files but now I am having trouble with a
file that is 8.5 GB. I am using some DOS version of awk (don't ask me
details about the version because I just do not know). I have successfully
formatted 1 GB files. The error is that it cannot open the data file. I am
running this on a fairly high end system. Insight would be greatly
appreciated.


Sat, 05 Nov 2005 10:41:01 GMT  
 Input File Size Limitations - Newbie question

X I use awk to format fairly large files but now I am having trouble with a
X file that is 8.5 GB. I am using some DOS version of awk (don't ask me
X details about the version because I just do not know). I have successfully
X formatted 1 GB files. The error is that it cannot open the data file. I am
X running this on a fairly high end system. Insight would be greatly
X appreciated.

This is a guess, but I thought that DOS was a 32 bit environment, so if
your awk is looking at the file size, 8.5GB would not fit in a 32 bit
variable.  Depending on how DOS behaves with a huge file, it may refuse
to process the file, or it might only read 500MB more or less (8.5GB
truncated would look like .5GB when stored in a 32 bit variable).

Also if you are storing information is an array, if you are collecting
too much information, the DOS environment might run out of its ability
to allocate enough memory for the task.

But that was just a guess.

Maybe the latest version of gawk would work on your system.

Again, I could be totally wrong about it.

                                        Bob Harris



Sat, 05 Nov 2005 11:36:43 GMT  
 Input File Size Limitations - Newbie question

Quote:

> I use awk to format fairly large files but now I am having trouble
> with a file that is 8.5 GB. I am using some DOS version of awk (don't
> ask me details about the version because I just do not know). I have
> successfully formatted 1 GB files. The error is that it cannot open
> the data file. I am running this on a fairly high end system. Insight
> would be greatly appreciated.

It is very difficult to even speculate on an answer without a) knowing
the precise version of DOS that you're using, b) whether it is "real"
DOS or just a DOS box under Windows, c) if the latter which version of
Windows [1], and, d) whether you are using cmd.exe or command.com to run
the awk program (since both are supplied on NT-like systems for
example).

As to the version of awk you could try any of the following to try to
determine the version number:

awk -V
awk --version
awk -W version

If none of these produce a result, then where did you get the awk from?
It isn't supplied with any standard Microsoft installation, and can't
have magically appeared on the PC.

As I say without a little more information it is difficult for us to
help.

The main alternative to providing the above information is to install
either a minimal DJGPP, Cygwin or MinGW system on your PC, then you
would know the version of awk and have a known type of installation.

[1] including which service packs are installed where these are
appropriate.

HTH
--
Peter S Tillier
"Who needs perl when you can write dc, sokoban,
arkanoid and an unlambda interpreter in sed?"



Sat, 05 Nov 2005 14:20:22 GMT  
 Input File Size Limitations - Newbie question
Thanks for responding, I appreciate the help. The version states:
 AWK 2.10
Copyright (c) 1988 Rob Duff
Vancouver BC Canada   ....hmmmm never heard of this one

This program is something that was given to me years back and I have been
running it under the DOS prompt (cmd.exe) of 2000 and XP and has served me
well. My files are always columnar data in which I have to reformat and
perform basic calculations. For example they are 9 columns in which I
subtract a value from the 4th and 7th column. As I stated I have not had
problems with files around 1GB.
Tim

Is there a version I could get that would run on my 2000 machine that would
open the large file?


Quote:

> > I use awk to format fairly large files but now I am having trouble
> > with a file that is 8.5 GB. I am using some DOS version of awk (don't
> > ask me details about the version because I just do not know). I have
> > successfully formatted 1 GB files. The error is that it cannot open
> > the data file. I am running this on a fairly high end system. Insight
> > would be greatly appreciated.

> It is very difficult to even speculate on an answer without a) knowing
> the precise version of DOS that you're using, b) whether it is "real"
> DOS or just a DOS box under Windows, c) if the latter which version of
> Windows [1], and, d) whether you are using cmd.exe or command.com to run
> the awk program (since both are supplied on NT-like systems for
> example).

> As to the version of awk you could try any of the following to try to
> determine the version number:

> awk -V
> awk --version
> awk -W version

> If none of these produce a result, then where did you get the awk from?
> It isn't supplied with any standard Microsoft installation, and can't
> have magically appeared on the PC.

> As I say without a little more information it is difficult for us to
> help.

> The main alternative to providing the above information is to install
> either a minimal DJGPP, Cygwin or MinGW system on your PC, then you
> would know the version of awk and have a known type of installation.

> [1] including which service packs are installed where these are
> appropriate.

> HTH
> --
> Peter S Tillier
> "Who needs perl when you can write dc, sokoban,
> arkanoid and an unlambda interpreter in sed?"



Sat, 05 Nov 2005 18:58:56 GMT  
 Input File Size Limitations - Newbie question

% Thanks for responding, I appreciate the help. The version states:
%  AWK 2.10
% Copyright (c) 1988 Rob Duff
% Vancouver BC Canada   ....hmmmm never heard of this one

Wow. I Haven't seen this for a long time. I expect it's a 16-bit program,
and I further expect that the 16-bit emulation layer doesn't allow
you to open files larger than 2 or 4 G.

Try getting awk95 or gawk. They are 32-bit applications, and should be
able to process the larger files you crave.
--

Patrick TJ McPhee
East York  Canada



Sat, 05 Nov 2005 23:57:04 GMT  
 Input File Size Limitations - Newbie question




% Thanks for responding, I appreciate the help. The version
states:
%  AWK 2.10
% Copyright (c) 1988 Rob Duff
% Vancouver BC Canada   ....hmmmm never heard of this one

Wow. I Haven't seen this for a long time. I expect it's a
16-bit program,
and I further expect that the 16-bit emulation layer doesn't
allow
you to open files larger than 2 or 4 G.

Try getting awk95 or gawk. They are 32-bit applications, and
should be
able to process the larger files you crave.
--

Patrick TJ McPhee
East York  Canada

You can get a later version of gawk, specifically for
windows, from www.gnu.org

Joe Gwozdecki
Houston, Texas



Sun, 06 Nov 2005 01:09:05 GMT  
 Input File Size Limitations - Newbie question
Quote:

> Thanks for responding, I appreciate the help. The version states:
>  AWK 2.10
> Copyright (c) 1988 Rob Duff
> Vancouver BC Canada   ....hmmmm never heard of this one

> This program is something that was given to me years back and I have
> been running it under the DOS prompt (cmd.exe) of 2000 and XP and has
> served me well. My files are always columnar data in which I have to
> reformat and perform basic calculations. For example they are 9
> columns in which I subtract a value from the 4th and 7th column. As I
> stated I have not had problems with files around 1GB.
> Tim

[...]

Interesting!  I have a copy of Rob's awk version 3.20 with the being
executable dated May 1991.

As Patrick says the best option would be to get a Win32 version such as
Brian Kernighan's awk95.exe:

http://www.cs.bell-labs.com/cm/cs/awkbook/index.html

and click the Windows executable link to download, or, a port of gawk.

HTH
--
Peter S Tillier
"Who needs perl when you can write dc, sokoban,
arkanoid and an unlambda interpreter in sed?"



Sun, 06 Nov 2005 02:53:47 GMT  
 
 [ 7 post ] 

 Relevant Pages 

1. newbie question for reading input file

2. newbie question re: forms with input type=file

3. Size Limitations of files

4. File Size limitation for Clipper

5. Memo file size limitations

6. File Size limitation under RM Cobol Version 5.XX

7. File size limitations

8. newbie would like to break input file and output to separate files

9. input file size limit?

10. Newbie question: input problem

11. (Newbie)Outputting file sizes in a dir. listing

12. Newbie Question - String Input

 

 
Powered by phpBB® Forum Software