Problem with very large files in NT 
Author Message
 Problem with very large files in NT

I have a general purpose program that assigns a code to a specified
location in each record.  I.E. I want to put "0" at position 499 in
every single record.

This program has been in use for the past several years and works just
fine for smaller files.  However, I have a very large file, 3.45 Gig
(3704612914 bytes) containing 7,379,707 records each one being 502
bytes long with the last 2 bytes of each record being crlf.

I am using VisualAge(TM) PL/I for Windows(R)  V2.R1.08

When I run the program I get the following error.

IBM0841I  ONCODE=1040  
A data set, open for output, used all available space.
At offset +00000175 in a BEGIN block                                  
        From offset +000005F4 in procedure with entry UNICOD

The data file is then corrupted because the size is changed to
4,294,967,798 bytes (which happens to be 4 Gig + 502 bytes)

When I step through the program with the de{*filter*} the file size is
changed when the first record is read.

When the On Error executes the output is
Error at record:              1

Here is the code

%process langlvl(saa2) margins(1,96) limits(extname(31)) macro;
%PROCESS RULES(NOLAXDCL NOLAXIF);
*PROCESS LIMITS(EXTNAME(100) fixedbin(63) fixeddec(31) name(100) );

unicod:procedure options (main);
on error begin;
   put skip edit('Error at record: ' || count)(a);
   call pliretc(212);
   resignal;
end;

/*Declarations*/
dcl buffer char(2048) var;
dcl infile file record;
dcl innam char(132) var;
dcl code char(80) var init ('');
dcl yn char(1) init('');
dcl (count,startcol,codlen,existing) fixed bin(31) init(0);

on endfile(infile) goto exit;

   put skip edit(
   'This program allows insertion of a code into a record at a
position that',
   'you specify.  Codes may be up to 80 characters in length, and are
entered',
   'exactly as you type them in.')(a,skip,a,skip,a);

   display('Name of the input file  : ') reply(innam);
   open file (infile) title('/'||innam) update;

   put skip list ('Code Control Information ...');
   put skip edit('  Code to insert : ')(a);  get edit(code)(l);

   do while(yn^='Y' & yn ^='N');
     put skip edit('  Shall I write over existing data? (Y/N)')(a);
get edit(yn)(l);
     yn = translate(yn,'YN','yn');
     if (yn ^= 'Y' & yn ^= 'N')
      then put skip edit('Only "Y" and "N" are valid responses.')(a);
   end;

   put skip list('What is the starting column number for this code ?
');
   get edit(startcol)(f(8));
   codlen = length(code);

   do while ('1'b);
      begin;
           read file (infile) into (buffer);
           count=count+1;
           if yn = 'Y' then do;
               substr(buffer,startcol,codlen)=code;
               rewrite file (infile) from (buffer);
           end;
           else do;
               if substr(buffer,startcol,codlen)=' ' then do;
                   substr(buffer,startcol,codlen)=code;
                   rewrite file (infile) from (buffer);
               end;
               else existing = existing + 1;
               /* put skip list('Didn''t overwrite existing data at
',count)*/
           end;
      end;
   end;

exit:
   close file(infile);
   put skip list ('The number of records processed is       :',count);
   put skip list ('The number of records not overwritten is
:',existing);
   put skip list ('The total number of records coded is     :',
   count-existing);
   put skip list ('UNICOD is done');
end unicod;



Wed, 09 Mar 2005 01:23:21 GMT  
 Problem with very large files in NT
What file system are  you using?


Quote:
>I have a general purpose program that assigns a code to a specified
>location in each record.  I.E. I want to put "0" at position 499 in
>every single record.

>This program has been in use for the past several years and works just
>fine for smaller files.  However, I have a very large file, 3.45 Gig
>(3704612914 bytes) containing 7,379,707 records each one being 502
>bytes long with the last 2 bytes of each record being crlf.

,,,,


Wed, 09 Mar 2005 11:25:32 GMT  
 Problem with very large files in NT
What do IBM say about this?

Hugh


Quote:
>I have a general purpose program that assigns a code to a specified
>location in each record.  I.E. I want to put "0" at position 499 in
>every single record.

>This program has been in use for the past several years and works just
>fine for smaller files.  However, I have a very large file, 3.45 Gig
>(3704612914 bytes) containing 7,379,707 records each one being 502
>bytes long with the last 2 bytes of each record being crlf.

>I am using VisualAge(TM) PL/I for Windows(R)  V2.R1.08

>When I run the program I get the following error.

>IBM0841I  ONCODE=1040  
>A data set, open for output, used all available space.
>At offset +00000175 in a BEGIN block                                  
>        From offset +000005F4 in procedure with entry UNICOD

>The data file is then corrupted because the size is changed to
>4,294,967,798 bytes (which happens to be 4 Gig + 502 bytes)

>When I step through the program with the de{*filter*} the file size is
>changed when the first record is read.

>When the On Error executes the output is
>Error at record:              1

>Here is the code

>%process langlvl(saa2) margins(1,96) limits(extname(31)) macro;
>%PROCESS RULES(NOLAXDCL NOLAXIF);
>*PROCESS LIMITS(EXTNAME(100) fixedbin(63) fixeddec(31) name(100) );

>unicod:procedure options (main);
>on error begin;
>   put skip edit('Error at record: ' || count)(a);
>   call pliretc(212);
>   resignal;
>end;

>/*Declarations*/
>dcl buffer char(2048) var;
>dcl infile file record;
>dcl innam char(132) var;
>dcl code char(80) var init ('');
>dcl yn char(1) init('');
>dcl (count,startcol,codlen,existing) fixed bin(31) init(0);

>on endfile(infile) goto exit;

>   put skip edit(
>   'This program allows insertion of a code into a record at a
>position that',
>   'you specify.  Codes may be up to 80 characters in length, and are
>entered',
>   'exactly as you type them in.')(a,skip,a,skip,a);

>   display('Name of the input file  : ') reply(innam);
>   open file (infile) title('/'||innam) update;

>   put skip list ('Code Control Information ...');
>   put skip edit('  Code to insert : ')(a);  get edit(code)(l);

>   do while(yn^='Y' & yn ^='N');
>     put skip edit('  Shall I write over existing data? (Y/N)')(a);
>get edit(yn)(l);
>     yn = translate(yn,'YN','yn');
>     if (yn ^= 'Y' & yn ^= 'N')
>      then put skip edit('Only "Y" and "N" are valid responses.')(a);
>   end;

>   put skip list('What is the starting column number for this code ?
>');
>   get edit(startcol)(f(8));
>   codlen = length(code);

>   do while ('1'b);
>      begin;
>           read file (infile) into (buffer);
>           count=count+1;
>           if yn = 'Y' then do;
>               substr(buffer,startcol,codlen)=code;
>               rewrite file (infile) from (buffer);
>           end;
>           else do;
>               if substr(buffer,startcol,codlen)=' ' then do;
>                   substr(buffer,startcol,codlen)=code;
>                   rewrite file (infile) from (buffer);
>               end;
>               else existing = existing + 1;
>               /* put skip list('Didn''t overwrite existing data at
>',count)*/
>           end;
>      end;
>   end;

>exit:
>   close file(infile);
>   put skip list ('The number of records processed is       :',count);
>   put skip list ('The number of records not overwritten is
>:',existing);
>   put skip list ('The total number of records coded is     :',
>   count-existing);
>   put skip list ('UNICOD is done');
>end unicod;



Wed, 09 Mar 2005 21:06:06 GMT  
 Problem with very large files in NT

Quote:

> What file system are  you using?

NTFS


Thu, 10 Mar 2005 00:28:08 GMT  
 Problem with very large files in NT
There is a 2GB limit on sequential files using VA PL/I (Windows) I/O.

I already tried opening a PMR on this, I was told it was a consequece of the
underlying C library routines - ANSI C uses signed 32-bit file offsets. You
can, of course, try again.

The only resolution I have found is to use native NT I/O, which supports
64-bit file sizes.


Quote:
> I have a general purpose program that assigns a code to a specified
> location in each record.  I.E. I want to put "0" at position 499 in
> every single record.

> This program has been in use for the past several years and works just
> fine for smaller files.  However, I have a very large file, 3.45 Gig
> (3704612914 bytes) containing 7,379,707 records each one being 502
> bytes long with the last 2 bytes of each record being crlf.

> I am using VisualAge(TM) PL/I for Windows(R)  V2.R1.08

> When I run the program I get the following error.

> IBM0841I  ONCODE=1040
> A data set, open for output, used all available space.
> At offset +00000175 in a BEGIN block
>         From offset +000005F4 in procedure with entry UNICOD

> The data file is then corrupted because the size is changed to
> 4,294,967,798 bytes (which happens to be 4 Gig + 502 bytes)

> When I step through the program with the de{*filter*} the file size is
> changed when the first record is read.

> When the On Error executes the output is
> Error at record:              1

> Here is the code

> %process langlvl(saa2) margins(1,96) limits(extname(31)) macro;
> %PROCESS RULES(NOLAXDCL NOLAXIF);
> *PROCESS LIMITS(EXTNAME(100) fixedbin(63) fixeddec(31) name(100) );

> unicod:procedure options (main);
> on error begin;
>    put skip edit('Error at record: ' || count)(a);
>    call pliretc(212);
>    resignal;
> end;

> /*Declarations*/
> dcl buffer char(2048) var;
> dcl infile file record;
> dcl innam char(132) var;
> dcl code char(80) var init ('');
> dcl yn char(1) init('');
> dcl (count,startcol,codlen,existing) fixed bin(31) init(0);

> on endfile(infile) goto exit;

>    put skip edit(
>    'This program allows insertion of a code into a record at a
> position that',
>    'you specify.  Codes may be up to 80 characters in length, and are
> entered',
>    'exactly as you type them in.')(a,skip,a,skip,a);

>    display('Name of the input file  : ') reply(innam);
>    open file (infile) title('/'||innam) update;

>    put skip list ('Code Control Information ...');
>    put skip edit('  Code to insert : ')(a);  get edit(code)(l);

>    do while(yn^='Y' & yn ^='N');
>      put skip edit('  Shall I write over existing data? (Y/N)')(a);
> get edit(yn)(l);
>      yn = translate(yn,'YN','yn');
>      if (yn ^= 'Y' & yn ^= 'N')
>       then put skip edit('Only "Y" and "N" are valid responses.')(a);
>    end;

>    put skip list('What is the starting column number for this code ?
> ');
>    get edit(startcol)(f(8));
>    codlen = length(code);

>    do while ('1'b);
>       begin;
>            read file (infile) into (buffer);
>            count=count+1;
>            if yn = 'Y' then do;
>                substr(buffer,startcol,codlen)=code;
>                rewrite file (infile) from (buffer);
>            end;
>            else do;
>                if substr(buffer,startcol,codlen)=' ' then do;
>                    substr(buffer,startcol,codlen)=code;
>                    rewrite file (infile) from (buffer);
>                end;
>                else existing = existing + 1;
>                /* put skip list('Didn''t overwrite existing data at
> ',count)*/
>            end;
>       end;
>    end;

> exit:
>    close file(infile);
>    put skip list ('The number of records processed is       :',count);
>    put skip list ('The number of records not overwritten is
> :',existing);
>    put skip list ('The total number of records coded is     :',
>    count-existing);
>    put skip list ('UNICOD is done');
> end unicod;



Fri, 11 Mar 2005 13:46:33 GMT  
 
 [ 5 post ] 

 Relevant Pages 

1. HELP: Tcl-DP 4.0 NT performance problems with large RPC messages

2. USENIX Large Installation Sysadmin of Windows NT Conf (LISA NT) - CFP

3. USENIX Large Installation Sysadmin of Windows NT Conf (LISA NT) - CFP

4. Large .DBF/Large NTX Key Length Indexing Problem

5. Large file problem

6. Large file problem

7. problems with clipper and large files 350000 recs

8. Problem Clipper 5.2D with large files

9. Problem with large NTX files.

10. Solved: problem uploading large files after RH 8.0 upgrade

11. Swap file size problem with large programs on alpha/unix

12. readline()-problem (processing large files)

 

 
Powered by phpBB® Forum Software