Use of Arrays Vs output .dbf files - for counts in large files 
Author Message
 Use of Arrays Vs output .dbf files - for counts in large files

Recently, I was given a task to develop 6 different reports. Each of
the 6 representing counts for all specific values of six different
fields within one record.

The problem was the source .dbf file, which is 1,250,000 records long.

I could come up with 2 schemes to do this in a very short programming
time.

I.  Using 6 output .dbf files of the same structure. Examine each      
    record then accumulate  or initiate a counter in the appropriate  
 output files.

2. Use a combination of single and 2 dimentional arrays to accomplish
    the same.

I decided to go with the scheme #2 which seemed to have the advantage
of faster processing using the memory exclusively for counting and
fewer accesses to the disc.. I thought this would improve the
processing time.

When I did some tests on smaller test files I was totally dissapointed
with the speed.

This prompted me to program the scheme # 1 as a comparison excercise.

I was totaly surprised at the improvement in speed of processing with
scheme #1.

I am not very experienced with array implementation. I am sure I am
missing a critcal step in using arrays.

Is there an experienced programmer here who could please starighten me
out?

The processing time by the way was 37 hours using a 486 66 MHZ machine
on a Novell Netware the latest version server, using scheme #2.

Thanks    ......... Syed Hussain



Thu, 19 Feb 1998 03:00:00 GMT  
 Use of Arrays Vs output .dbf files - for counts in large files

Quote:

>I.  Using 6 output .dbf files of the same structure. Examine each      
>    record then accumulate  or initiate a counter in the appropriate  
> output files.

>2. Use a combination of single and 2 dimentional arrays to accomplish
>    the same.

>I decided to go with the scheme #2 which seemed to have the advantage
>of faster processing using the memory exclusively for counting and
>fewer accesses to the disc.. I thought this would improve the
>processing time.

>When I did some tests on smaller test files I was totally dissapointed
>with the speed.

Clipper programs with large arrays tend to run slowly when linked in real mode.
I understand that linking in protected mode with Blinker and running under
adequate extended memory may speed up the program.


Fri, 20 Feb 1998 03:00:00 GMT  
 
 [ 3 post ] 

 Relevant Pages 

1. DBU DBF FILE nad FOXPRO 2.6A dbf file

2. Reading an array from file and outputting another file

3. Indexing a very large DBF file

4. Largest .DBF and .NTX files worked with

5. Outputting info from awk and using the info to name the awk output file

6. How to obtain the record count of very large files

7. Printing output corrupts dbf-files

8. reading file and count and decimate array

9. Converting Clarion file to .dbf file

10. Updating a TPS file with DBF file

11. Using TBrowse without a DBF file

12. Unable to read dbf file as dbase file

 

 
Powered by phpBB® Forum Software