speeding up access and processing of 650,000 record database 
Author Message
 speeding up access and processing of 650,000 record database

        I currently have a program which takes a game file (consisting
of 60 moves) and then searches a 650,000 record database file (46
megabyte) to count game configurations that are consistent with the
moves at each move. Currently the program takes 3 hours on a single
game file which is much too long.  I've tried to load the whole
database file in an array on a 64 megabyte Sparc II , but Perl gives
me an "Out of Memory!" error message which I find hard to believe. Do
I have to reconfigure Perl in order to exploit the memory available on
the workstation? If so , how?
        I believe a significant amount of processing time is spent in
comparing each record in the database file with each move in the game
file. Currently I use the pattern matching operator =~ /pattern/ to
test equality between the accumulated move pattern and the record
pattern. Is there a faster way to do this? I've appended the
subroutine which processes the game file below (any help would be much
appreciated):

# the database is structured so that a record consists of an 8
# numeric string that is the configuration and a 64 numeric string
# that is the pattern.
$template="A8 A64";
sub process_datafile {
    local($recnum);
    $recnum = 1;
    open(BB,"<$bb_database") || die "Couldn't open $bb-database";
       while ( (read(BB,$record,$recordsize))) {
           #process record.
           ( $config , $pattern ) = unpack ($template, $record);
           #process moves of datafile
           for ($j = 1; $j <= $game ; $j++ ) {
               #variables for each game
               for ($k = 1; $k <= 32 ; $k++) {
                   $gamepattern[$k] = "\\d\\d"; }
               for ($i = 1; $i <= $moves[$j]; $i++  ) {
                   #variables for each move
                   if ($event{$j,$i} =~ /Shoot Ray/){
                       if ($to{$j,$i} < 10) {
                           $gamepattern[$from{$j,$i}] = "0" . $to{$j,$i}; }
                       else {
                           $gamepattern[$from{$j,$i}] = $to{$j,$i} ; }

                       if ($pattern =~ /$testpattern/ ) {
                           #yes $testpattern matches $pattern
                           $yes{$j,$i} = $yes{$j,$i} + 1; # of patterns in database
                                                      # consistent with game pattern
                       }
                       else { # no need to continue to cycle through moves since $pattern is not consistent
                           last ; # exits loop through moves for statement.
                       }
                   }
               }
           }
            $recnum++;
       }

Quote:
}

--
---------------------------------------------------------------------

Laboratory for Knowledge Based Medical Systems
571 Health Science Library,376 W. 10th , Columbus, Ohio, 43210


Fri, 17 Feb 1995 22:06:32 GMT  
 
 [ 1 post ] 

 Relevant Pages 

1. mySQL - More records or longer records - Speed?

2. PHP vs PERL speed when accessing MYSQL database

3. Delete record from fixed-length random access databases

4. Perl Database interface to a Microsoft Access database MS-Access

5. Perl Database interface to a Microsoft Access database MS-Access

6. Perl Database interface to a Microsoft Access database MS-Access

7. Accessing the last row of a Access Database

8. Accessing MS-Access database with Perl

9. perl to access MS Access database

10. Using Perl To Access Microsoft Access Databases???

11. Accessing MS Access Database from Unix

12. Accessing MS Access Database from Perl (Solaris)

 

 
Powered by phpBB® Forum Software