Doing two tasks at once 
Author Message
 Doing two tasks at once

I am trying to distribute some work among several machines.  The basic
task is to acquire a file (I'm doing it via Net::FTP), then run my perl
script on it, then send the results back (probably via ftp as well).

ie:

repeat {
  FTP next file from server;
  system("perl calc.pl filename");
  FTP results back to server;

Quote:
}

The input files are 60-150Mb. The calculations take about the same time
as transferring the file, often less time.  So, I would like to
ftp the next file while I'm doing the calculations.  The calculations
are basically CPU bound anyway, so this seems like a good fit that
wouldn't slow things down very much.

ie:

repeat {
  while (FTP'ing the next file)
    do calcs on previous file;
  wait (if needed) until FTP is done;
  FTP results back to server;

Quote:
}

Any pointers on where to look; I've browsed thru threads, exec() and
system() so far, but haven't figured how to put it together yet.  This
would be running on Win32 / ActiveState Perl

thanks



Wed, 30 Mar 2005 18:38:15 GMT  
 Doing two tasks at once

Quote:

> I am trying to distribute some work among several machines.  The basic
> task is to acquire a file (I'm doing it via Net::FTP), then run my perl
> script on it, then send the results back (probably via ftp as well).

> ie:

> repeat {
>   FTP next file from server;
>   system("perl calc.pl filename");
>   FTP results back to server;
> }

> The input files are 60-150Mb. The calculations take about the same time
> as transferring the file, often less time.  So, I would like to
> ftp the next file while I'm doing the calculations.  The calculations
> are basically CPU bound anyway, so this seems like a good fit that
> wouldn't slow things down very much.

> ie:

> repeat {
>   while (FTP'ing the next file)
>     do calcs on previous file;
>   wait (if needed) until FTP is done;
>   FTP results back to server;
> }

> Any pointers on where to look; I've browsed thru threads, exec() and
> system() so far, but haven't figured how to put it together yet.  This
> would be running on Win32 / ActiveState Perl

> thanks

You might want to consider "fork".  I'd recommend you fork two separate
processes - one for copying files over to your local box, and one for
computation on files (and sending results back). The former simply drops
files into a designated directory, the latter keeps scanning for new files,
moves them to another directory for processing. The thing you have to watch
out for is you pick up and process a file in the middle of a FTP download. I
recommend you save FTPed files with a certain extention such as .tmp while
downloading and rename them to the original name afterwards.  If you keep
everything on one disk, Perl  built-in function "rename" should suffice,
otherwise, use File::Copy "move" function. From my understanding, they are
both atomic.


Wed, 30 Mar 2005 20:58:18 GMT  
 
 [ 2 post ] 

 Relevant Pages 

1. doing a replace task in a file

2. Working with two streams at once: ack pth!

3. Matching two patterns at once

4. File I/O: want to compare two or more lines @ once

5. Two scripts at once?

6. Running Two Scripts at Once

7. Sending results to two frames at once

8. Perl to two Frames at once?

9. Two Text files at once?

10. Three questions related to SNMP, external task handling, paralell tasking

11. Comparison of two fields in two text files

12. passing two arrays two a sub

 

 
Powered by phpBB® Forum Software