Perl, Doing two tasks at once

Doing two tasks at once

Author	Message
Fr€d #1 / 2	Doing two tasks at once I am trying to distribute some work among several machines. The basic task is to acquire a file (I'm doing it via Net::FTP), then run my perl script on it, then send the results back (probably via ftp as well). ie: repeat { FTP next file from server; system("perl calc.pl filename"); FTP results back to server; Quote: } The input files are 60-150Mb. The calculations take about the same time as transferring the file, often less time. So, I would like to ftp the next file while I'm doing the calculations. The calculations are basically CPU bound anyway, so this seems like a good fit that wouldn't slow things down very much. ie: repeat { while (FTP'ing the next file) do calcs on previous file; wait (if needed) until FTP is done; FTP results back to server; Quote: } Any pointers on where to look; I've browsed thru threads, exec() and system() so far, but haven't figured how to put it together yet. This would be running on Win32 / ActiveState Perl thanks
Wed, 30 Mar 2005 18:38:15 GMT

Tan Nguye #2 / 2	Doing two tasks at once Quote: > I am trying to distribute some work among several machines. The basic > task is to acquire a file (I'm doing it via Net::FTP), then run my perl > script on it, then send the results back (probably via ftp as well). > ie: > repeat { > FTP next file from server; > system("perl calc.pl filename"); > FTP results back to server; > } > The input files are 60-150Mb. The calculations take about the same time > as transferring the file, often less time. So, I would like to > ftp the next file while I'm doing the calculations. The calculations > are basically CPU bound anyway, so this seems like a good fit that > wouldn't slow things down very much. > ie: > repeat { > while (FTP'ing the next file) > do calcs on previous file; > wait (if needed) until FTP is done; > FTP results back to server; > } > Any pointers on where to look; I've browsed thru threads, exec() and > system() so far, but haven't figured how to put it together yet. This > would be running on Win32 / ActiveState Perl > thanks You might want to consider "fork". I'd recommend you fork two separate processes - one for copying files over to your local box, and one for computation on files (and sending results back). The former simply drops files into a designated directory, the latter keeps scanning for new files, moves them to another directory for processing. The thing you have to watch out for is you pick up and process a file in the middle of a FTP download. I recommend you save FTPed files with a certain extention such as .tmp while downloading and rename them to the original name afterwards. If you keep everything on one disk, Perl built-in function "rename" should suffice, otherwise, use File::Copy "move" function. From my understanding, they are both atomic.
Wed, 30 Mar 2005 20:58:18 GMT

Page 1 of 1

[ 2 post ]