Checking if file writing by another process is complete 
Author Message
 Checking if file writing by another process is complete

I'm writing a store and forward FTP application to pick up files in NT
directories and move them from a staging area.  The files are deposited at
random times, and some are quite large, >6GB.  I obviously don't want to
start the FTP tranfer of the files until they have been completely written
to the staging area.  Since the file directory entry is created at the time
the transfer begins to the staging area, I don't know how to test when the
transfer to the staging area is complete.  I've looked at Lockfile::Simple,
Win32::File and Win32API::File, Win32::NetAdmin, Win32::NetResource,
Win32::Process, Win32API::Net.  None seem to be able to tell when a file is
"closed" for writing by other processes.

I would like to do something simple like:
    if ( ! $fileopen =
SOME_UNKNOWN_PACKAGE::some_unknown_method($filename) ) {
            ....don't process file, some other process has it open for
writing
    } else {
           ....begin FTP process
    }

Has anyone run across this before?



Mon, 08 Dec 2003 20:31:17 GMT  
 Checking if file writing by another process is complete

[mailed and posted]

Quote:
> I'm writing a store and forward FTP application to pick up files in
> NT directories and move them from a staging area.  Since the file
> directory entry is created at the time the transfer begins to the
> staging area, I don't know how to test when the transfer to the
> staging area is complete.

I've done this in the past, and the cleanest solution is to have the
transmitter deposit a second, empty file after the first one is
complete.  The name of the empty file should be related to the name of
the data file.  The receiver can then check to see if the empty file
is present or not; if it is, it represents the sender's guarantee that
the data file has been sent complete.  

This is simple and portable, and robust in the face of host crashes
and network outages.  For example, if the sender's host crashes with
the file still incomplete, the sender can simply retransmit the file
when it comes back up; since it hasn't sent the empty file yet, there
is no chance of the receiver picking up the incomplete file.  If the
receiver explicitly checks for 'file still open', as you suggest, it
will fail in this case.

Quote:
>     if ( ! $fileopen =
> SOME_UNKNOWN_PACKAGE::some_unknown_method($filename) ) {
>             ....don't process file, some other process has it open for
> writing
>     } else {
>            ....begin FTP process
>     }

This becomes:

        if (-e "$file.COMPLETE") {
          # begin ftp $file process
        } else {
          # don't
        }



Mon, 08 Dec 2003 21:37:31 GMT  
 Checking if file writing by another process is complete

Quote:

> I've done this in the past, and the cleanest solution is to have the
> transmitter deposit a second, empty file after the first one is
> complete.  The name of the empty file should be related to the name of
> the data file.  The receiver can then check to see if the empty file
> is present or not; if it is, it represents the sender's guarantee that
> the data file has been sent complete.

A similar alternative, IMHO a little cleaner, is to write to the file as
one name and rename it after closure. The second process can rename it
again if it wishes.

It has the additional advantage that if you have an older (but
acceptable) version of the file, it can be left in place until the
rename is ready. Delete/rename isn't atomic, but is pretty damn' close.

Ian



Tue, 09 Dec 2003 00:23:11 GMT  
 Checking if file writing by another process is complete

Quote:

> Delete/rename isn't atomic, but is pretty damn' close.

ISTR renaming over an existing file is atomic on a POSIX compliant system.

Oh, dear I appear to have drifted OT.

--
     \\   ( )
  .  _\\__[oo

 .  l___\\
  # ll  l\\
 ###LL  LL\\



Tue, 09 Dec 2003 01:00:39 GMT  
 Checking if file writing by another process is complete


Quote:

> [mailed and posted]
> > I'm writing a store and forward FTP application to pick up files in
> > NT directories and move them from a staging area.  Since the file
> > directory entry is created at the time the transfer begins to the
> > staging area, I don't know how to test when the transfer to the
> > staging area is complete.

> I've done this in the past, and the cleanest solution is to have the
> transmitter deposit a second, empty file after the first one is
> complete.  The name of the empty file should be related to the name of
> the data file.  The receiver can then check to see if the empty file
> is present or not; if it is, it represents the sender's guarantee that
> the data file has been sent complete.

works, but over complicated

mv .name name

atomic, simple & easy.

( for perl mv == rename; same thing though, really )

Tried & tested, by millions of programmers worldwide.

HTH

James



Tue, 09 Dec 2003 11:48:49 GMT  
 
 [ 5 post ] 

 Relevant Pages 

1. Egavga.bgi

2. Checking if write to file is complete

3. Help: need to check file not being written

4. Checking, locking, writing files

5. Backticks waits for all processes to complete

6. Win32::Process and completing cgi script

7. I'm looking for someone who wants to write a complete perl-script (for free)

8. Win32::Process and completing CGI script

9. Reading from a file another process has written

10. files in the process of being written

11. files in the process of being written to

12. tie()ing STDOUT to a file being written to by anothe (child) process

 

 
Powered by phpBB® Forum Software