The Old New Thing - Does the CopyFile function verify that the data reached its final destination successfully?
A customer had a question about data integrity via file copying.
I am using theFile.Copy
to copy files from one server to another. If the call succeeds, am I guaranteed that the data was copied successfully? Does theFile.Copy
method internally perform a file checksum or something like that to ensure that the data was written correctly?The
File.Copy
method uses the Win32CopyFile
function internally, so let's look atCopyFile
.
CopyFile
just issuesReadFile
calls from the source file andWriteFile
calls to the destination file. (Note: Simplification for purposes of discussion.) It's not clear what you are hoping to checksum. If you wantCopyFile
to checksum the bytes when the return fromReadFile
, and checksum the bytes as they are passed toWriteFile
, and then compare them at the end of the operation, then that tells you nothing, since they are the same bytes in the same memory....
But wait, there's also the problem of caching controllers. Even when you tell the hard drive, "Now read this data from the physical media," it may decide to return the data from an onboard cache instead. You would have to issue a "No really, flush the data and read it back" command to the controller to ensure that it's really reading from physical media.
And even if you verify that, there's no guarantee that the moment you declare "The file was copied successfully!" the drive platter won't spontaneously develop a bad sector and corrupt the data you just declared victory over.
This is one of those "How far do you really want to go?" type of questions. You can re-read and re-validate as much as you want at copy time, and you still won't know that the file data is valid when you finally get around to using it.
Sometimes, you're better off just trusting the system to have done what it says it did.
If you really want to do some sort of copy verification, you'd be better off saving the checksum somewhere and having the ultimate consumer of the data validate the checksum and raise an integrity error if it discovers corruption.
I've run into this issue... Using the standard file copy API's, files were not getting fully copied, but somehow corrupted in the middle somewhere. And due to the pipeline it wasn't being found for a while. No errors, not disk issues, not anti-virus, but the 0's and 1's were not the same (nor of course were their hashes). And it wasn't consistent. For months, no issues and then a bunch. And it wasn't all or even many files, a few in a million, but enough to be a headache. So I built in a series of validations, checks, retries and such to try to keep this from happening or at least recovering when it did. And all that time I really didn't know why it was happening. I mean, I thought I should be good using FileCopy/CopyFile, right? I thought I was crazy...
Ah... then I read Raymond's post. See I wasn't crazy!
No comments:
Post a Comment
NOTE: Anonymous Commenting has been turned off for a while... The comment spammers are just killing me...
ALL comments are moderated. I will review every comment before it will appear on the blog.
Your comment WILL NOT APPEAR UNTIL I approve it. This may take some hours...
I reserve, and will use, the right to not approve ANY comment for ANY reason. I will not usually, but if it's off topic, spam (or even close to spam-like), inflammatory, mean, etc, etc, well... then...
Please see my comment policy for more information if you are interested.
Thanks,
Greg
PS. I am proactively moderating comments. Your comment WILL NOT APPEAR UNTIL I approve it. This may take some hours...