[BBLISA] Odd Rsync Behavior...

[Edward Ned Harvey]

Actually, I have the same behavior - with files "vanished" - But when I do my rsync, the source is a snapshot.  So it's impossible for the file to have changed in between the initial scan and actual attempted copy.  So truly I have no explanation.  And I am made uneasy by this as well.
 
I agree with something someone else said though - Put the command into a wrapper script to ensure you never have two concurrent rsync's doing the same thing.
 
There is some real value also, in making sure you have snapshots of the destination, or perhaps multiple destinations so you have some number of days of history.  It really stinks when somebody loses a file, and wants to restore it, only to discover it was deleted from the backups too.
 
Since you mentioned dump & tar - Dump is written by the people who write the filesystem (ufs, ext3, etc) so there's no such thing as dump for an NFS mount.  You could only run dump on the actual server that does the serving.  When dump works right, it works well, but it often doesn't work right.  Meaning - I've had serious problems before, when the version of dump&restore which created&restored the backup was different from the version of mkfs which formatted the filesystem, which yielded corrupted restored files.  No warnings, no errors . So if you use dump, use religious obsession to ensure the same versions.  Also, you're technically not allowed to dump a filesystem while it's mounted, but everybody does anyway.  Personally, when I use dump, I always prepare in advance by installing the OS twice.  Once to be the real OS, and one minimal version just for the purpose of doing dump on the original, unmounted filesystem.  By installing the OS twice, and using dump in the 2nd to backup the 1st, I ensure the dump I use is the same as the filesystem version, and I have a repeatable restore procedure if needed sometime in the future.  Tar doesn't have the same problem (assuming you stay with gnu tar all around) but I've seen some limitation on tar's ability to handle enormous backups.
 
And lastly - If you're using a mac to do the rsync, I hope you remembered to format the filesystem case sensitive.  By default, OSX formats case insensitive.  Yah, I got bit by that one too.
 
 
 
From: bblisa-bounces at bblisa.org [mailto:bblisa-bounces at bblisa.org] On Behalf Of Richard 'Doc' Kinne
Sent: Friday, January 09, 2009 4:12 PM
To: bblisa at bblisa.org
Subject: [BBLISA] Odd Rsync Behavior...
 
Folks:
Thanks much for all of the responses yesterday with regard to my backup issue.
 
One of the advantages I thought I had with regard to the "cp" method was, if something turned up missing, I'd be able to nearly "click-and-drag" the backup file back to the system thereby restoring it. This was one of the reasons I'd not investigated "dump" or "tar" more closely yet.
 
I decided to try "rsync" because not only did some of you folks recommend and use it, but also I had a friend in MN who recommended it.
 
The partition I am trying to back up is coming from an NFS mounted volume and going to an attached external Firewire drive. The command I used in this test was:
 
/usr/bin/rsync -ral --progress --stats --times --perms /Volumes/mira/scsi /Volumes/burly_1/Mira/ --delete
 
Initially /Volumes/burly_1/Mira was empty.
 
All went fine for a few hours. Then, seemingly toward the end, with the "web" directory (probably the most important, of course!) I started getting lines like:
 
file has vanished: "/Volumes/mira/scsi/web/vsx/submit/qry_getRemarks.php"
file has vanished: "/Volumes/mira/scsi/web/vsx/submit/qry_getStar.php"
file has vanished: "/Volumes/mira/scsi/web/vsx/submit/qry_getVarTypes.php"
file has vanished: "/Volumes/mira/scsi/web/vsx/submit/qry_updateMethod.php"
scsi/web/zapper/
file has vanished: "/Volumes/mira/scsi/web/zapper/.DS_Store"
file has vanished: "/Volumes/mira/scsi/web/zapper/._index.shtml"
file has vanished: "/Volumes/mira/scsi/web/zapper/._zapper.jnlp"
file has vanished: "/Volumes/mira/scsi/web/zapper/._zapperButton.jpg"
file has vanished: "/Volumes/mira/scsi/web/zapper/._zapperdoc.shtml"
file has vanished: "/Volumes/mira/scsi/web/zapper/._zapperhelp.shtml"
file has vanished: "/Volumes/mira/scsi/web/zapper/Zapper.jar"
 
This is a small sample toward the end. There were hundreds of lines like this, almost as if rsync had just decided to give up.
 
I went to /Volumes/mira/scsi/web/* and I was able to determine that these files were indeed there!
 
Rsync ended with this data:
 
Number of files: 766415
Number of files transferred: 746551
Total file size: 66995757081 bytes
Total transferred file size: 66995737633 bytes
Literal data: 43101421090 bytes
Matched data: 0 bytes
File list size: 17422528
File list generation time: 779.208 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 43129805144
Total bytes received: 16543326
 
sent 43129805144 bytes  received 16543326 bytes  6411523.66 bytes/sec
total size is 66995757081  speedup is 1.55
rsync warning: some files vanished before they could be transferred (code 24) at /SourceCache/rsync/rsync-35.2/rsync/main.c(992) [sender=2.6.9]
 
============
 
I'm not sure where to go from here. The data says that 746551 files were transfered out of 766415, but I'm not even sure about that.
 
The number of bytes in the "Total size" and in the "Literal data" are confirmed by running "du" on those areas, so those numbers are right.
 
I'm not sure what would stop the rsync, or what I may be doing wrong.  Any thoughts from those who've used this?
---
Richard 'Doc' Kinne, [KQR]
American Association of Variable Star Observers
<rkinne @ aavso.org>
 
 
 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.bblisa.org/pipermail/bblisa/attachments/20090110/0509ff50/attachment.htm