[BBLISA] Backing up sparse files ... VM's and TrueCrypt ... etc

[John Hanks]

Could you accomplish what you want with xdelta? I've been interested
in something like this for versioning whole disk images used with VMs
and AMIs. Example follows which shows that xdelta is pretty speedy at
least in my simplified 1 GB sparse file test. For the cost of some
very small delta files I assume I could restore the image to any
version. Shouldn't be too much work to script something around this.

jbh


[F 100%] griznog at amusium ~/sparse $ dd if=/dev/zero of=disk.img
bs=1024 seek=1048576 count=0
0+0 records in
0+0 records out
0 bytes (0 B) copied, 1.0546e-05 s, 0.0 kB/s
[F 100%] griznog at amusium ~/sparse $ ls -alh
total 12K
drwxr-xr-x  2 griznog griznog   21 Feb 20 16:50 .
drwx------ 86 griznog griznog 8.0K Feb 20 16:42 ..
-rw-r--r--  1 griznog griznog 1.0G Feb 20 16:50 disk.img
[F 100%] griznog at amusium ~/sparse $ cp --sparse=always disk.img disk.img.bkup
[F 100%] griznog at amusium ~/sparse $ ls -alh
total 12K
drwxr-xr-x  2 griznog griznog   41 Feb 20 16:51 .
drwx------ 86 griznog griznog 8.0K Feb 20 16:42 ..
-rw-r--r--  1 griznog griznog 1.0G Feb 20 16:50 disk.img
-rw-r--r--  1 griznog griznog 1.0G Feb 20 16:51 disk.img.bkup
[F 100%] griznog at amusium ~/sparse $ sudo losetup /dev/loop0 disk.img
[F 100%] griznog at amusium ~/sparse $ echo ",,82" | sudo tee  /dev/loop0
,,82
[F 100%] griznog at amusium ~/sparse $ sudo losetup -d /dev/loop0
[F 100%] griznog at amusium ~/sparse $ ls -alhs
total 16K
   0 drwxr-xr-x  2 griznog griznog   41 Feb 20 16:51 .
 12K drwx------ 86 griznog griznog 8.0K Feb 20 16:42 ..
4.0K -rw-r--r--  1 griznog griznog 1.0G Feb 20 16:50 disk.img
   0 -rw-r--r--  1 griznog griznog 1.0G Feb 20 16:51 disk.img.bkup
[F 100%] griznog at amusium ~/sparse $ xdelta delta disk.img.bkup
disk.img disk.inc00
[F 100%] griznog at amusium ~/sparse $ ls -al
total 20
drwxr-xr-x  2 griznog griznog         58 Feb 20 16:58 .
drwx------ 86 griznog griznog       8192 Feb 20 16:42 ..
-rw-r--r--  1 griznog griznog 1073741824 Feb 20 16:50 disk.img
-rw-r--r--  1 griznog griznog 1073741824 Feb 20 16:51 disk.img.bkup
-rw-r--r--  1 griznog griznog        220 Feb 20 16:58 disk.inc00
[F 100%] griznog at amusium ~/sparse $ mke2fs -L 'disk' -m 0 -j disk.img
mke2fs 1.41.9 (22-Aug-2009)
disk.img is not a block special device.
Proceed anyway? (y,n) y
Filesystem label=disk
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
65536 inodes, 262144 blocks
0 blocks (0.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=268435456
8 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
	32768, 98304, 163840, 229376

Writing inode tables: done
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 20 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.
[F 100%] griznog at amusium ~/sparse $ ls -alhs
total 49M
   0 drwxr-xr-x  2 griznog griznog   58 Feb 20 16:58 .
 12K drwx------ 86 griznog griznog 8.0K Feb 20 16:42 ..
 49M -rw-r--r--  1 griznog griznog 1.0G Feb 20 17:00 disk.img
   0 -rw-r--r--  1 griznog griznog 1.0G Feb 20 16:51 disk.img.bkup
4.0K -rw-r--r--  1 griznog griznog  220 Feb 20 16:58 disk.inc00
[F 100%] griznog at amusium ~/sparse $ time xdelta delta disk.img.bkup
disk.img disk.inc01

real	0m11.101s
user	0m10.213s
sys	0m0.812s
[F 100%] griznog at amusium ~/sparse $ ls -alhs
total 49M
   0 drwxr-xr-x  2 griznog griznog   75 Feb 20 17:00 .
 12K drwx------ 86 griznog griznog 8.0K Feb 20 16:42 ..
 49M -rw-r--r--  1 griznog griznog 1.0G Feb 20 17:00 disk.img
   0 -rw-r--r--  1 griznog griznog 1.0G Feb 20 16:51 disk.img.bkup
4.0K -rw-r--r--  1 griznog griznog  220 Feb 20 16:58 disk.inc00
 16K -rw-r--r--  1 griznog griznog  14K Feb 20 17:00 disk.inc01
[F 100%] griznog at amusium ~/sparse $ mkdir disk
[F 100%] griznog at amusium ~/sparse $ sudo mount -o loop disk.img disk
[F 100%] griznog at amusium ~/sparse $ sudo cp
/home/griznog/Downloads/pvfs2-quickstart.pdf disk/
[F 100%] griznog at amusium ~/sparse $ ls -al disk/
total 128
drwxr-xr-x 3 griznog griznog   4096 Feb 20 17:02 .
drwxr-xr-x 3 griznog griznog     86 Feb 20 17:01 ..
drwx------ 2 root    root     16384 Feb 20 17:00 lost+found
-rw-r--r-- 1 root    root    104524 Feb 20 17:02 pvfs2-quickstart.pdf
[F 100%] griznog at amusium ~/sparse $ sudo umount disk
[F 100%] griznog at amusium ~/sparse $ time xdelta delta disk.img.bkup
disk.img disk.inc02

real	0m11.049s
user	0m10.233s
sys	0m0.717s
[F 100%] griznog at amusium ~/sparse $ ls -alhs
total 49M
   0 drwxr-xr-x  3 griznog griznog  103 Feb 20 17:03 .
 12K drwx------ 86 griznog griznog 8.0K Feb 20 16:42 ..
   0 drwxr-xr-x  2 griznog griznog    6 Feb 20 17:01 disk
 49M -rw-r--r--  1 griznog griznog 1.0G Feb 20 17:00 disk.img
   0 -rw-r--r--  1 griznog griznog 1.0G Feb 20 16:51 disk.img.bkup
4.0K -rw-r--r--  1 griznog griznog  220 Feb 20 16:58 disk.inc00
 16K -rw-r--r--  1 griznog griznog  14K Feb 20 17:00 disk.inc01
104K -rw-r--r--  1 griznog griznog 102K Feb 20 17:03 disk.inc02