[jaypz]

I am looking for a command line solution to a backup problem. I usually use rsync, which works great; however, I have a large quantity (~170GB) of text files. These files shrink down a lot when compressed, and if I can sync to a compressed copy or tar-type archive, that would be ideal. Any ideas?

Of course, tar has compression options, but isn't ideal for differential backups. As I understand it, the update (-u) feature appends changed files to the end of the archive, so as you run it over and over your archive will get unmanageably huge.

What I want, I think, is basically tar -cjvuf but with an update that deletes the old copy and replaces it with the new one.....any suggestions?

[mduell]

Using tar seems like a difficult and obtuse solution. Why not just use the -z (--compress) option in rsync?

[jaypz]

My understanding of the rsync compression option is that it compresses data during transfer, but not at the final destination. So it saves bandwidth during the sync operation, but not on the other end. My main concern is just that---storage size for the backup. I would like to sync a local copy of documents for example with a remote file documents.tgz, or something like that. Maybe this isn't possible...

[Gavin]

you're right about rsync's compression during transfer

you may need a little more scripting than a single rsync command to get this done

like a cron job to tar up your files on the server,
then rsync that file,
then another action that uncompresses the file and finally rsync or untar it locally


I don't know if rsync is good enough to just get the changes within a compressed file like it can do with a text file.

I have 3 or 4 Gb that I back up with rsync, it takes it about 1 min. to run unless someone has added a video file or lots of pics. An earlier version or rsync was choking on large numbers of files so I broke it into several calls, each getting a subfolder.

you could have several backups running concurrently - each getting a subfolder. There is a trade off between bandwidth and processor time in archiving all that data.