[adamfishercox]

I will have a MacBook and an iMac in a few days, and I have this one piece of dream software in mind. I need you guys to tell me if there in anything like it that I can get.

My dream software would essentially be Time Machine between two computers. I would be able to select the folders I want to sync. From then on, when I make a change to a file on my MacBook, it will update said file on my iMac at the next time it is connected to the internet. Likewise, if I make a change on my iMac, it should sync to my MacBook. If I make a new file, it should create that new file on the other computer.

The end goal is to have the contents of my user folder remain the same on both computers without me constantly having to manually update them.

Is there any software that offers some semblance of this?

[Hal Itosis]

Chronosync perhaps.
[I'm not sure about that "next time it is connected to the internet" part.]

[besson3c]

Your dream software has existed for probably around 10 years or so. It's called rsync, and is triggered by a cronjob.

[adamfishercox]

f'rlz?

I knew it couldn't be too hard. More info, please?

[AKcrab]

I like psyncx:
PsyncX - Free Backup Software for Mac OS 10

No idea really what the difference between psync and rsync is, but it's flawlessly handled my syncing/backups for years.

[besson3c]

Psync used to have an advantage back in the day before rsync handled OS X metadata, but as of some prior version of OS X (at least 10.4) rsync has supported OS X metadata.

Rsync has always been faster than psync, and performs really well when dealing with a large number of files. When I benchmark a psync backup against an rsync backup, rsync beats psync handily.

Several OS X utilities actually use psync or rsync on the backend if you research this. For instance, Deja Vu uses psync.

We backup 6 terrabytes of data every night using rsync over fiber. It is an extremely stable and well performing tool.

[besson3c]

Originally Posted by adamfishercox 

f'rlz?

I knew it couldn't be too hard. More info, please?

Rsync is already installed on your Mac (unlike psync). To use it, simply enter the following into your terminal:

rsync -avE /path/to/sourcedirectory /path/to/destinationdirectory

This will copy over whatever files don't exist or have changed from source to destination. If you do a:

rsync -avE --delete /path/to/sourcedirectory /path/to/destinationdirectory

This will do the same thing but will also delete whatever files no longer exist in the destination directory - i.e. you will be mirroring your source directory with your destination directory. If you want to do a merge, you can run the first command twice reversing source and destination directory.

Where rsync gets really useful is that not only will it sync between local directories like in my above examples, but it will also sync across multiple machines. The only requirement is that both machines have SSH enabled ("remote login in your sharing pref pane"), and have a copy of rsync installed (rsync comes standard on most Unix/Linux distros these days including OS X). The transport mechanism between the two machines is SSH, so your data is sent through a secure, encrypted tunnel.

To do this:

rsync -avE /path/to/sourcedirectory remoteusername@remotemachine:/path/to/destinationdirectory

This will "push" data out to the destination directory. You can add in the --delete argument if you want to mirror your data, just like in the above example. If you want to "pull" data, simply reverse the above command so that the destination directory is listed first.

Now, you'll notice that each time you do a machine to machine sync that you will be prompted for a password. To get around this, you will need to setup SSH public key authentication. This is pretty easy to do, and there are many guides you can Google for that will help you do this.

The way public key authentication works is that a private/machine key is created, and it is paired with a public key. You'll copy your public key out to other machines you want to connect to and install it into the appropriate directory. When you connect to this machine, if the signature of your private key (which you need to keep safe) matches the signature of the public key, you will be able to authenticate without having to type in your password. This is called a challenge and response form of authentication, and is the basis for many authentication types including Kerberos, and even "what is your mother's maiden name?" in a sense. It is actually more secure than manually entering a password.

To automate these backups, simply setup a cron or launchd job to do this. There are many guides pertaining to this. You can have the sync run every minute, every 2 minutes, whatever. Rsync is so fast that if you could run a backup in well under a minute, you could probably invoke the backup every minute and you wouldn't notice a significant overhead.

[besson3c]

The -a flag means "archive" - it preserves permissions, file ownership, time stamps, etc. The -v flag is optional, but it provides a verbose output meaning that you can watch the files being synced. Since the output of a cronjob is normally emailed somewhere, you'll probably want to remove the -v flag in your cronjob. The -E flag means to honor and handle OS X metadata appropriately.

[AKcrab]

Originally Posted by besson3c 

Rsync has always been faster than psync, and performs really well when dealing with a large number of files. When I benchmark a psync backup against an rsync backup, rsync beats psync handily.

We backup 6 terrabytes of data every night using rsync over fiber. It is an extremely stable and well performing tool.

My backup needs are purely personal, so the speed of psync has never been an issue for me. I just spent a bit of time test driving some of the GUI rsync front ends, and sadly they all seem lacking. (At least the free ones.)

I'm not afraid of mucking about in the terminal, but I certainly prefer a GUI if it's decent. Psync makes it easy for me.

(Thanks for all the psync-rsync info.)

[besson3c]

No doubt the GUIs you saw tried to provide controls for all of the 203429834 options rsync provides?

If you are going to automate rsync jobs via cron, obviously a GUI is not desirable, but perhaps a GUI for doing simple syncs is a Mac developer opportunity waiting to happen!

[AKcrab]

Originally Posted by besson3c 

No doubt the GUIs you saw tried to provide controls for all of the 203429834 options rsync provides?

Actually, it was mostly the opposite. Two of them didn't interface at all with cron, and simply had a "do it now" button.

If you are going to automate rsync jobs via cron, obviously a GUI is not desirable, but perhaps a GUI for doing simple syncs is a Mac developer opportunity waiting to happen!

Wonder why the psyncX dev has stuck with psync instead of moving to rsync?

[besson3c]

Originally Posted by AKcrab 

Actually, it was mostly the opposite. Two of them didn't interface at all with cron, and simply had a "do it now" button.

Wonder why the psyncX dev has stuck with psync instead of moving to rsync?

Because then he'd have to call it rsyncX (which already exists) and rewrite his entire product?

[adamfishercox]

So, if I type

rsync -avE --delete /Users/Adam Adam@Adam's iMac:/Users/Adam

on my MacBook, It will make my iMac users folder mirror my MacBook users folder?

[mduell]

rsync is a great tool, but the right tool for the job is FolderShare. It does exactly what adamfishercox described that he wanted, without needing the computers to be on the same network or remembering IPs or poking holes in firewalls. Pick the folders you want to sync, run the app in the background, and viola, any time a file changes on one it's updated on the other over the internet.

[besson3c]

Originally Posted by mduell 

rsync is a great tool, but the right tool for the job is FolderShare. It does exactly what adamfishercox described that he wanted, without needing the computers to be on the same network or remembering IPs or poking holes in firewalls. Pick the folders you want to sync, run the app in the background, and viola, any time a file changes on one it's updated on the other over the internet.

Yeah, if you want your data transfers facilitated by a third party, and Microsoft nonetheless...

Not for me. I'm sure FolderShare isn't free either.

[adamfishercox]

Well, I downloaded it to test it out... and it SEEMS to be free... though it said "Try Free" there is no indication that it's only a trial. I'm going to give the cronjob a go first, though.

[besson3c]

Originally Posted by adamfishercox 

So, if I type

rsync -avE --delete /Users/Adam Adam@Adam's iMac:/Users/Adam

on my MacBook, It will make my iMac users folder mirror my MacBook users folder?

Assuming that "Adam's iMac" is network accessible (i.e. the name resolves properly), SSH is turned on and listening on Adam's Mac, then yes, except one change is necessary:

rsync -avE --delete /Users/Adam "Adam@Adam's\ iMac:/Users/Adam"

You'll need these changes for the syntax to negotiate the space in your computer name. If you literally want to connect to your computer by teh name of "Adam's Mac" and your Mac is actually called that, you'll need to do "Adam's\ iMac.local". In this case the machine will have to be on your LAN too. If you need to connect outside of your LAN you will need to open up port 22 in your firewall, and use your internet IP to connect. Alternatively, you could setup a dyndns name and enter this into your router config if you want a static domain you can use to connect to your home Mac from outside of your network.

[besson3c]

Originally Posted by adamfishercox 

Well, I downloaded it to test it out... and it SEEMS to be free... though it said "Try Free" there is no indication that it's only a trial. I'm going to give the cronjob a go first, though.

Make sure the command works manually before you create a cronjob, or else it is just going to keep dutifully attempting the badly formatted command.

[adamfishercox]

Ok, is there a way to exempt a folder from this?

Basically, Inside my Home folder I have a folder for my video projects. It's a lot of data that will eat up my MacBook and I don't need it there. Is there something I could tag on to that command to tell it to sync the user folder but not the contained "projects" folder?

[besson3c]

yup: --exclude=projects


You can read more in the rsync man page:

man rsync

type: /exclude

to search for "exclude"

[mduell]

Originally Posted by besson3c 

I'm sure FolderShare isn't free either.

Originally Posted by adamfishercox 

Well, I downloaded it to test it out... and it SEEMS to be free... though it said "Try Free" there is no indication that it's only a trial.

Yes, it's free. I've been using it for years.

[besson3c]

How does MS make money off of it then? MS and free usually do not belong in the same sentence...

[mduell]

They don't directly. Adds mindshare to the Live family of services.

[besson3c]

Hmmm... Well, I sure as hell wouldn't trust a service like this, call me irrational or whatever you want. Then again, I'm quite comfortable with rsync, so I have little reason to anyway

[mduell]

That's fine, and I wasn't (and wouldn't) suggesting it for you.

rsync is a great tool. Keeping a couple personal computers that may be far apart in sync isn't a great use for it for a non-geek.

[besson3c]

Yeah, I can see how FolderShare would be a good tool for some... Sorry, I didn't mean to discount your point with my anti-MS bias

[Hal Itosis]

Originally Posted by adamfishercox 

From then on, when I make a change to a file on my MacBook, it will update said file on my iMac at the next time it is connected to the internet. Likewise, if I make a change on my iMac, it should sync to my MacBook. If I make a new file, it should create that new file on the other computer.

Sounds like a bidirectional sync.

As far as command-line tools, I didn't know rsync did bidirectional operations easily.
I thought that's why all the geeks used Unison for that.    It's a ton of fun configuring
all those exclusions in a script file. Don't forget /Users/*/Library/Preferences/ByHosts

Let's see... what else? . . .

[adamfishercox]

yeah, you make a good point. With rsync, it sounds like any changes I make on my MacBook would be mirrored on my iMac, but any files I make on my iMac will just be deleted because they aren't on my MacBook.

I guess the way to do it would be run the rsync cronjob on both computers, and don't add the --delete.

[besson3c]

Originally Posted by adamfishercox 

yeah, you make a good point. With rsync, it sounds like any changes I make on my MacBook would be mirrored on my iMac, but any files I make on my iMac will just be deleted because they aren't on my MacBook.

I guess the way to do it would be run the rsync cronjob on both computers, and don't add the --delete.

Exactly.. But it doesn't have to be run on both computers. You can run the two commands on one of the computers.

[adamfishercox]

Oh, true.

[adamfishercox]

Ok, I am completely lost... I read some things on cronjobs and they made it worse ...

Can someone help me write a cronjob to have my iMac and MacBook sync hourly? This is as far as I got:

rsync -avE /Users/Adam "Adam@iMac:/Users/Adam" --exclude=projects --exclude=downloads
rsync -avE Adam@iMac:/Users/Adam "/Users/Adam" --exclude=projects --exclude=downloads

Did I use the "Excludes" right?

[besson3c]

Originally Posted by adamfishercox 

Ok, I am completely lost... I read some things on cronjobs and they made it worse ...

Can someone help me write a cronjob to have my iMac and MacBook sync hourly? This is as far as I got:

rsync -avE /Users/Adam "Adam@iMac:/Users/Adam" --exclude=projects --exclude=downloads
rsync -avE Adam@iMac:/Users/Adam "/Users/Adam" --exclude=projects --exclude=downloads

Did I use the "Excludes" right?

This should be:

rsync -avE --exclude="projects" --exclude="downloads" /Users/Adam/ Adam@iMac:
rsync -avE --exclude="projects" --exclude="downloads" Adam@iMac: /Users/Adam/

The arguments have to go before the source/destintation directories. The quotation marks are to help you use spaces in file names if you have to. In this above example, they are all optional since there aren't any extra spaces.

If you decide you want more excludes, there is an argument for pointing to a text file of directories to exclude. This might work better if you decide to exclude a lot more stuff.

Also, with the syntax for the remote machine, everything after the : is relative to the home directory of the person you are logging in as. Therefore, you could simplify by replacing Adam@iMac:/Users/Adam with simply Adam@iMac:

I included a trailing slash in /Users/Adam/ to indicate that the backup should be of the contents of /Users/Adam but should not include the parent directory "Adam". Otherwise, your backup will be to /Users/Adam/Adam.

Finally, the -v (verbose) flag is only really useful when you are invoking this command manually. You can take this out for your cronjob once it is working. If you want to see the verbose output while the cronjob is working, open up your Console app. It will be output there (Console.app is simply a GUI for displaying the contents of log files that live in /var/log).

You are very close!

[besson3c]

Adam: is this working for you so that you can do your rsync job without entering your password? You'll need it setup this way if you want your cronjob to work too...

[adamfishercox]

Host key verification failed.
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at /SourceCache/rsync/rsync-30/rsync/io.c(359)

What's that mean?

[besson3c]

It means you don't have public key authentication setup. When you enter the rsync command manually, it prompts you for your password, right?

[adamfishercox]

Nope.

[adamfishercox]

Is there any port that could be closed that I might need open? It doesn't prompt for a password, but just tells me that it fails.

[besson3c]

Can you SSH to the destination computer? If so, SSH is working and you meet rsync's requirements... When you do SSH to the destination computer, does it prompt you for your password then?

Does your public SSH key have a password? if so, you will have to point your cronjob at a separate public/private key pair that are not password protected. If you do this, you'll need to make sure that your public key has been entered into the destination computer's authorized keys file.

[adamfishercox]

I can SSH into my MacBook from my iMac, but it asks for a password. I'm thinking that I'll probably run the commands from the iMac, as that is the one I'll be using less, and so the rsync can only happen when it's on anyway.

What are the rsync requirements? Or do I meet them because I could SSH in?

I have no idea about a password. Would I know if my public key had a pass?

[besson3c]

If it asks for your password either your private key is password protected, or else your public key hasn't been installed correctly on the destination computer.

When you do a:

$ ssh <yourmachine>

do you get a:

Adam@<yourmachine>'s password:

prompt?

All rsync needs is the rsync binary to be in your path (which it is by default), and for SSH to be enabled. There are other transport mechanisms other than SSH too, but SSH is the default and your best bet.

[adamfishercox]

No, I get just a regular password prompt. Both computer's passwords are the same, so I can't tell which it's asking for.

Full Terminal Output:

Last login: Wed Feb 6 15:37:58 on ttys000
imac:~ Adam$ ssh MacBook
Password:
Last login: Wed Feb 6 15:37:49 2008 from imac
macbook:~ Adam$

[Hal Itosis]

Originally Posted by adamfishercox 

Ok, I am completely lost... I read some things on cronjobs and they made it worse ...

Can someone help me write a cronjob to have my iMac and MacBook sync hourly? This is as far as I got:

rsync -avE /Users/Adam "Adam@iMac:/Users/Adam" --exclude=projects --exclude=downloads
rsync -avE Adam@iMac:/Users/Adam "/Users/Adam" --exclude=projects --exclude=downloads

Did I use the "Excludes" right?

I'm not overly familiar with rsync, but you might find it easier to use a text file to store a list of things to exclude... and use the "--exclude-from=/path/to/file" option instead.

As I mentioned, ~/Library/Preferences/ByHost is one folder to always exclude.

[adamfishercox]

Thanks a whole bunch to besson3c! I have rsync working.

[melbdean]

i have a similar dream for a syncing app but on a much simpler level/scale. i have an iMac and a MacBook of varying abilities and require that one key folder be synced. that folder is essentially my WIP and contains 8-10 sub-folders that all contain many various files/file formats. psyncx doesn't seem to do what i need. i want that WIP folder to be made identical on each computer after syncing. so if i use my laptop for the day, when i get to the office, i connected them and the app syncs just the nominated WIP folder. ditto if i've been using the iMac for a few days, i'd connect the laptop and the app would sync the folders.
am i dreaming? or is it that i just can't use psyncx properly?

[mduell]

Originally Posted by melbdean 

i have a similar dream for a syncing app but on a much simpler level/scale. i have an iMac and a MacBook of varying abilities and require that one key folder be synced. that folder is essentially my WIP and contains 8-10 sub-folders that all contain many various files/file formats. psyncx doesn't seem to do what i need. i want that WIP folder to be made identical on each computer after syncing. so if i use my laptop for the day, when i get to the office, i connected them and the app syncs just the nominated WIP folder. ditto if i've been using the iMac for a few days, i'd connect the laptop and the app would sync the folders.
am i dreaming? or is it that i just can't use psyncx properly?

FolderShare

Syncs them any time both machines are online.

[TravisReynolds]

.mac

I love .mac, it syncs your computers together, system preferences, mail, safari bookmaerts, calanders, contacts, and you can transfer files between computers so you can have them on the same computer

here is a review Is .Mac Worth it

[Gankdawg]

Originally Posted by TravisReynolds 

.mac

I love .mac, it syncs your computers together, system preferences, mail, safari bookmaerts, calanders, contacts, and you can transfer files between computers so you can have them on the same computer

here is a review Is .Mac Worth it

Originally Posted by adamfishercox

I would be able to select the folders I want to sync.

While .mac is good for what you listed, it doesn't update what adam is looking for.

[melbdean]

Originally Posted by mduell 

FolderShare

Syncs them any time both machines are online.

... but other threads say it is one of those PC conspiracy type things ...?

[mduell]

Originally Posted by melbdean 

... but other threads say it is one of those PC conspiracy type things ...?

Yea yea, everything owned by Microsoft must be evil.
Those concerns are entirely unfounded.

[besson3c]

Originally Posted by mduell 

Yea yea, everything owned by Microsoft must be evil.
Those concerns are entirely unfounded.

What's in it for Microsoft? Either they are trying to get you to buy one of their products by giving this one free, or they are mining your data and selling it to advertisers. Either way, it is important to remember that nothing is free without some sort of catch. Until this information is disclosed, it is entirely realistic to think that MS is mining data. No thanks, not for me...