[staypuffinpc]

hey all,
I just got switched to a new computer at work (Windows PC to a new Intel Mac!). I hardly used my PC for personal stuff before b/c I have an ibook that I do a lot of my work/notetaking/calendaring on. Now that I have an awesome Mac machine, though, I'd like to be able to somehow sync the two so that I can work on my Mac and work and iBook elsewhere. Is there a way to sync certain parts of the two machines without having a .mac account? I'd mostly like to:

1. sync my documents folder
2. sync my calendars/GTD
3. sync my iPhoto.

Does anyone have any good recommendations of how to do this? I'm willing to buy a small 3rd party app if there's a good one out there for doing this sort of thing. Also, it needn't be an online solution, as I have a firewire connection that I'd prefer to use to sync the machines (the faster the better, IMO). Thanks in advance.


-Peter-

[staypuffinpc]

OK, as I was thinking about this, I've thought that perhaps I could create an Applescript to check for the modificaition dates between certain files on the two computers (my iBook would be connected as an external drive). The big problem is that I have never used Applescript before. So, before I venture into the land of the unknown, I'd like to ask all the pros out there: 1, if it would be possible, and 2. Where a good (Free) source would be to learn the Applescript to do it. In simple terms, this is what I would need to do:

1. Determine the files and folders I want to sync
2. Check the folders on each drive to see if anything has been added that didn't exist before
3. Loop through each of the files in the folders and check the modification dates. If the same file exists in both places, replace the older one with the newer one.

I think that's basically it. I'm open to suggestions. Thanks in advance for the expertise and recommendations.

[Macola]

I've been pretty happy with ChronoSync. if you know which folders to sync, that should be all you need.

[besson3c]

Originally Posted by staypuffinpc

OK, as I was thinking about this, I've thought that perhaps I could create an Applescript to check for the modificaition dates between certain files on the two computers (my iBook would be connected as an external drive). The big problem is that I have never used Applescript before. So, before I venture into the land of the unknown, I'd like to ask all the pros out there: 1, if it would be possible, and 2. Where a good (Free) source would be to learn the Applescript to do it. In simple terms, this is what I would need to do:

1. Determine the files and folders I want to sync
2. Check the folders on each drive to see if anything has been added that didn't exist before
3. Loop through each of the files in the folders and check the modification dates. If the same file exists in both places, replace the older one with the newer one.

I think that's basically it. I'm open to suggestions. Thanks in advance for the expertise and recommendations.

Why reinvent the wheel? This has already been done. Check out rsync, it has been in use for ages in the Unix world.

[besson3c]

Rsync + public key SSH authentication is your friend. The rsync algorithm is killer - fast, flexible, efficient. We've used it at work to sync terabytes of data, it is really an invaluable tool.

[staypuffinpc]

Folks,
Thanks so much. This is exactly what I was talking about. Both of these look like great tools. One with a GUI, one without. Now I just have to figure out if I'm willing to mess around in the terminal or if I want to be lazy about it and just click buttons for $30.

[harrisjamieh]

Also, look at a service called 'Foldershare'. Just google it. Its an over-the-net service that will auto sync two or more machines.

It only works for folders/documents, so it will not sync your calendars, but I find it really useful, and most of al, its free. Its actually a service that has recently been bought by Microsoft, and is now part of the Windows Live! service, but it still has a Mac client (although for PPC, though it runs under Rosetta).

Another cool feature is that, if one machine is on, and connected to the net, you can be anywhere in the world and pick up your files through a web-based service. Very useful.

[besson3c]

Originally Posted by staypuffinpc

Folks,
Thanks so much. This is exactly what I was talking about. Both of these look like great tools. One with a GUI, one without. Now I just have to figure out if I'm willing to mess around in the terminal or if I want to be lazy about it and just click buttons for $30.

Just cut to the chase and use the tool that the GUI may actually simply be a front end to, I promise you it isn't too hard to grasp using rsync. I'd be happy to help you understand its syntax. Really, under most conditions the syntax is as simple as:


rsync -av <source directory> <username>@<destination computer ip>:<destination directory>

To rsync your home directory:

rsync -av ~/ username@mycomputer:

or:

rsync -av /Users/myusername username@mycomputer:


If the usernames on the computers are different, I'd suggest altering this syntax... let me know if this is the case.

[Macola]

The advantage of ChronoSync (besides the GUI) is that you can schedule syncs, perform backups, and a few more functions that may or may not be important to you. Yes, all these can be done with a combination of basic commands in Terminal, but try it out before you decide.

[besson3c]

No offense, but unless you have the money to spend on a piece of shareware like this, it really isn't a good way to spend your cash, IMHO. It really isn't that hard to setup schedules of rsync commands via a variety of mechanisms already included in the OS. I can't see anything listed on the webpage that can't be done using rsync. I'm willing to bet that this thing is simply a GUI to rsync anyway, so why is it necessary to pay $30 for an app that would run in a non-interactive fashion anyway?

[Macola]

It's entirely a matter of personal preference (no offense taken). Although I'm fairly adept with a CLI (started out on a MicroVax using VMS), I like some of the features that ChronoSync offers, such as archival of deleted files.

[hemant]

try "arRsync" its a GUI for rsync and works great.

[besson3c]

Originally Posted by hemant

try "arRsync" its a GUI for rsync and works great.

And open source, which is a big plus... I'm just not personally a big fan of paying for something that I already have and which works quite well, only because it is simply assumed that taking advantage of what one already has would be too intimidating or too time-consuming. We are all smart people, we could learn rsync if we chose to do so.

That's just me though, I realize I don't speak for everybody, but I do hope that people are at least aware of these free options.

[staypuffinpc]

Originally Posted by hemant

try "arRsync" its a GUI for rsync and works great.

Thanks for this. Although it's nice to get down to brass tacks and know what's going on behind the scenes, sometimes it's nice to not have to deal with "did I spell that right?" I just tried this little GUI out and it worked beautifully.

So, now I have a very stupid question--Where is my calendar file located? I read that it would be in /~usr/library/calendars, but when I go to that file there is obviously nothing there (is is hidden?). In addition to my documents, there are a few preferences and management files that I would really like to sync (e.g. Firefox bookmarks, etc.). Thanks, again, for entertaining a pretty silly question. (all the feedback has been GREAT!).

-Peter-

[besson3c]

Originally Posted by staypuffinpc

Thanks for this. Although it's nice to get down to brass tacks and know what's going on behind the scenes, sometimes it's nice to not have to deal with "did I spell that right?" I just tried this little GUI out and it worked beautifully.

So, now I have a very stupid question--Where is my calendar file located? I read that it would be in /~usr/library/calendars, but when I go to that file there is obviously nothing there (is is hidden?). In addition to my documents, there are a few preferences and management files that I would really like to sync (e.g. Firefox bookmarks, etc.). Thanks, again, for entertaining a pretty silly question. (all the feedback has been GREAT!).

-Peter-

~/Library/Calendars

All other application data is located in your home directory, most typically in ~/Library.

[staypuffinpc]

Originally Posted by besson3c

~/Library/Calendars

All other application data is located in your home directory, most typically in ~/Library.

Unfortunately, I checked that and it doesn't exist. That is, in my Library folder there is no "Calendars" folder.

[Macola]

Originally Posted by staypuffinpc

Unfortunately, I checked that and it doesn't exist. That is, in my Library folder there is no "Calendars" folder.

Try ~/Library/Application Support/iCal/Sources. There should be subfolders for each iCal calendar, and the data are in the .ics files.

[thuldai]

Another nice Syncing application is Synk: Synk

I prefer and constantly use rsync though. Once you dug your way through the man pages and set up a couple of shell scripts, it's the best.

[Spliff]

Originally Posted by besson3c

Rsync + public key SSH authentication is your friend. The rsync algorithm is killer - fast, flexible, efficient. We've used it at work to sync terabytes of data, it is really an invaluable tool.

I thought Rsync doesn't properly support resource forks.

[besson3c]

Originally Posted by Spliff

I thought Rsync doesn't properly support resource forks.

That was the case at one point, but I think Apple's bundled rsync as of a particular version added support for resource forks.

Makes no difference to me though, I have absolutely no OS 9 stuff on my drive.

[Spliff]

Originally Posted by besson3c

That was the case at one point, but I think Apple's bundled rsync as of a particular version added support for resource forks.

Makes no difference to me though, I have absolutely no OS 9 stuff on my drive.

I thought carbon apps still add resource forks to files, like preview icons (thumbnails) for jpgs and pdfs.

[besson3c]

Originally Posted by Spliff

I thought carbon apps still add resource forks to files, like preview icons (thumbnails) for jpgs and pdfs.

I don't think so.... anybody?

[Spliff]

I found this. I think I'm more confused now. But I think that if you're backing up to another HFS+ volume, you're okay. I think the resource fork issue is only a problem when backing up to non-OS X volumes.

[besson3c]

Originally Posted by Spliff

I found this. I think I'm more confused now. But I think that if you're backing up to another HFS+ volume, you're okay. I think the resource fork issue is only a problem when backing up to non-OS X volumes.

The latest version of rsync is 2.6.8, and updates don't occur very frequently. This page was last updated Dec. 2005. I believe this information and software is now deprecated.

Apple has pushed a move away from using resource forks since OS X's inception. They are pretty much dead now, and irrelevant unless you care about preserving OS 9 metadata.

That is my understanding of them.

[Spliff]

Originally Posted by besson3c

The latest version of rsync is 2.6.8, and updates don't occur very frequently. This page was last updated Dec. 2005. I believe this information and software is now deprecated.

Apple has pushed a move away from using resource forks since OS X's inception. They are pretty much dead now, and irrelevant unless you care about preserving OS 9 metadata.

That is my understanding of them.

If you're new to the Mac or have an Intel Mac, I don't think the Rsync issues will be a problem. But I still have Classic on my system, as well as old documents. As a result, I'm avoiding rsync for the time being. I really should get around to converting the documents to plain-text to avoid legacy issues.

I found a long, bitchy thread on Apple's Mailing Lists about several serious and inconvenient bugs in Apple's implementation of Rsync in Tiger. There's a good summary here:

Problems
Many files on HFS+, the Mac's most common file system, have metadata. This is partly a leftover from the past (resource forks), and partly a new development (ACLs, extended attributes). Plain rsync doesn't (yet) cope with this metadata.

Since OS X 10.4 (aka Tiger) the MacOS ships with a modified version of rsync. An added option, -E, enables the transfer of extended attributes. This is done by encapsulating the resource fork, Finder data et al in a synthetic file which is added to the rsync transfer list. The name of this file is formed by prepending ._ to the name of the original file, a technique which is also used when copying data from HFS+ partitions to non-Apple file systems such as NFS mounts. It may not be pretty or foolproof (what happens when both foo and ._foo exist?), but at least it's documented by Apple and not likely to change in the very near future. This rsync derivative is based on rsync-2.6.3.

Four problems with Apple's rsync. In order of severity, worst first:

1. The rsync sender will frequently crash with a Bus Error / Segmentation Fault after generating the file list, but before transferring any files. This turns out to be caused by a buffer overrun.
2. When used with the --delete option, the rsync receiver will try to unlink the (fake) synthetic files, flooding the syslog with failure reports, possibly filling the entire boot disk.
3. When files with extended attributes are transferred, the modification time will be set to the time of the transfer, even when the user has specified that modification times be preserved. As a result, using mtime to determine whether a file has changed is broken.
4. Extended attributes have no modification time of themselves. Since a file's mtime is not updated when its attributes are changed, only checksumming can be used to determine whether attribute data needs to be transferred. With default settings, this means that ALL extended attributes are ALWAYS copied.
5. Update 20051215: I have received a report that, even with this patch, ACLs are not backed up properly. I don't use ACLs, and I haven't verified this claim.

[staypuffinpc]

Originally Posted by Macola

Try ~/Library/Application Support/iCal/Sources. There should be subfolders for each iCal calendar, and the data are in the .ics files.

Aha! There they are. It worked like a charm. Thanks a million (which is about how many calendars I have )

[besson3c]

Interesting. I'm actually using a copy of rsync built by MacPorts (who used to be DarwinPorts). This version is a completely virgin stock rsync. It works extremely well on all sorts of different Unix systems, and is used extensively in everything ranging from small business/simple server installs to high performance computing.

I personally don't care about resource forks or ACLs, so this isn't a concern for me. I hope Apple stops pushing new formats and proprietary stuff, the world doesn't need more of that. I'm happy that Spotlight is based on some sort of database/flat file rather than throwing another complication into the mix of differing file formats and protocols. No other platform has had to do so many bizarre things with their files and formats, and Apple certainly isn't ni a position to have this sort of leverage before this act grows pretty old.

In my opinion, what the ideal should be is to be able to use a tool like rsync to transfer stuff to and from the Mac without special Mac-only things being destroyed, or the recipient confused with strange, unexpected stuff such as hidden files that are automatically generated.

[Spliff]

Originally Posted by besson3c

Interesting. I'm actually using a copy of rsync built by [url=http://macports.org]I hope Apple stops pushing new formats and proprietary stuff, the world doesn't need more of that. I'm happy that Spotlight is based on some sort of database/flat file rather than throwing another complication into the mix of differing file formats and protocols. No other platform has had to do so many bizarre things with their files and formats, and Apple certainly isn't ni a position to have this sort of leverage before this act grows pretty old.

In my opinion, what the ideal should be is to be able to use a tool like rsync to transfer stuff to and from the Mac without special Mac-only things being destroyed, or the recipient confused with strange, unexpected stuff such as hidden files that are automatically generated.

[Spliff]

Originally Posted by besson3c

The latest version of rsync is 2.6.8, and updates don't occur very frequently. This page was last updated Dec. 2005. I believe this information and software is now deprecated.

I just typed "rsync --version" in the Terminal and it says I have v. 2.6.3 installed. WTF?

Also, I ran a test backup, it Rsync didn't preserve any of my custom folder icons and it changed a bunch of documents into generic Unix executable files. That can't be right.

[besson3c]

Originally Posted by Spliff

I just typed "rsync --version" in the Terminal and it says I have v. 2.6.3 installed. WTF?

Also, I ran a test backup, it Rsync didn't preserve any of my custom folder icons and it changed a bunch of documents into generic Unix executable files. That can't be right.

The versions of Unix tools Apple includes in OS X are absolutely ancient, some of them well over a year old. I have no idea why neglects these tools, it really doesn't take much effort to roll in something like the upgrade to rsync. It's not like they have to develop anything or even do any testing of these tools

Tell me details about your rsync backup attempt. What file system did you write to? What flags did you include in your backup?

[besson3c]

BTW, if you want a newer version of rsync, check out Macports... it will provide you with a fully automated install process of rsync, or thousands of other Unix tools.

[Spliff]

Originally Posted by besson3c

The versions of Unix tools Apple includes in OS X are absolutely ancient, some of them well over a year old. I have no idea why neglects these tools, it really doesn't take much effort to roll in something like the upgrade to rsync. It's not like they have to develop anything or even do any testing of these tools

Tell me details about your rsync backup attempt. What file system did you write to? What flags did you include in your backup?

I just tried a simple backup from my boot drive (10.4.7) to an external firewire drive (formatted HFS+). In the Terminal I entered:

iMac-G5:~ user$ rsync -av -E /Documents/\ MANUALS/ /Volumes/Server\ HD/Test\ Backup/

The files all copied, but I lost all my custom folder icons and any file without a filename suffix showed up as a generic Unix executable file. I also got the same result using the arRsync GUI listed earlier in this thread. Losing my custom folder icons is unacceptable to me. In comparison, Psync retains my custom folder icons and it correctly identifies file that lack filename suffixes.

Let me know if I'm doing something wrong.

[Spliff]

Originally Posted by besson3c

BTW, if you want a newer version of rsync, check out Macports... it will provide you with a fully automated install process of rsync, or thousands of other Unix tools.

I tried MacPorts (DarwinPorts) a few weeks ago. I wanted it mainly for the video tools. Most of them wouldn't install without errors: files missing or unavailable or GUIs not working for Macs, but which work for Linux or Windows. I posted a request for help on the mailing list but got no response. So I gave up and uninstalled it.

[besson3c]

Originally Posted by Spliff

I tried MacPorts (DarwinPorts) a few weeks ago. I wanted it mainly for the video tools. Most of them wouldn't install without errors: files missing or unavailable or GUIs not working for Macs, but which work for Linux or Windows. I posted a request for help on the mailing list but got no response. So I gave up and uninstalled it.

Which tools? I"m actually one of the port maintainers, so maybe I can help.

MacPorts is experiencing some growing pains and transitions. There has been some talk of automating port installs so we can be more preemptive in catching compiler spews and other problems. However, if you file a bug, it will eventually be addressed.

[besson3c]

Originally Posted by Spliff

I just tried a simple backup from my boot drive (10.4.7) to an external firewire drive (formatted HFS+). In the Terminal I entered:

iMac-G5:~ user$ rsync -av -E /Documents/\ MANUALS/ /Volumes/Server\ HD/Test\ Backup/

The files all copied, but I lost all my custom folder icons and any file without a filename suffix showed up as a generic Unix executable file. I also got the same result using the arRsync GUI listed earlier in this thread. Losing my custom folder icons is unacceptable to me. In comparison, Psync retains my custom folder icons and it correctly identifies file that lack filename suffixes.

Let me know if I'm doing something wrong.

Well, the -E is redundant since your -a will perserve your permissions, but other than that it looks fine.

I'm honestly not sure how OS X maintains custom folder icons or file assocations, so I don't know whether Apple's rsync implementation or anything else will deal with this. I'm more concerned with the integrity of my data than anything else.

I actually use psync to backup my Mac data, but it too often seems to mess up my file associations (being transferred to a non-Mac OS). I don't use custom icons, so I have no experience to share there, but essentially I've given up on massaging the needs of OS X in this department, it just seems like a futitle effort - easier to recreate my file associations if I'm ever in a bind where I need to pull from my backup.

[Spliff]

Originally Posted by besson3c

Which tools? I"m actually one of the port maintainers, so maybe I can help.

MacPorts is experiencing some growing pains and transitions. There has been some talk of automating port installs so we can be more preemptive in catching compiler spews and other problems. However, if you file a bug, it will eventually be addressed.

I can't remember all of them (maybe I reinstall MacPorts and try again). However, the one that comes to mind is mkvtoolnix. It installs fine and works great in the Terminal, but there is a GUI interface for mkvextract (I think) called "mmg." It also installs fine. When I run it, I get the GUI but none of the buttons are responsive; they don't respond to clicks, so you can't actually do anything with mmg.

I remembered a couple more: Avidemux and mpgtx wouldn't fully install, either.

I find it frustrating that support for Unix/Linux apps on Macs seems far behind that of Windows and Linux. I guess it's because there are so many more programmers that work in Windows and Linux, so finding someone to maintain and update various programs isn't so difficult.

[besson3c]

Originally Posted by Spliff

I can't remember all of them (maybe I reinstall MacPorts and try again). However, the one that comes to mind is mkvtoolnix. It installs fine and works great in the Terminal, but there is a GUI interface for mkvextract (I think) called "mmg." It also installs fine. When I run it, I get the GUI but none of the buttons are responsive; they don't respond to clicks, so you can't actually do anything with mmg.

Does it have a config file? Unfortunately, all MacPorts can be responsible for is building and installing apps correctly. If this particular problem is a bug in the app or a misconfiguration, MacPorts won't even attempt to help.

[Spliff]

Originally Posted by besson3c

Does it have a config file? Unfortunately, all MacPorts can be responsible for is building and installing apps correctly. If this particular problem is a bug in the app or a misconfiguration, MacPorts won't even attempt to help.

Ah, I didn't realize that. So I would have to go I dunno about the config file since it's no longer installed, but if I reinstalled it, I'll look for it.

I came across this:

MacOS X 10.3 (and probably 10.4)

Shawn Holwegner is no longer maintaining his OpenSource packages. Therefore 1.6.5 might be the last release for Mac OS X.

[alphasubzero949]

I'm surprised that there hasn't been a mention of this, but for those who don't want to leave the comfort of the .Mac sync preference setup:

MySync

[besson3c]

Originally Posted by Spliff

Ah, I didn't realize that. So I would have to go I dunno about the config file since it's no longer installed, but if I reinstalled it, I'll look for it.

I came across this:

MacOS X 10.3 (and probably 10.4)

Shawn Holwegner is no longer maintaining his OpenSource packages. Therefore 1.6.5 might be the last release for Mac OS X.

Well, FYI, the version available in Macports right now is 1.7.0. Don't know if it compiles/installs cleanly, but you could always try it and notify the list if it doesn't - sometimes there are quick fixes depending on what the nature of the problem is.

[staypuffinpc]

Folks,
you've all been a great help. I've been using arRsync and it works beautifully (and amazingly fast!). What I am doing is a bidirectional sync, which synchronises the same file on the two computers. The problem I have is that I will often create new documents on one computer that I would then like to sync to the other computer. I work on both computers about the same (iMac at work, iBook at home) and may create new files in the same directory on both computers between syncs. So, what I would like is not only to sync existing files, but also to transfer new files (and possibly delete ones that were deleted on one computer). In order to do this, I would really need a preview of the changes about to occur before the changes took place. Is there a way to do this with rsync? Does Chronosync do this? (their product overview page shows a picture of this preview happening, but I don't know if it allows transfer of completely new files). If it were just one or two new files in one directory each time, I could just copy them myself, but I often create several new files in different directories. Keeping track of these changes can be something of a pain.

As always, your feedback is very much appreciated.

[besson3c]

Originally Posted by staypuffinpc

Folks,
you've all been a great help. I've been using arRsync and it works beautifully (and amazingly fast!). What I am doing is a bidirectional sync, which synchronises the same file on the two computers. The problem I have is that I will often create new documents on one computer that I would then like to sync to the other computer. I work on both computers about the same (iMac at work, iBook at home) and may create new files in the same directory on both computers between syncs. So, what I would like is not only to sync existing files, but also to transfer new files (and possibly delete ones that were deleted on one computer). In order to do this, I would really need a preview of the changes about to occur before the changes took place. Is there a way to do this with rsync? Does Chronosync do this? (their product overview page shows a picture of this preview happening, but I don't know if it allows transfer of completely new files). If it were just one or two new files in one directory each time, I could just copy them myself, but I often create several new files in different directories. Keeping track of these changes can be something of a pain.

As always, your feedback is very much appreciated.

Rsync can make the determination on what the newest files are based on their timestamps, metadata (permissions, etc.), and/or file size.

Examples of bidirectional sync circumstances:

- If the file in the destination directory is newer than the file in the source directory based on this criteria, no copying will take place, the destination file will be left intact.

- If the file in the source directory is newer, it will replace the file in the destination directory (unless you direct rsync to not do so).

- If the file simply doesn't exist in the destination directory but does in the source directory, it will be created in the destination directory.

What I would do is learn about all of the rsync flags/options, and set your syncing on a specific schedule. I've never used arSync so I don't know what rsync features it supports. If you run into a wall with arSync, you may have to cut to the chase and look at rsync. Personally, if there is a margin of error, I would just cut to the chase and work with rsync.