[Gossamer]

First off, I have a G5 tower at work, a desktop at home, and an iBook (all running 10.4.7), all with seperate iTunes and iPhoto libraries and such, but I'd like to have a way to sync them all so that when I add a song or two or import some pictures on one computer it will eventually appear on all computers. I have a G4 tower at work running Ubuntu 6.06 server edition that's currently a web/dns server, and it's got a spare 40 gig drive already installed. What I'd like to do is set up that linux server as my directory server, but I don't know the first thing to do.
So if you could give me a little detail on setting up both the Linux and the Mac OS sides to work, that would be awesome.
Thanks!

[besson3c]

Originally Posted by Gossamer

First off, I have a G5 tower at work, a desktop at home, and an iBook (all running 10.4.7), all with seperate iTunes and iPhoto libraries and such, but I'd like to have a way to sync them all so that when I add a song or two or import some pictures on one computer it will eventually appear on all computers. I have a G4 tower at work running Ubuntu 6.06 server edition that's currently a web/dns server, and it's got a spare 40 gig drive already installed. What I'd like to do is set up that linux server as my directory server, but I don't know the first thing to do.
So if you could give me a little detail on setting up both the Linux and the Mac OS sides to work, that would be awesome.
Thanks!

Directory server? I'm not sure what you mean? LDAP server?

options:

1) Store all of these files on a centralized server, access them directly off of the server

cons: probably slow from home if your cable's upload speeds are as slow as ours

2) Setup rsync and public/private SSH key pairs to automatically sync your files at scheduled intervals

3) You can use something like mt-daapd to provide a centralized iTunes share on your Linux box, this won't take care of your photo sharing though.


I'd look at this second option, perhaps I'll think of others, but this should be enough to get us started

[Gossamer]

Yeah, LDAP. I think the best thing is to have a 'master' home director that each computer compares their own 'home' directory with, then updates to the latest version of whatever.
All of my computers are on the campus network, so I've got 10+ MB/sec transfers between comptuers. I tranferred the entire XCode tools (823MB) to my server at work from home in under a minute.
I suppose if it's possible I could sync my work computer at the end of the work day and sync my home computer at the end of the full day, then each morning, the work computer could get the newest version, and after work my home computers would too.

[besson3c]

Originally Posted by Gossamer

Yeah, LDAP. I think the best thing is to have a 'master' home director that each computer compares their own 'home' directory with, then updates to the latest version of whatever.
All of my computers are on the campus network, so I've got 10+ MB/sec transfers between comptuers. I tranferred the entire XCode tools (823MB) to my server at work from home in under a minute.
I suppose if it's possible I could sync my work computer at the end of the work day and sync my home computer at the end of the full day, then each morning, the work computer could get the newest version, and after work my home computers would too.

Why not create a login hook that would invoke an rsync command that would sync up your home directory? Rsync is very cool, powerful, and fast. I think it will be your best bet. You can also invoke the sync via a double-clickable icon, or a scheduled cron or anacron job.

I'm still not sure what you envisioned with LDAP, LDAP is used for storing attributes containing to a single object (such as account/contact information for a username). How were you hoping to integrate an LDAP server with this setup?

(You should actually be relieved you won't need LDAP, it can be a bear to setup).

[Gossamer]

Originally Posted by besson3c

Why not create a login hook that would invoke an rsync command that would sync up your home directory? Rsync is very cool, powerful, and fast. I think it will be your best bet. You can also invoke the sync via a double-clickable icon, or a scheduled cron or anacron job.

I'm still not sure what you envisioned with LDAP, LDAP is used for storing attributes containing to a single object (such as account/contact information for a username). How were you hoping to integrate an LDAP server with this setup?

(You should actually be relieved you won't need LDAP, it can be a bear to setup).

Alright, well I really have no clue as to what I need to have set up to do what I want to do, I just read a little about setting up an active directory and thought that would work. Really, I'm looking for the easiest way to have a 'master' home directory that I can sync each individual directory to, to make sure if I import some photos, make a new document, or download something from the iTMS, that it will appear on all three computers. If you've got a way to do that and can explain in some detail, that would be wonderful.
Thanks!
Mikael

[Gossamer]

oops, dp

[besson3c]

Originally Posted by Gossamer

Alright, well I really have no clue as to what I need to have set up to do what I want to do, I just read a little about setting up an active directory and thought that would work. Really, I'm looking for the easiest way to have a 'master' home directory that I can sync each individual directory to, to make sure if I import some photos, make a new document, or download something from the iTMS, that it will appear on all three computers. If you've got a way to do that and can explain in some detail, that would be wonderful.
Thanks!
Mikael

Active Directory is a Microsoft implementation of LDAP. OS X Server uses OpenLDAP, which is probably what you meant, as Active Directory is closed and therefore Windows-only (although you can query it within OS X). At any rate, this isn't what you want.

Which of these machines exist on static IP addresses? Are any behind a firewall or router? We'll first need to determine which machines should push or pull data from the other machines. Having a dynamic IP address would suggest that you would want to push data to a machine (since the other machine cannot count on pulling data from your machine since it's address is not constant).


From your terminal's command line, the basic gist of what you need to do is this:

rsync -av <source directory> <destination directory>

This will syncronize these directories so that the destination directory matches the source directory. If files are identical between these directories, they will be ignored, saving needless bandwidth and time. Changed files are determined based on their size, last modifed date, and other attributes.

For directories that do not reside on your local computer, the syntax will need to match the following. In this example, say you want to sync your entire directory with another computer's home directory:

rsync -av ~/ myusername@myserver:

(the tilda is short form for "my home directory")

The -a flag means archive - it sets a bunch of options you would use for archiving files, and -v is verbose, so that you can see what is going on. Are the usernames the same on each of these machines? If not, there is a flag you'll have to set that will ignore copying over the owners of the files, and thus the destination machine will inherit these files.

There is also a flag for delete, if you want the destination machine to delete any files that do not also exist on the source:

rsync -av --delete ~/ myusername@myserver:


In these examples, the source is pushing out data to the destination machine. You can do the exact opposite from the desintation machine:

rsync -av myusername@myserver: ~/

(the source machine is always the machine on the left side of these arguments)


Experiment with using this command in your terminal now. Once we have the issues of matching username, network status of the machines, etc. figured out, the rest will simply be to setup automation hooks, which should be pretty easy.

What I'd suggest doing for testing is create a dummy folder on a computer, and try to sync the contents of it. So, something like:

rsync -av ~/my_folder myusername@myserver:myfolder

(everything following the colon indicates the path on the server, with no path indicated it will assume the desintation computer's home directory).


SSH needs to be turned on on both computers, as by default rsync uses a secure tunnel provided by SSH to transfer this data (this is a good thing).

PM me if you want my personal contact info.

[Gossamer]

Alright, I'll give this a shot. My work computer and server both have static IPs, my home desktop and laptop both have dynamic IPs. None are behind (local) routers. I can SSH from home to work (or vice versa) easily so I'm guessing that rsync should work. I have the same username on all machines (I read something about the userid, does that matter?). I'll go ahead and give this a shot and see what happens.
Thanks!

[besson3c]

Originally Posted by Gossamer

Alright, I'll give this a shot. My work computer and server both have static IPs, my home desktop and laptop both have dynamic IPs. None are behind (local) routers. I can SSH from home to work (or vice versa) easily so I'm guessing that rsync should work. I have the same username on all machines (I read something about the userid, does that matter?). I'll go ahead and give this a shot and see what happens.
Thanks!

How can your home laptop and desktop be connected at the same time if they are not behind a router or hub?

[Gossamer]

Okay, I created a folder in my personal computer's home directory called 'sync' and then created a folder within my server's home directory also called 'sync.' I put a single file in my home computer and then tried this command:
rsync -av ~/sync mikael@129.xxx.xxx.xxx:/home/mikael

It asked for my server password, which I entered, then it tried to sync each file (the one I put in there plus some '.something' files, but told me that permission was denied on each of them.
Do I need to do something to my user account on the server to give me read/write access over SSH?

[besson3c]

If this is an OS X machine, there is no /home/mikael, but rather a /Users/mikael. However, you can also add a relative path relative to the home directory, i.e.:


rsync -av ~/sync mikael@ip:

This will copy all files from and including your local "sync" folder that don't exist in your destination home directory. If you do a:

rsync -av ~/sync/ mikael@ip:

This will copy the *contents* of your sync folder (but not the folder itsself) to your destination home directory.

[Gossamer]

The home computers are behind a simple 10/100 hub, I've gotten two switches, but one's power was sketchy and the other was DOA so I'm still using the hub.
The server is running Ubuntu 6.06 server edition, I set it up to be my roommate's web server so he didn't have to use his crappy (and LOUD) PII server in our room. It's a G4/400/896/2x40GB.

So there is a /home/mikael/sync folder that I created using mkdir. Here's what I get:

Code:
cats:~ mikael$ rsync -av ~/sync/ mikael@129.xxx.xxx.xxx:/home/mikael/
mikael@129.xxx.xxx.xxx's password: 
building file list ... done
rsync: failed to set times on "/home/mikael/.": Operation not permitted (1)
IP addresses.rtf
rsync: mkstemp "/home/mikael/.IP addresses.rtf.3NzMYi" failed: Permission denied (13)
rsync: failed to set times on "/home/mikael/.": Operation not permitted (1)

sent 566 bytes  received 40 bytes  173.14 bytes/sec
total size is 422  speedup is 0.70
rsync error: some files could not be transferred (code 23) at /SourceCache/rsync/rsync-24/rsync/main.c(717)
cats:~ mikael$

my OS X machine has a folder in my home folder called 'sync' that contains a file called 'IP addresses.rtf'
my server has a folder called 'sync' inside the /home/mikael/ directory

[SoBayJake]

Originally Posted by besson3c

How can your home laptop and desktop be connected at the same time if they are not behind a router or hub?

Most ISPs will give you multiple IPs if you'd like. Uusally for an extra fee of course.

For most users, the one-time cost of a router far outweighs the cost of that extra monthly fee. For others, its nicer to have a "real" routable IP.

[Gossamer]

Originally Posted by SoBayJake

Most ISPs will give you multiple IPs if you'd like. Uusally for an extra fee of course.

For most users, the one-time cost of a router far outweighs the cost of that extra monthly fee. For others, its nicer to have a "real" routable IP.

Well, I'm living on campus at a college, so I've got a direct ethernet connection to the school's network. Their DHCP server gives my computers IP addresses. I do have static hostnames that I can use if I need to, though.

[besson3c]

Originally Posted by Gossamer

The home computers are behind a simple 10/100 hub, I've gotten two switches, but one's power was sketchy and the other was DOA so I'm still using the hub.
The server is running Ubuntu 6.06 server edition, I set it up to be my roommate's web server so he didn't have to use his crappy (and LOUD) PII server in our room. It's a G4/400/896/2x40GB.

So there is a /home/mikael/sync folder that I created using mkdir. Here's what I get:

Code:
cats:~ mikael$ rsync -av ~/sync/ mikael@129.xxx.xxx.xxx:/home/mikael/
mikael@129.xxx.xxx.xxx's password: 
building file list ... done
rsync: failed to set times on "/home/mikael/.": Operation not permitted (1)
IP addresses.rtf
rsync: mkstemp "/home/mikael/.IP addresses.rtf.3NzMYi" failed: Permission denied (13)
rsync: failed to set times on "/home/mikael/.": Operation not permitted (1)

sent 566 bytes  received 40 bytes  173.14 bytes/sec
total size is 422  speedup is 0.70
rsync error: some files could not be transferred (code 23) at /SourceCache/rsync/rsync-24/rsync/main.c(717)
cats:~ mikael$

my OS X machine has a folder in my home folder called 'sync' that contains a file called 'IP addresses.rtf'
my server has a folder called 'sync' inside the /home/mikael/ directory

WHat are the permissions on /home/mikael on the Ubuntu server? Does user mikael have writable access to his home directory?


BTW:

rsync -av ~/sync/ mikael@129.xxx.xxx.xxx:

would also work just the same

[Gossamer]

woo hoo!
I went ahead and set the correct permissions on my server, and I was able to sync the contents of the ~/sync/ folder to my server. I'm going to go ahead and sync my home folder from my work computer.
-So once I do that, what happens when I run the same command from my home computer?
-If there are files on the server that are not on the home computer, what will it do?
-If there are identical files on the home computer, it will use the newest one, right?
-What command should I use to sync from the server to my work/home computers?
-What does it do with identical/missing/new files?

Thanks so much, this is cool!

[Gossamer]

Side note: I'm syncing my home directory from a G4/2x1GHz/1.5GB with 10/100/1000 to a G4/400/896MB with 10/100/1000 directly connected through a Netgear Gigabit switch, and I'm seeing transfer speeds of about 5MB/sec, which is quite slow for gigabit. Looking at Activity Monitor on the 1GHz machine, it looks like rsync+shh is taking about 45% of the processor, but on the server, they're taking 100%. My boss ordered a Mac Pro last week, so I might be getting his Intel iMac, so I can use the G4/2x1GHz as the backup server, making everything much faster. We'll see what happens. Thanks for the help so far!

[besson3c]

Originally Posted by Gossamer

woo hoo!
I went ahead and set the correct permissions on my server, and I was able to sync the contents of the ~/sync/ folder to my server. I'm going to go ahead and sync my home folder from my work computer.
-So once I do that, what happens when I run the same command from my home computer?

The same thing, rsync is platform agnostic.

-If there are files on the server that are not on the home computer, what will it do?

Without the --delete flag, it will leave them as is, basically merging the latest stuff from your home computer to the server (destination machine).

-If there are identical files on the home computer, it will use the newest one, right?

Yes. You can tell rsync how to make this determination, but if you use the -a flag as I've prescribed, it will make this determination based on the file size, modification, and/or any metadata that has changed (file permissions, etc.). If you just wanted to go by the modification times for instance, you could do that too.

-What command should I use to sync from the server to my work/home computers?

Using the same syntax, making appropriate substitions to the source and destination computers.

-What does it do with identical/missing/new files?

Anything that is in the source directory will get sent over to the destination directory providing there isn't a newer file already in the destination directory.

Thanks so much, this is cool!

No problem... sounds like you should do a little more test runs and experimentation so that you are more comfortable with the questions and answers you've asked, but I think you'll find by reading and working through this carefully that both the sync syntax and mechanism is pretty simple.

[besson3c]

Originally Posted by Gossamer

Side note: I'm syncing my home directory from a G4/2x1GHz/1.5GB with 10/100/1000 to a G4/400/896MB with 10/100/1000 directly connected through a Netgear Gigabit switch, and I'm seeing transfer speeds of about 5MB/sec, which is quite slow for gigabit. Looking at Activity Monitor on the 1GHz machine, it looks like rsync+shh is taking about 45% of the processor, but on the server, they're taking 100%. My boss ordered a Mac Pro last week, so I might be getting his Intel iMac, so I can use the G4/2x1GHz as the backup server, making everything much faster. We'll see what happens. Thanks for the help so far!

Encryption is expensive, but highly recommended as soon as your data leaves your network.

As far as the bandwidth goes, rsync is probably copying over a ton of very small files. You won't saturate your bandwidth copying over zillions of small files, Best to test it copying over a large movie file or something.

However, like I said, even copying over a large file will be a little slower and more CPU intensive using an encrypted protocol.

[Gossamer]

Actually it was copying over a series of large .avi files at that time. It looks like the server's processor is the bottleneck right now, but that could be fixed soon with a little luck.

[besson3c]

Originally Posted by Gossamer

Actually it was copying over a series of large .avi files at that time. It looks like the server's processor is the bottleneck right now, but that could be fixed soon with a little luck.

Possibly, but keep in mind that as long as these files don't change, they won't be copied over again. With future syncs rsync will determine that the file hasn't changed, and simply ignore it.

[Gossamer]

Originally Posted by besson3c

Possibly, but keep in mind that as long as these files don't change, they won't be copied over again. With future syncs rsync will determine that the file hasn't changed, and simply ignore it.

That is amazing. I'm so happy right now

[besson3c]

Originally Posted by Gossamer

That is amazing. I'm so happy right now

Let me know when you'd like to learn about scheduling and having these syncs occur automagically

[Gossamer]

So what happens if I download a song from the iTMS on one computer? Will it update my itunes library on the rest of my computers?
Also, after I sync back from the server, it creates mail folders in Mail.app that exist on the other machines, but the messages that should be in those folders (stuff I've dragged from my inbox to the folder) are not there. Any ideas?

[besson3c]

Originally Posted by Gossamer

So what happens if I download a song from the iTMS on one computer? Will it update my itunes library on the rest of my computers?
Also, after I sync back from the server, it creates mail folders in Mail.app that exist on the other machines, but the messages that should be in those folders (stuff I've dragged from my inbox to the folder) are not there. Any ideas?

Your iTunes Library and individual song files are all saved in ~/Music, so they will be included in the backup of your home directory.

Local mail folders will probably need to be reindexed each sync to catch these new messages. Since you are working from multiple machines, I'd highly recommend not using local folders, and storing your folders on your mail server.