[amagab]

We have an Apple server running Snow Leopard that needs to be replicated on a nightly basis to an identical server in another building. This is for backup and security.

What would be the best way of doing this?

Thanks!

[besson3c]

No redundancy or load balancing? This server will never be accessed directly, just used for offsite backup? What do you mean by security?

I would focus on backup up your actual user data, not imaging the entire computer. Could you clarify what this server will be used for though?

[amagab]

It will be used for file share and database and will be accessed directly if the loose access to the primary server. Users will switch to the off site server should something serious happen to the primary server. Minimal downtime expected. The servers are in separate buildings but on the same network.

[besson3c]

Would you be opposed to putting your data on a common share accessed/mounted by both machines served via one of AFP/NFS/SSHfs? You could then easily replicate this data to another set of disks via an rsync cronjob. Since these two machines will be in different buildings I would probably recommend SSHfs so that data sent to and from this share is encrypted.

Otherwise, if you want two machines with two direct attach drives to be in sync with each other, you will have to look at two way database replication and an rsync cronjob to sync between both file shares at a regular interval. This will be more complicated. I would recommend separating the data from the services so that the data is shared (i.e. the former recommendation of a network share)

[besson3c]

Also, if you separate the data that way essentially the front-end servers will be completely interchangeable. Under heavy loads (if you have any of those) you could do DNS round-robin between both front-end servers to balance your loads, and take one of the machines out of the pool as desired.

[amagab]

Thanks for your response. I like your first idea. I'll take a look at using that.

Btw, is it possible to run Time Machine from one server to another, over the network

[amagab]

Sorry, I'm not usually a server admin and am not too comfortable with the command line. Is there a good GUI or software for your first suggestion? Thanks!

[besson3c]

Originally Posted by amagab 

Thanks for your response. I like your first idea. I'll take a look at using that.

Btw, is it possible to run Time Machine from one server to another, over the network

I think so, but I definitely wouldn't use Time Machine to backup a server. I don't think Apple recommends it even for just a locally attached disk, it is definitely not a product suitable for a server. For starters, network services store data in places that TM does not backup, AFAIK...

[besson3c]

Originally Posted by amagab 

Sorry, I'm not usually a server admin and am not too comfortable with the command line. Is there a good GUI or software for your first suggestion? Thanks!

What database are you using? Do you know where it stores its data?

If you aren't a server admin this is going to be very tough. This is not a point-and-click sort of job...

[amagab]

We are using MySQL. I just realized we currently have Cronnix installed and use it for database backup.

I've done some server admin and learn as I go along. I've maintained servers for some time but have never got into the gritty configs and scripts.

[besson3c]

Originally Posted by amagab 

We are using MySQL. I just realized we currently have Cronnix installed and use it for database backup.

Ahhh.. well, if Cronnix is just your GUI for creating file-based cronjob backups, that's a *very* dangerous way of backing up database data, I would not do this. By simply transferring the files you are not putting locks on the data, meaning any writes that are going on when you transfer the data could corrupt this data. This is why MySQL provides tools such as mysqldump which properly handle the locks. I would check to see what cronjobs are being run, what they are doing, and if you aren't using mysqldump to backup your data, start.

Additionally, MySQL does support one-way data replication to another server, but it would be a PITA to cut back to your production database server when you want to switch back to it, it's really designed for one-way replication. This would further amplify my suggestion for working towards sharing data between machines and backing up this data.

I've done some server admin and learn as I go along. I've maintained servers for some time but have never got into the gritty configs and scripts.

Even if there was some sort of GUI to help, for this I would definitely not recommend using it. You really need to understand what is going on here, or else you'd be asking for trouble.

[amagab]

I would have to look exactly how the Cronnix and MySQL backup is setup.

I've done some command line mysql backups in the past. From what you are saying it seems like I need to take a crash course in knowing the command line better. Thanks!

Can you elaborate some more on the sharing part?

[besson3c]

Sure... the model is this:

- Mac #1
- Mac #2
- third machine (preferably Unix based) that hosts the data
- fourth machine/disk (backup for the data on the third machine)

Mac #1 and Mac #2 will mount the data hosted by this third machine to a directory on these machines, and the applications being served from these Macs will access the data from this network share. You would use symbolic links to direct the MySQL datadir to this network mount.

The result is that Mac #1 and Mac #2 would be completely interchangeable since they wouldn't actually host any data of their own. You could add additional nodes as desired, too. The only caveat is that as far as your MySQL data goes you'll want to make sure that only one MySQL server is running at a time so that you don't run into problems where multiple machines are trying to write to the same file at the same time. This will not be a problem with your straight up file sharing though, just MySQL data. If it were me I'd actually bring up a dedicated MySQL server and a backup for that, and if this database was serving something like your public website that warrants having consistent performance I'd put it on a separate machine so that it doesn't compete for resources with your general file sharing.

You're probably noticing that several new machines are being brought into the picture now, which is true. However, they don't have to be physical machines (and if it were me, they wouldn't be). Virtual machines would work great for stuff like MySQL providing you have some decent underlying hardware that would support the loads. Since you are talking about an offsite machine I would strongly recommend virtualizing everything anyway, it is much easier to simply ship a VM image off to another machine than it is to keep two separate physical machines up-to-date, and virtualizing is obviously a far cheaper and more effective way to go about an environment like this anyway. None of these machines need to be Macs either, in case you'd have some limitations. These VM guests would work best with non-OS X operating systems though, OS X server consumes a ton of resources.

I realize that this is probably overwhelming, and I apologize for that, but this is all stuff worth giving serious thought to. Does the basic sharing model make sense now? It is good of you to think about this sort of stuff, too few admins do. You never want to be completely tethered to a single server with no disaster or recovery plan in place.

[Drizzt]

For MySQL : Dual master replication setup is easy to implement, stable and handy. You servers will keep in sync, if one goes down, when it'll get back up, it will catch up with the other.

For files, if you can tolerate being 24h late, I'd say rsync is the best bet here.

[besson3c]

Originally Posted by Drizzt 

For MySQL : Dual master replication setup is easy to implement, stable and handy. You servers will keep in sync, if one goes down, when it'll get back up, it will catch up with the other.

For files, if you can tolerate being 24h late, I'd say rsync is the best bet here.

The problem with the MySQL replication is exactly what you say though - it's dual master. It is a PITA to make one server become slave and then reverse this replication the other direction for when you want to change which server is the primary production server and which is backup. Also, I've had problems where the replay would generate an error and stop for no obvious reason. I've been less than impressed with MySQL replication, although it is indeed easy to setup. Maybe it has improved in recent versions.

[Drizzt]

Originally Posted by besson3c 

The problem with the MySQL replication is exactly what you say though - it's dual master. It is a PITA to make one server become slave and then reverse this replication the other direction for when you want to change which server is the primary production server and which is backup. Also, I've had problems where the replay would generate an error and stop for no obvious reason. I've been less than impressed with MySQL replication, although it is indeed easy to setup. Maybe it has improved in recent versions.

Thing is, you don't need to switch to slave!
Both are master with the other slave. Just make a search, it's documented in the MySQL documentation on how to do it.
Say you have Server A et B
Server A gets update, Server B pulls it from Server A
Server B gets update, Server A pulls it from Server B

The server ids need to be different, slave-log-updates activated too.

[besson3c]

Originally Posted by Drizzt 

Thing is, you don't need to switch to slave!
Both are master with the other slave. Just make a search, it's documented in the MySQL documentation on how to do it.
Say you have Server A et B
Server A gets update, Server B pulls it from Server A
Server B gets update, Server A pulls it from Server B

The server ids need to be different, slave-log-updates activated too.

Dual master and slave configurations?

It's been a while for me, but I remember having great difficulty trying to avoid situations where the replica would try to replay data it already had due to temporarily making the replica the master while the original master had a bad drive and was offline and would crap out. I used to have to reset the replication with a fresh dump file fairly often (and it would error sometimes for no apparent reason, like I said).

It has just proven to be a little too inflexible for me, you sort of have to hold its hand. It would be great if the slave could scan through the entire logs and replay data it is missing - sort of like an incremental backup. I don't mind repopulating the database in an event like this with a fresh dump file, but having to constantly figure out the replay offsets and all of that can be a PITA.

To the original poster, if you want to play around with rsync + MySQL replication, go for it, but I'd definitely practice cutting both ways so that you can practice making both servers primary. This is going to be a multi-step process.

[Drizzt]

it was exactly what you were looking for....

[besson3c]

Maybe I ought to give the whole replication thing another try, it has been several years since I last did so... I was probably on either MySQL 3.x or 4.x at the time.