[Partisan01]

Originally posted by osxisfun:
yah but for selfish old me ethernet should be plenty to send over parts of my source code to distCC...

Just curious, what type of compilations are you doing that you require a second computer to offload the work to? I understand with larger projects it's not uncommon to do only do a full compile once a week, and do incremental compilers, or library compiles more often.

In my development experience when working with larger projects items will be broken out into libraries, and a driver is written to test the functionality of the library. Eventually everything is merged together.

This isn't to take a hit at you, I'm just interested to see what type of stuff you're working on that requires such strength, and it begs the question, why not purchase an XServe, or a blade center?

[osxisfun]

its mostly for the complete rebuild which takes about 15-20 minutes. you are right my incremental builds do not take much time but when i get into sections that i touch header files the build times take 15-20. if i can spend a thousand bucks and get that down to 6 minutes (i also have a dual g4 laying around) it would be worth it to me.

i just hate waiting those 15 minutes and love the concept of a stack o' minis kicking in when i need them.

edit: i should also say, i am on a dual 2 g5 and like the mini cluster idea because i can buy into a cluster in 500$ at a time, whereas a $4000 dual xserve is out of my budget (at least until this summer)

[Xaositect]

More info (repost of top lines for easy comparison):

Xserve Dual 2GHz, 4GB RAM: at 10 SPM, avg 2 seconds a search. Goes over 30sec/search at 3247 SPM.
iMac G5 1.6GHz, 1.5GB RAM: at 10 SPM, avg 5 seconds a search. Goes over 30sec/search at 967 SPM.
Mac Mini 1.42GHz, 512MB RAM: at 10 SPM, avg 6 seconds a search. Goes over 30sec/search at 524 SPM.
Mac Mini 1.42Ghz, 1GB RAM: at 10 SPM, avg 5 seconds a search. Goes over 30sec/search at 601 SPM.
Mac Mini 1.25GHz, 256MB RAM: at 10 SPM, avg 9 seconds. See below for more.
Mac Mini 1.25GHz, 512MB RAM: at 10 SPM, avg 6 seconds. Goes over 30sec/search at 492 SPM.

As noted above, one 1.25 mini came with a Fujitsu drive. When I openned the other, it had a Toshiba similar to the 80GB drives in the 1.42s. The Fujitsu drive failed during testing!

Went to local Apple store, they tested it and just gave me a new one as I didn't have AppleCare on it. The new one has a Toshiba drive. Loading server on it now, will setup cluster and run it overnight with various tests.

[typoon]

Originally posted by Xaositect:
More info (repost of top lines for easy comparison):

Xserve Dual 2GHz, 4GB RAM: at 10 SPM, avg 2 seconds a search. Goes over 30sec/search at 3247 SPM.
iMac G5 1.6GHz, 1.5GB RAM: at 10 SPM, avg 5 seconds a search. Goes over 30sec/search at 967 SPM.
Mac Mini 1.42GHz, 512MB RAM: at 10 SPM, avg 6 seconds a search. Goes over 30sec/search at 524 SPM.
Mac Mini 1.42Ghz, 1GB RAM: at 10 SPM, avg 5 seconds a search. Goes over 30sec/search at 601 SPM.
Mac Mini 1.25GHz, 256MB RAM: at 10 SPM, avg 9 seconds. See below for more.
Mac Mini 1.25GHz, 512MB RAM: at 10 SPM, avg 6 seconds. Goes over 30sec/search at 492 SPM.

As noted above, one 1.25 mini came with a Fujitsu drive. When I openned the other, it had a Toshiba similar to the 80GB drives in the 1.42s. The Fujitsu drive failed during testing!

Went to local Apple store, they tested it and just gave me a new one as I didn't have AppleCare on it. The new one has a Toshiba drive. Loading server on it now, will setup cluster and run it overnight with various tests.

What 10.3 server version are you using? I assume that 10.3.0 server won't install on it.

[Xaositect]

Ok, Cluster of four minis, (2) 1.42s and (2) 1.25s, all at 1GB RAM, gave me a search time over 30sec/search at 1423 SPM when IP was over ethernet and heartbeat was over Firewire, 1481 SPM when IP was over Firewire and heartbeat was over ethernet.

I retrieved the iMac from the family member who has been using it and ran a few compile jobs on it.

gcc 3.3 from the Xcode tools disk image on the mini were installed to both the iMac and all units in the cluster. The cluster ran about 13-15% faster. I should point out all tests with the iMac were run with Energy Saver>Processor set to highest, not auto. (I don't do a lot of coding, this was run for osxisfun's benefit). The primary slowdown was 100Base-t ethernet. Running IP over Firewire, heartbeat on ethernet, the cluster was about 15-18% faster than the iMac.

~~~~~

My opinions, based on above tests: The 80GB drive is worth the $50, just so you don't get a Fujitsu drive. The 1.42 may be worth the $50 (assuming the $50 for the drive would be spent) depending on your usage - database no, video/compile yes. Memory should be taken to 512MB or 1GB, performance increase is tremendous (obviously). Database cluster users would probably prefer the Xserve as disk is the limiting factor there - moreso than video/compile, it seems. Also, check license costs. I use mySQL for the database, Filemaker users would have to buy a liscense for each CPU. If your compiler or rendering software follows Filemaker's style, go Xserve.

~~~~~

How to install OS X Server on a mini:

1. Just use install disk 2 from the OS X Server package, and run all updates. The OS X Server 10.3.7 combined update won't run this way (hopefully 10.3.8, when available, will) but it will work.

2. Install OS X Server on an eMac, update to 10.3.7, CCC it to the mini, then run the OS X installer disc that came with the mini if you want sound (it will upgrade the install with 900k of sound drivers).

Which you use will obviously depend on availability of an eMac. I used option 2 (used a friend's eMac). I did not bother with the sound files except on one unit, just to make sure it would work. However, one eMac (as a direct window to the cluster) added in would not be a bad idea.

[osxisfun]

Hi,

Thank you very much:

gcc 3.3 from the Xcode tools disk image on the mini were installed to both the iMac and all units in the cluster. The cluster ran about 13-15% faster. I should point out all tests with the iMac were run with Energy Saver>Processor set to highest, not auto. (I don't do a lot of coding, this was run for osxisfun's benefit). The primary slowdown was 100Base-t ethernet. Running IP over Firewire, heartbeat on ethernet, the cluster was about 15-18% faster than the iMac.


I'm a little confused. 5 macs and only a 15% reduction in compile times

Ugg.

Or am i misunderstanding it.

Double ugg. The ethernet seems to be a problem even when sending over "source/compile code" from gcc.

[typoon]

Originally posted by Xaositect:
Ok, Cluster of four minis, (2) 1.42s and (2) 1.25s, all at 1GB RAM, gave me a search time over 30sec/search at 1423 SPM when IP was over ethernet and heartbeat was over Firewire, 1481 SPM when IP was over Firewire and heartbeat was over ethernet.

I retrieved the iMac from the family member who has been using it and ran a few compile jobs on it.

gcc 3.3 from the Xcode tools disk image on the mini were installed to both the iMac and all units in the cluster. The cluster ran about 13-15% faster. I should point out all tests with the iMac were run with Energy Saver>Processor set to highest, not auto. (I don't do a lot of coding, this was run for osxisfun's benefit). The primary slowdown was 100Base-t ethernet. Running IP over Firewire, heartbeat on ethernet, the cluster was about 15-18% faster than the iMac.

~~~~~

My opinions, based on above tests: The 80GB drive is worth the $50, just so you don't get a Fujitsu drive. The 1.42 may be worth the $50 (assuming the $50 for the drive would be spent) depending on your usage - database no, video/compile yes. Memory should be taken to 512MB or 1GB, performance increase is tremendous (obviously). Database cluster users would probably prefer the Xserve as disk is the limiting factor there - moreso than video/compile, it seems. Also, check license costs. I use mySQL for the database, Filemaker users would have to buy a liscense for each CPU. If your compiler or rendering software follows Filemaker's style, go Xserve.

~~~~~

How to install OS X Server on a mini:

1. Just use install disk 2 from the OS X Server package, and run all updates. The OS X Server 10.3.7 combined update won't run this way (hopefully 10.3.8, when available, will) but it will work.

2. Install OS X Server on an eMac, update to 10.3.7, CCC it to the mini, then run the OS X installer disc that came with the mini if you want sound (it will upgrade the install with 900k of sound drivers).

Which you use will obviously depend on availability of an eMac. I used option 2 (used a friend's eMac). I did not bother with the sound files except on one unit, just to make sure it would work. However, one eMac (as a direct window to the cluster) added in would not be a bad idea.

Sweet. What do you mean use the install disk 2? You mean boot from it or just install everything on it? What about just reformatting the mini and cccing it from another machine with the 10.3.7 updates and stuff already applied?

What files off the Mini installer do i need for sound?

[Partisan01]

Originally posted by osxisfun:
its mostly for the complete rebuild which takes about 15-20 minutes. you are right my incremental builds do not take much time but when i get into sections that i touch header files the build times take 15-20. if i can spend a thousand bucks and get that down to 6 minutes (i also have a dual g4 laying around) it would be worth it to me.

i just hate waiting those 15 minutes and love the concept of a stack o' minis kicking in when i need them.

edit: i should also say, i am on a dual 2 g5 and like the mini cluster idea because i can buy into a cluster in 500$ at a time, whereas a $4000 dual xserve is out of my budget (at least until this summer)

If all you're doing is compiling could you do it diskless and just use the processor? I did a bit with beowolf clustering about 6 years ago, and we booted all the machines off a main machine diskless, then just shared the files over NFS and used the processors on each machine for the horsepower.

If you choose to use MacOS here's a great thread on netbooting OS X with opendarwin
here

[Xaositect]

Ok, questions:

1. The four minis in cluster were about 13-15% (or 15-18%) faster than the iMac alone. Oops, iMac aquired for compile tests was a 1.8GHz 17" model with 1GB, not the same as the one used for database tests. Testing four units clustered was not exactly optimal. I read somewhere (source escapes me at the moment) that the tools I used have overhead hits at 2, 4 and 16 CPU points (not 8, for some odd reason, which is why I wish I could find the link). So, while four minis outperform an iMac that costs a bit less, as you add more you will get better returns (until you take the next hit at 16, where adding the 16th unit creates almost as much overhead as it processes). YMMV.

2. Install disk 2 of Mac OS X Server: run the installer(s) on it. CCC is covered in option 2 to install. The mini comes with sound driver files not in the standard release of 10.3.7, they total about 900k. If you run the OS X install disk that came with the mini after CCCing another 10.3.7 disk to the mini, it will upgrade the install with these drivers.

3. As stated above, I used Apple's clustering tools, which do require the machine have a drive, and the majority of my tests were database related, requiring disk access to read the tables (spread across the volumes for higher throughput speeds). This is why the minis did so low, but again, it was about what I had expected from a solution based on 4200RPM drives. The Linux/BSD options for clustering are available, and take a little longer to setup, but may be better for the compiler/video renderer than what I used. Anyone looking at doing anything this advanced will probably do a little homework and pick the best solution of those offered. I used the simplest to setup free ones for a rough idea of performance.

[osxisfun]

thanks Partisan01. will bookmark this link.

and a big thanks to Xaositect for performing the tests. guess i'll have to win the lottery and just go with a rack of xserves and xsan fibre connections...

3-49-59-291-4

no that's not it

1-3-555-6-7-34

nope. let me buy another one...

[typoon]

Originally posted by Xaositect:

2. Install disk 2 of Mac OS X Server: run the installer(s) on it. CCC is covered in option 2 to install. The mini comes with sound driver files not in the standard release of 10.3.7, they total about 900k. If you run the OS X install disk that came with the mini after CCCing another 10.3.7 disk to the mini, it will upgrade the install with these drivers.

I'm still not sure what you mean the files off the OS X Server CD It doesn't have anything related to sound. Also when I use the mac Mini installer it comes up with only the install that wants to restart. Do I need to do that? Do I need to use Pacifist and just grab the package file for those sound drivers? It's a server so it doesn't really need sound but it would be nice.

[larkost]

Originally posted by osxisfun:
I'm a little confused. 5 macs and only a 15% reduction in compile times

Ugg.

Or am i misunderstanding it.

Double ugg. The ethernet seems to be a problem even when sending over "source/compile code" from gcc.

No, you are not misunderstanding the numbers. There is a really high cost to distributed computing. 2 computers does not equal (or even come close to) double the speed. You usually assume that you are going to lose at least %40-%60 of the speed of each computer in the overhead, unless you can break down the job into small-data, long run-time packets. This does not describe compiling.

If you want to do clustering, then you need to ignore the minis and go with the much better performing cluster XServes.

[osxisfun]

right.

i was hoping for 40-50% range. 13% is depressing...oh well back to compiling...

[JustinHoMi]

Actually, distcc gives pretty good returns. Here is a snippet from their website:

distcc is nearly linearly scalable for small numbers of machines: Building Linux 2.4.19 on a single 1700MHz Pentium IV machine with distcc 0.15 takes 6 minutes, 45 seconds. Using distcc across three such machines on a 100Mbps switch takes only 2 minutes, 30 seconds: 2.6x faster. The (unreachable) theoretical maximum speedup is 3.0x, so in this case distcc scales with 89% efficiency.

I did some tests a while back, and got similiar results. I wonder if there is something wrong with your setup, or if there is a problem with the old version of distcc that xcode uses.

What was the compile time using only one Mini?

[qnc]

What about the mini and steraming server?

[osxisfun]

Originally posted by JustinHoMi:
Actually, distcc gives pretty good returns. Here is a snippet from their website:



I did some tests a while back, and got similiar results. I wonder if there is something wrong with your setup, or if there is a problem with the old version of distcc that xcode uses.

What was the compile time using only one Mini?

keep hope alive!

(i really have to get my but in gear and test this out with my spare dual 867 g4. i just have multiple squared things i am working on.)