[DaGuy]

I have a 17" iMac and I really tend to push it in terms of CPU cycles. So, say that in the future, I purchase an XServe. Now assume that I'm running some intensive computations. Could I cluster the Xserve and the iMac's CPUs to work in parallel? Apparently, you can do this multiple XServes but what about with an iMac?

What about a cluster with just iMacs? Say an old bondi blue 233MHZ revB and a newer 17" 800 MHZ flat screen iMac.

Thanks!

[TC]

Originally posted by DaGuy:
I have a 17" iMac and I really tend to push it in terms of CPU cycles. So, say that in the future, I purchase an XServe. Now assume that I'm running some intensive computations. Could I cluster the Xserve and the iMac's CPUs to work in parallel? Apparently, you can do this multiple XServes but what about with an iMac?

What about a cluster with just iMacs? Say an old bondi blue 233MHZ revB and a newer 17" 800 MHZ flat screen iMac.

Thanks!

When Apple sells the xserve as a cluster node it has less i/o options (no cdrom or graphics card) as it is just used for processing.

The problem is that you rely on the software you are using to give you clustering capabilities. Right now the main software to harness this is 3D rendering applications. Using software like Lightwave you could harness the power over your iMac and an xserve connected together to render an animation more quickly.

Hopefully Apple will make clustering much easier for application writers to produce in the future by harnessing the power of rendezvous. Computers which could be used as part of a cluster could 'advertise' their services using rendezvous making the set up process much easier.

To give the Mac true clustering capabilities you should tie it in at a low level. This would allow the easy moving of tasks between cluster nodes if it is needed (a typical clustering opurtunity is a network of office computers which remain unused during periods of the day but have to give processing power back if a user needs it).

I wrote my thesis at University on Network Clusters + would love to see it in the mainstream. Apple has already gone a long way to making it possible by making an OS which supposrts multiple processors, fast communications via Firewire 800 would also help a lot.

Just imagine a scenario when you could harness the power of your office of 30 macs to get your iDVD source material converted to MPEG2 in a fraction of the time with little set up required.

[real]

CAN'T WAIT FOR THAT!

[bstad]

There is some clustering software called Pooch (I really didn't make that up!) and they have a trial you can download. I played with it at one time and found out a couple of things:

1. You have to have software written specifically to take advantage of clustering.

2. You need to use machines of fairly equivalent capabilities. In my case I used a 233 iMac, a 266 minitower, and a dual 450 G4.
When using the 233 and the 266, you could notice a big improvement in capability, really showing off the clustering.
When using the dual with one or both of the others, the increase was barely noticeable, and sometimes it was even slower, probably because of overhead.

Fun to play with, but if you don't have software that can make use of it, it's just a toy.

[DaGuy]

So I gather than even with a rack full of XServes I still would not get automatic parallelism out of the CPUs. It really depends on the software. Did I get it right?

What about just plain load distribution i.e., if I ran 20 apps in a cluster. Would MacOSX know how to run 5 in this CPU, 5 in this other CPU and maybe 10 on that other CPU? This scenario is not relying on the individual software package to leverage the cluster it would rely on the OS to decide where to run them. Does this make any sense?

Thanks!

[Catfish_Man]

Originally posted by DaGuy:
So I gather than even with a rack full of XServes I still would not get automatic parallelism out of the CPUs. It really depends on the software. Did I get it right?

What about just plain load distribution i.e., if I ran 20 apps in a cluster. Would MacOSX know how to run 5 in this CPU, 5 in this other CPU and maybe 10 on that other CPU? This scenario is not relying on the individual software package to leverage the cluster it would rely on the OS to decide where to run them. Does this make any sense?

Thanks!

The OSX kernel (the low level core part of the system) knows how to do clustering (it's what it was designed for, in fact.), but Apple has yet to make use of it. An ideal solution would be one that used rendezvous to find other machines on the network, then clustered them with the mach kernel. The only part they need to implement is something that decides whether it's worth it to share a thread/program over the network (and perhaps a preference pane for setting it up, although there really wouldn't be much to set up).
The load sharing thing is really easy to make, except for handling the user interface. The unix graphics system (X Windows) can run over a network, but Quartz isn't designed to do that. Apple Remote Desktop does it, but I don't know how it does it.

[Coxy]

As it stands now, (SOHO) clustering is better for computationally intensive tasks that don't rely on particularly large amounts of data.

For example, if you wanted multiple machines to work on encoding MPEG2, you're still talking about transferring gigabytes of data across a network.

[TC]

Originally posted by Catfish_Man:
As it stands now, (SOHO) clustering is better for computationally intensive tasks that don't rely on particularly large amounts of data.

For example, if you wanted multiple machines to work on encoding MPEG2, you're still talking about transferring gigabytes of data across a network.

Absolutely right, the main barrier to clustering is network bandwidth. But, with networks connected up using Firewire 800 or 1000Mb Ethernet then it could be worthwhile.

[TC]

Just noticed that Apple has started using clustering in the latest version of Shake:

"New Shake Qmaster network render management software included with Shake for Mac OS X uses Apple's Rendezvous networking technology to automatically identify available render nodes and clusters on a network while load balancing optimizes usage across each machine in the workflow. If one Xserve or Power Mac G4 goes offline, built-in fault tolerance allows Shake Qmaster to re-route tasks to other render nodes in the cluster, so the project gets done on time. As Shake Qmaster is based on an open architecture, it can also provide distributed rendering support for many leading third-party applications, such as Alias/Wavefront's Maya."

Further investigation shows up that qmaster is linked to a Sun project called Grid Engine:
http://gridengine.sunsource.net

Doesn't mention anything about Mac OS X but it does some to have some parts written expecially for Darwin.

Finally there is a pdf here which does talk about a patch being available to build Grid Engine on OS X:
http://conferences.oreillynet.com/pr...chen_final.pdf

Sounds like some progress is being made by Apple in this field.

[Richard Edgar]

The answer to the original question depends on what you want to do. For some tasks, network bandwidth usage is negligible and load balancing is trivial (e.g. RC5 cracking). At the other end of the scale, SGI will be quite happy to sell you a system with bandwidth and latency which make gigabit ethernet look like a 2400 baud modem. For 3D animations, each frame should be completely independent, so I imagine that load balancing on a heterogeneous cluster will be fairly straightforward (provided the number of frames is large).

[Drizzt]

Hoo.. just had an idea..

Imagine having a tower, and buying yourself a Powerbook or an iBook.

You startup your new laptop and the assistant asks you "Will this laptop be a standalone workstation or a slave?"

You choose slave, you hook a FireWire cable from the PowerBook to your PowerMac/iMac and the data on your desktop's HD is starting to be copied on your laptop.

A few minutes later, you get on your desktop, you have 2 CPUs and load balancing on HD read.

You take the laptop appart, it is now in 'un-mastered' mode (i.e. working separately.. like a normal laptop, but you have exactly the same content on you laptop that you have on your desktop).

You hook it back, the 2 machines sync together and you get RAID 0 on read, RAID 1 on writes and Bi-CPU!

I know it's day-dreaming.. but that could be someday

[TC]

At the moment this technology is only harnessed for things like 3d rendering.

If Apple added this to products like DVD Studio Pro, iDVD, iMovie and Final Cut Pro it would kick ass. Basically for most of those it would just mean adding clustering capabilities to Quicktime. Any time you have some encoding to do which takes over x number of minutes check if there are other machines around to help with the job.

Maybe even Photoshop filters which take minutes to complete could harness this. Adobe already uses the idea of breaking an image into panels, these could be farmed out to individual computers.

Anyone think of any other computational intensive tasks which this could be used for?

[sambeau]

Originally posted by TC:
Maybe even Photoshop filters which take minutes to complete could harness this. Adobe already uses the idea of breaking an image into panels, these could be farmed out to individual computers.

As long as the computation would be significantly slower than breaking the file into chunks, copying the graphics file segements to the other machines, filtering them and stitching them back together.

Also, assuming that the filter can work in this way (ie consider a blur - it requires data from around the chunk edge to calculate a correct blur).

The best applications for this are ones that require turning calculations or algorithms into data.

Postscript ripping (anything to do with rasterizing vectors in fact), animation tweening, soft-synths, sound filters (maybe not realtime though, as would be latency problems unless the link was fast).

Firewire 800 is very fast though.. so clustering should be possible with these cables..

But there is loads that you can do without clustering and when we have multiple graphical logins, we should be aple to use more than one machine for such tasks without moving seat. You could be burning a CD on one machine while ripping Mp3s on another.. leave one machine rendering.. Of course - those of use who use the Terminal are allready using multiple machines for multiple tasks.

I'm logged into my machine at home as we speak - from an IBM thinkpad at work - and editing files straight to my desktop..

[TC]

Just to keep this thread alive looks like Xcode fits in quite well with this idea:

http://www.apple.com/macosx/panther/xcode.html

Even includes using rendezvous to find other computer to help with compilations.