[Eug Wanker]

VT



University of Maine (Also has camera feeds.)



(Also check my sig if you're interested in my stupid comments. )

[SafariX]

Tastes like purple.

[MindFad]

Damn, Apple really makes some sexy machines.

[olePigeon]

Imagine a Beowulf cluster of... oh.

[Boochie]

We just set up a cluster of a dozen Xserves at my office, along with a pair of RAID arrays. They are some nice looking machines, that's for certain. The fit and finish on equipment that people will never really look at except to perfrom maintenance is something else.

I had XGrid up and running on all of the machines in a matter of hours, it was incredibly straightforward.

[chris v]

Daaaaaamn. Tech Pr0n at its finest.

CV

[ReggieX]

http://www.clusters.umaine.edu/gallery/album05/dsc00465

So pretty!

[Eug Wanker]

So Apple sold 13000 of them in the last quarter. Impressive.

[olePigeon]

I guess Microsoft and the Linux groups can argue about cost all they want, meanwhile, everyone else will buy XServes with OS X.

[Eug Wanker]

Originally posted by olePigeon:
I guess Microsoft and the Linux groups can argue about cost all they want, meanwhile, everyone else will buy XServes with OS X.

Actually, for clusters, they're still mostly gonna be Linux boxes, and more than Xserves. Windows has essentially no presence in this market.

[Graymalkin]

While Linux-based clusters aren't going away I think Apple's going to become a much bigger player in the cluster market in the next few years. The G5s in the new Xserves run significantly cooler than the Xeons and Opterons of the same performance class. This is a huge factor when building a cluster because as the power requirements increase so do the HVAC and UPS requirements. Research groups on limited budgets have to figure in not only the up front cost but the cost to run the cluster for several years. Even the G5 towers run fairly cool for their power compared to Pentium 4 based systems. The low power requirements of the G5s is one of the main reasons VT built their super cluster out of them.

For Apple's part they've really been pushing the Xserve as a HPC research system. A lot of clustering is done using Linux, the major downside to this is it requires technical expertise that a lot of researches don't have the time or money to procure. Apple' is selling pre-built research systems like their bioinformatics workgroup clusters to such researchers. Being able to develop some code on a Powerbook and then transfer that directly to a cluster is a big time saver. Model runs can be done on personal desktops or laptops to tweak the performance and then the real heavy crunching can be shipped off to the cluster in the lab with no/little fuss.

[Boochie]

Quite right about the expertise issue -- I've set up Globus on Linux clusters and it was a royal pain in the a** compared to Xgrid on the XServes. Now, Globus is considerably more powerful than the current XGrid tech preview, but Apple's approach to simplifying the complexity goes above and beyond many of the cluster toolkits out there.

Apple's ability to capitalize on their potential strength remains to be seen...

Originally posted by Graymalkin:

For Apple's part they've really been pushing the Xserve as a HPC research system. A lot of clustering is done using Linux, the major downside to this is it requires technical expertise that a lot of researches don't have the time or money to procure. Apple' is selling pre-built research systems like their bioinformatics workgroup clusters to such researchers. Being able to develop some code on a Powerbook and then transfer that directly to a cluster is a big time saver. Model runs can be done on personal desktops or laptops to tweak the performance and then the real heavy crunching can be shipped off to the cluster in the lab with no/little fuss.

[Eug Wanker]

Originally posted by Eug Wanker:
VT



University of Maine (Also has camera feeds.)



(Also check my sig if you're interested in my stupid comments. )

U of M has added a ton of pix.

[Peter]

Originally posted by Eug Wanker:

(Also check my sig if you're interested in my stupid comments. )

"BTW, I took the liberty of fertilizing your caviar."

[Eug Wanker]

Originally posted by Peter:

Heh. I meant here, for this post: G5 Supercomputers Galore, in 2004.

[Eug Wanker]

U of M's

[Graymalkin]

D-amn that is a lot of Xserves. While the Xserve cluster nodes are well-built machines I really wish they would come down in price just a bit. For $3k you can pick up PowerMacs that are 25% faster and work just as well in a distributed environment running Panther client as long as you put a little elbow grease into the project. Obviously the PMs are larger and don't have as many headless management features but they do provide a bit more bang for the buck.

Idealy I'd love to see all the Xserves drop in price by at most $500. The 10-client Xserves and cluster nodes would be the same price as the 2GHz PowerMacs and the unlimited client Xserve would only be a little bit more than the 2.5GHz PowerMac.

[Eug Wanker]

Originally posted by Graymalkin:
D-amn that is a lot of Xserves. While the Xserve cluster nodes are well-built machines I really wish they would come down in price just a bit. For $3k you can pick up PowerMacs that are 25% faster and work just as well in a distributed environment running Panther client as long as you put a little elbow grease into the project. Obviously the PMs are larger and don't have as many headless management features but they do provide a bit more bang for the buck.

Idealy I'd love to see all the Xserves drop in price by at most $500. The 10-client Xserves and cluster nodes would be the same price as the 2GHz PowerMacs and the unlimited client Xserve would only be a little bit more than the 2.5GHz PowerMac.

VT's take: No ECC support in the Power Macs.

On September 23, 2003 we turned it all on and began the arduous process of stabilizing and benchmarking this machine. After many more sleepless nights and countless grams of caffeine we finally reached our 10 teraflop goal.

Well with the concept proven we now had to make sure we had a system capable of conducting scientific computation. We needed to upgrade the system to something with error correcting code (ECC) RAM. The Power Macs did not support it and the XServes were coming. So in January we tore the system down and started prepping for the XServes. And now they're here and we have our final system. The best is yet to come.

[kupan787]

How come the pictures of the U of M (and VT) xserves all look different than the pic of the xserve on the Apple website? If you look at th eracks of xserves, it appears that there is only one hot swapable drive and lots of gratting. The xserve on the apple website clearly has 3 removable drives, and 2 big intake vents. Anyone have any thoughts (or am I missing something really obvious).

[edit] NM, I just noticed the "cluster" page and those images look exactly the same as what is in the VT cluster.

[Peter]

drooooooool.

[chris v]

Originally posted by kupan787:
How come the pictures of the U of M (and VT) xserves all look different than the pic of the xserve on the Apple website? If you look at th eracks of xserves, it appears that there is only one hot swapable drive and lots of gratting. The xserve on the apple website clearly has 3 removable drives, and 2 big intake vents. Anyone have any thoughts (or am I missing something really obvious).

[edit] NM, I just noticed the "cluster" page and those images look exactly the same as what is in the VT cluster.

They make 2 different Xserves-- the regular Xserve is made to stand alone, and has the three drives. The ones in those pics are Xserve Cluster Nodes, which are made only for the purpose of being driven by other boxes in a multi-computer rig. They don't need CD drives because all the software is being installed/adminned by another head elsewhere on the cluster, and they don't need multiple drives, since all the data is being served/backed up from elsewhere on the cluster. The Cluster Nodes are a little cheaper, consequently.

CV

[Graymalkin]

Originally posted by Eug Wanker:
VT's take: No ECC support in the Power Macs.

On September 23, 2003 we turned it all on and began the arduous process of stabilizing and benchmarking this machine. After many more sleepless nights and countless grams of caffeine we finally reached our 10 teraflop goal.

Well with the concept proven we now had to make sure we had a system capable of conducting scientific computation. We needed to upgrade the system to something with error correcting code (ECC) RAM. The Power Macs did not support it and the XServes were coming. So in January we tore the system down and started prepping for the XServes. And now they're here and we have our final system. The best is yet to come.

There's plenty of clusters using non-ECC memory without much of a problem. The key tp using non-ECC systems is adding in-loop sanity checks to your math. Your loops end up slower but in the end you can get away with overall cheaper cluster nodes. VT's main problem with the non-ECC PowerMacs was they had to tune Linpack to be more cautious (slower) to make up for the lack of ECC. They wanted to really bump their Top500 score so this was a major thorn in their side.

To a researcher losing .1 TFLOPS in order to run manual error checking routines isn't really important. The difference in overall computation time might be measured in minutes, enough time to finish a cup of tea and jot some notes. When trying to place on the Top500 list every last little FLOP is dreadfully important, especially when grant money is on the line.

I don't however believe VT was wrong to replace the PowerMacs with Xserves, they're going to be much more power and space efficient. Lacking ECC however doesn't necessarily mean PowerMacs would make bad cluster nodes. They held their own quite well in the original incarnation of SystemX/Big Mac.

[Eug Wanker]

Originally posted by Graymalkin:
There's plenty of clusters using non-ECC memory without much of a problem.

Yeah, but how many have 2200 CPUs, and running jobs using all 2200?

To a researcher losing .1 TFLOPS in order to run manual error checking routines isn't really important. The difference in overall computation time might be measured in minutes, enough time to finish a cup of tea and jot some notes. When trying to place on the Top500 list every last little FLOP is dreadfully important, especially when grant money is on the line.

How important is 5th vs 8th to getting grants?

I don't however believe VT was wrong to replace the PowerMacs with Xserves, they're going to be much more power and space efficient. Lacking ECC however doesn't necessarily mean PowerMacs would make bad cluster nodes. They held their own quite well in the original incarnation of SystemX/Big Mac.

They themselves say lack of ECC was the main problem.

[Graymalkin]

The "official" speed of the cluster was really important for VT because they can now say "hey we have a 10 TFLOP cluster, we need money to do research with it". Even though it was fairly inexpensive to set up and build actually running it and developing code for it is not trivial and will take research dollars.

As for other clusters running non-ECC systems VT themselves runs a 400 CPU cluster built around AMD chips. ECC is not the end-all be-all of clustering, it just helps keep raw throughput up because less time is spent validating data integrity.

[Eug Wanker]

Originally posted by Graymalkin:
The "official" speed of the cluster was really important for VT because they can now say "hey we have a 10 TFLOP cluster, we need money to do research with it". Even though it was fairly inexpensive to set up and build actually running it and developing code for it is not trivial and will take research dollars.

As for other clusters running non-ECC systems VT themselves runs a 400 CPU cluster built around AMD chips. ECC is not the end-all be-all of clustering, it just helps keep raw throughput up because less time is spent validating data integrity.

VT's 200 node x86 cluster doesn't support ECC? I didn't know that.

BTW, here is an (old) article by VT on the G5 cluster.

Oh, and benchmarking on the G5 Xserve VT cluster has finally begun.

[Graymalkin]

I'm betting that the Xserve based cluster will actually outperform the PowerMac cluster because the Linpack benchmark will be run more aggressively. They now also have IBM's XL compiler suite to use as well as their custom tuned BLAS libraries. Having the XL compiler will speed up the non-critical parts of the code they're running which will simply increase throughput and efficiency.

I found an article somewhere with a small interview with Dr. Varadarajan asking about whether VT is going to publish their notes about building SystemX. He replied that they're intending to publish a full documentation package describing how to set up a SystemX-esque cluster starting at 64 nodes and going up from there. Hopefully Apple stays on the ball and reproduces that documentation on their site. Their existing HPC developer docs are pretty sparse. They really need to provide some good documentation on how exactly to build awesome Mac clusters.

edit: "I'm better..." geez I need to think ten times before typing.

[Eug Wanker]

Originally posted by Graymalkin:
I'm betting that the Xserve based cluster will actually outperform the PowerMac cluster because the Linpack benchmark will be run more aggressively. They now also have IBM's XL compiler suite to use as well as their custom tuned BLAS libraries. Having the XL compiler will speed up the non-critical parts of the code they're running which will simply increase throughput and efficiency.

They were running XL right from the start, although it's possible it may be more tweaked now.

I betcha they ran Linpack VERY aggressively, but just ran it a bazillion times. AFAIK, there is no limit to how many times you can run Linpack, and you can just report the fastest result.

By the way, VT 400 CPU AMD cluster uses a motherboard that does support ECC. In the VT specs it doesn't say if ECC is actually used or not though.