[Eug Wanker]

See here:

1100x2 G5 2.0

Each with 4 GB RAM and 160 GB hard drive space.

Holy crap! 4.4 terabytes RAM and 175 terabytes hard drive space.

Between Sun, Dell, IBM/AMD, HP/Itanium, and Apple, Apple was the cheapest. The strange part is that they came with "high end graphics cards".

[mac freak]

Maybe they're using it as a 3D render farm as well

[Eug Wanker]

Originally posted by mac freak:
Maybe they're using it as a 3D render farm as well

A render farm doesn't need video cards, if that's what you were getting at. Everything is done via the CPU.

Summary of the specs:

1100 Dual CPU G5 2.0 GHz machines
4 GB RAM per box (total 4.4 terabytes RAM)
160 GB HD space each (total 176 terabytes HD space)
"High end graphics cards" - why I don't know - I'm guessing Radeon 9600 Pro since that's the stock config.
4 head nodes
1 management node
Mac OS X
C/C++ compilers are IBM xlc and GCC 3.3
Fortran 95/90/77 compilers are IBM xlf and NAGWare
24 Mellanox Infiniband switches
Cisco Gigabit Ethernet switches
Liebert liquid cooling system - 2+ million BTUs
Custom designed racks


Reason why they bought it from Apple:

Slide Four
Choosing the Right Architechture
- cost vs. performance (purely)
- total cost $5.2 million includes system itself, memory, storage, and communication fabrics
- one of the cheapest systems of its kind

Slide Five
Architectural Options [or something like that]
- Dell - too expensive [one of the reasons for the project being so "hush hush" was that dell was exploring pricing options during bidding]
- Sun (sparc) - required too many processors, also too expensive
- IBM/AMD (opteron) - required twice the number of processors and was twice the price in the desired configuration; had no chassis available
- HP (itanium) - ditto
- Apple (IBM PPC970) - system available with chassis for lowest price

[AssassyN]

Very impressive!

I know why they went for 4GB per system instead of the max of 8GB; price to performance ratio. The added cost it would inflict to install 8 1GB DIMMS into 1,100 machines would boost the bottom line pricetag up GREATLY...still though, a very insane setup.

[Hydra]

I'm just wondering... If I drove down there this weekend and slipped one of those volunteers a $100, would they reaaaallly notice 1099 machines instead of 1100 Anyone want to carpool from NY?


-Jerry C.

[mac freak]

They seriously don't use GPU's in render farms? Isn't that rather inefficient, seeing as video cards from several years ago are still better 3D renderers than most modern processors? (I once read an article comparing the 3D rendering power of a 3 GHz P4 to a 32 MB Radeon DDR, and the P4 just barely won. Obviously, a comparison to a 9600 Pro or 9800 Pro would be an absolute slaughter by ATI).

Is there simply no way to effectively combine the power of multiple graphics cards in multiple systems?

[Eug Wanker]

An interesting tidbit:

The IBM Opteron supercomputer will be part of a 2636 processor setup, including 2116 Opterons 246 2.0 GHz chips, and 520 Itanium 2s.

The total power will be 11.1 Teraflops with the 2636 processors, with 8.5 of those Teraflops coming from the 2116 Opterons.

Interestingly, the dual G5 system is supposed to hit over 10 Teraflops, with 2200 chips.

[bauhaus]

Well see about the 10Tflops. The math works out to about 4Tflops on LinPack (8Tflops non-LinPack)... I have no idea what kind of voodoo will be done to milk 2 more Tflops out of this system. Technically it should be impossible. 8Tflops should be the max.

Apple marketing???

[Eug]

What is Linpack and non-Linpack? How do you get your numbers?

Still, 8 Teraflops should put them into the top 10, assuming we can use the non-Linpack number.

[Scotttheking]

Originally posted by Hydra:
I'm just wondering... If I drove down there this weekend and slipped one of those volunteers a $100, would they reaaaallly notice 1099 machines instead of 1100 Anyone want to carpool from NY?


-Jerry C.

[Crusoe]

Originally posted by Eug Wanker:

"High end graphics cards" - why I don't know - I'm guessing Radeon 9600 Pro since that's the stock config.

Pehaps they also wish to develop a 1100 multiplayer first person shooters or 3d virtual classrooms.

[graveguy]

Oh, sure it sounds good, but how fast can it rip my latest Coldplay CD?

[jclarkv]

Originally posted by mac freak:
Maybe they're using it as a 3D render farm as well

Or maybe they've figured out a way to tap the horsepower in the graphics cards.

Or maybe the plan on giving these to incoming Freshman next year so they can get dual 3 GHz machines then.

[Cipher13]

Originally posted by AssassyN:
Very impressive!

I know why they went for 4GB per system instead of the max of 8GB; price to performance ratio. The added cost it would inflict to install 8 1GB DIMMS into 1,100 machines would boost the bottom line pricetag up GREATLY...still though, a very insane setup.

The max is 16, really.

[Eriamjh]

Originally posted by graveguy:
Oh, sure it sounds good, but how fast can it rip my latest Coldplay CD?

It's so fast, it will be done before you put in the disk!

Seriously, what do you USE an 1100 DP2GHzG5 computer for?

[chris v]

Originally posted by Eriamjh:


Seriously, what do you USE an 1100 DP2GHzG5 computer for?

Whatever the h*ll you want.

Actually, (quote) "Virginia Tech will use the cluster to perform research on nanoscale electronics, chemistry, aerodynamics, molecular statics, computational acoustics and molecular modeling, among other tasks. "

CV

[Bbazzarrakk]

They seriously don't use GPU's in render farms? Isn't that rather inefficient, seeing as video cards from several years ago are still better 3D renderers than most modern processors? (I once read an article comparing the 3D rendering power of a 3 GHz P4 to a 32 MB Radeon DDR, and the P4 just barely won. Obviously, a comparison to a 9600 Pro or 9800 Pro would be an absolute slaughter by ATI).

Is there simply no way to effectively combine the power of multiple graphics cards in multiple systems?

Graphics cards are used to make real time graphics we dump to the screen, in a video game for example. Current graphics cards draw with as much quality as they possibly can while still cranking out a solid framerate.

Render farms are generally used to build top notch graphics, like Pixar's movies. They typically use some varient of raytracing, which is pretty much drawing objects with math. Math happens in the processor. This can be a painfully lengthy process, but it produces gorgeous graphics, with the right amount of effort. You can easily get into needing hours to build a second of animation this way, depending on the quality.

So to some up, while they do similar things, they target two different kinds of problems.

There has been some attempts at using graphics cards for "Data Munging", which could cloud the issue a little, but my understanding is that it has been of limited success.

Well, I hope that answers your questions.

Baz

[Eug]

Originally posted by graveguy:
Oh, sure it sounds good, but how fast can it rip my latest Coldplay CD?

Well, considering that to "rip a CD" is to extract the audio from CD to the hard drive, it would take the same amount of time as a single dual G5 Power Mac, since the optical drive is the bottleneck.

Obviously, encoding would be a bazillion times faster (if the software was written for this cluster), but again, the optical drive is the bottleneck for a rip and encode of a CD.

In short, a 10 Teraflop computer would rip and encode a CD no faster than about 25-50X, depending on the drive.

[DrBoar]

Niw, if we could sneak that setup into the "team macnn" fold at home or D2OL that would be nice

[AssassyN]

The real question is...which of those lucky 1,100 actually get a monitor plugged into it?

[devmage]

Has anyone heard what they are using to cluster them together? Is this software they are writing or something that exsists?

Devmage

[stuffedmonkey]

Might I now suggest the true purpose of this thing, a massive denial of service attack against uva.edu

-stuffed Monkey, VT class of 1999

[juanpacolopez]

Originally posted by Bbazzarrakk:
Graphics cards are used to make real time graphics we dump to the screen, in a video game for example. Current graphics cards draw with as much quality as they possibly can while still cranking out a solid framerate.

Render farms are generally used to build top notch graphics, like Pixar's movies. They typically use some varient of raytracing, which is pretty much drawing objects with math. Math happens in the processor. This can be a painfully lengthy process, but it produces gorgeous graphics, with the right amount of effort. You can easily get into needing hours to build a second of animation this way, depending on the quality.

So to some up, while they do similar things, they target two different kinds of problems.

There has been some attempts at using graphics cards for "Data Munging", which could cloud the issue a little, but my understanding is that it has been of limited success.

Well, I hope that answers your questions.

Baz

You're both right and wrong.

You're right in the sense that pro 3D rendering is based on raytracing. You're a bit confused however, on what that means exactly.

Modern 3D API's (OpenGL for example) use a series of triangles to form polygons, and groups of polygons to form 3 dimensional shapes. To those shapes, they apply textures, blending modes, and things like lightmaps to determine how an object's appearance is changed by the light source.

Raytracing refers to "tracing" the path of individual "rays" of light from the light source to the object and through the the shadow. Dynamically calculating pixel values based on reflection intensity along the way. This is VERY slow, but also quite pretty.

Current top-end 3D cards (such as the RADEON 9600/9800 and newest GF FX's) have programmable pixel/vertex shaders, and are starting to have LARGE portions of their render path be entirely programmable.

It has long been a goal of 3D hardware designers to find a way to leverage the ENORMOUS power found in modern 3D cards (think on the order of 100Gflops for a top-end RADEON 9800 at _CERTAIN_ tasks. Once we see 3D cards who are programmable enough to actually let them run unsigned code (say, for a distributed video render) we will see the likely DEATH of the "render farm" concept. Or, and this is much more likely, the quality of 3D graphics will evolve to the point of being almost unrecognizable from "real" footage.

[KeilwerthSX90R]

I did mechanical engineering undergrad and ran a lot of fairly complex computer models. Man would I have loved to run them on a system like the one VT put together.