[OwlBoy]

So....

Yeah...

ftp://ftp.apple.com/developer/Tool_C...rid1.0.dmg.bin


Imagine waking up one day to find your Mac has solved a vexing scientific problem. While the cure to cancer, super-efficient solar power and ending world hunger are a ways off, you can combine your computing resources using Xgrid ? and help usher in a new era of biological breakthroughs, rocket science and advanced models of scientific phenomena.

-Owl

[Peter]

Awesome!
I'll use this on my G5 Cluster in my shed!

[mishap]

This will be cool once i figure out how to use it.

[ShotgunEd]

In case anyone was wondering...

from the readme:
"Xgrid is a solution for parallel and distributed high performance computing. Xgrid is made up of a collection of software, APIs, and protocols. The Xgrid architecture enables administrators to group locally networked computers into clusters and allow others on the network to remotely submit long-running computations to the clusters. "

[OwlBoy]

Originally posted by ShotgunEd:
In case anyone was wondering...

from the readme:
"Xgrid is a solution for parallel and distributed high performance computing. Xgrid is made up of a collection of software, APIs, and protocols. The Xgrid architecture enables administrators to group locally networked computers into clusters and allow others on the network to remotely submit long-running computations to the clusters. "

Yeah I was about to post that.

I guess its mainly for developers and scientists.

-Owl

[sushiism]

Always found that stuff like SETI really pathetic, like i'd hurt my powerbook making it run 100% cpu all the time just for garbage like that. Its like a HOW BIG IS YOUR **** contest for geeks

[Gee4orce]

On a slightly related topic, I see Oracle announced 10g for Mac OS X - the 'g' stands for 'grid computing' (like 9i was for 'internet'). Grid is a hot button topic and it's good to see that Apple is at the forefront, and especially good to see Oracle on board.

[TC]

Originally posted by sushiism:
Always found that stuff like SETI really pathetic, like i'd hurt my powerbook making it run 100% cpu all the time just for garbage like that. Its like a HOW BIG IS YOUR **** contest for geeks

So you don't understand what this is at all then?
It's for people to set up their own local clusters to do their own computational tasks.

This kind of technology is already used in Shake to harness the resources of local computers and also in Xcode to speed up compiling of code. Maybe in the future this kind of thing can be used to get iDVD to use the 20 macs in your office to create your MPGs about 20 times faster.

[yaro]

How do I use the shell to execute ffmpeg CLI. Any documentation on creating plug-ins?

[Fillman]

Methinks that Xgrid and Rendezvous is like getting a few extra processors for free. Fast forward a year or two and imagine when there should be Xgrid aware consumer level programs out there and any computationally intensive program can use any idle processor available.

Apple should build this into iLife applications asap and this will blow the opposition out of the water. Any home with more than one mac can be a multi processing cluster. Just build it into 10.4 as the default.

I'm impressed!

[unimaxium]

I'm not sure how to get it to work for me. Can I distribute the mandelbrot demo over my two computers to see how fast I can make the tachometer go? And can I use this to run unix commands that distribute the workload over multiple comps? It'd be really cool if it's possible to use this to compile large programs that I have to compile in the terminal and can't use the XCode workload distributing, but I'm not sure if it's possible to do that. Anyone know if it is? Can anyone give me a tip or two to understand how to get it to do stuff ? Or is it way too complicated for me to do stuff like that?

[Fillman]

Originally posted by unimaxium:
I'm not sure how to get it to work for me. Can I distribute the mandelbrot demo over my two computers to see how fast I can make the tachometer go? And can I use this to run unix commands that distribute the workload over multiple comps? It'd be really cool if it's possible to use this to compile large programs that I have to compile in the terminal and can't use the XCode workload distributing, but I'm not sure if it's possible to do that. Anyone know if it is? Can anyone give me a tip or two to understand how to get it to do stuff ? Or is it way too complicated for me to do stuff like that?

The answer to your questions is yes, but, like you I'm struggling with the details of client, controller and agent to make it all work in demo mode.

[OwlBoy]

Originally posted by yaro:
How do I use the shell to execute ffmpeg CLI. Any documentation on creating plug-ins?

Ohh never thought about that... Video Compression.

-Owl

[dr. zoidberg]

for those wondering ...

(from the FAQ-pdf)

Can I use Xgrid to speed up my favorite application?

� The short answer is no.

� The long answer is that if your favorite application performs long-running calculations that can be factored out into smaller support programs, then the application developer can write a small Xgrid plug-in to submit those long-running computations to the grid. Xgrid was designed with this expandability in mind.

[Laurence]

So has anyone figured out how the command line "create custom plugin" or "shell" options work. It would be really nice to tie into ffmpeg or mencoder. If those apps need to be modified to work well with this, I hope that gets a high priority in future versions!

The possibilities that this API lead to could be amazing.

[parsec_kadets]

Originally posted by TC:
So you don't understand what this is at all then?
It's for people to set up their own local clusters to do their own computational tasks.

This kind of technology is already used in Shake to harness the resources of local computers and also in Xcode to speed up compiling of code. Maybe in the future this kind of thing can be used to get iDVD to use the 20 macs in your office to create your MPGs about 20 times faster.

Actually, not really. Video compression has to be done sequentially. So it's possible you could use Xgrid to process 20 different videos at the same time, but you couldn't split one video into 20 different segments.

[TC]

You can normally join and split these things, hell if it just passed the processing of the audio off to another machine it would still help to speed up the whole process.

I think future versions of software will prove you wrong and I wouldn't be surprised to see a version of Apple's own Compressor able to do these kinds of things.

[BJPirt]

As I encode a fair few divxs, I thought I'd see about trying xgrid for this.

It _almost_ works - very close indeed. After spending quite some time figuring out a way of doing it I found the following method;

1) demux the .vob - we will encode the audio at a later date - I used bbDemux to do this

2) split the m2v file into 10 second segments. I understand that splitting a file may have bad effects, but myreasoning was - A 10 second segment is the correct number of frames between keyframes and so as each file is compressed, there will be a keyframe at the beginning and no more (with the keyframe rate set to 10+ seconds). So once we rejoin them we keep the nice even keyframe rate. The downside to this entire technique is of course that the vbr encoding is pretty much unused as there isn't enough data to make accurate rate estimations.
Anyway, I did this step using mpgtx

3) next, launch xgrid and start a new service.

4) choose the job type "create custom plugin"

5) I use mencoder to encode with, so your parameters will probably change. But, the following things need setting;
command: /usr/bin/mencoder (or wherever yours is)
argument 1: file list = whatever folder holds the split file

Now, here was the first stumbling block - getting the parameters fed nicely to mencoder. The command line I would usually use is;

/usr/bin/mencoder /storage/movies/mymovie/chunks/mymovie-01.mpg -ovc lavc -sws 0 -lavcopts vcodec=mpeg4:vbitrate=803:keyint=240:vqmin=2:vqmax =15 -ofps 25 -vop scale=640:352,crop=718:452:2:62 -o /storage/movies/mymovie/chunks/movie.avi

the first two parts of that are the binary and the input file. These are taken care of with 'command' and 'argument 1'. Which leaves us with;

-ovc lavc -sws 0 -lavcopts vcodec=mpeg4:vbitrate=803:keyint=240:vqmin=2:vqmax =15 -ofps 25 -vop scale=640:352,crop=718:452:2:62 -o /storage/movies/mymovie/chunks/movie.avi

What you need to do to get these variables in is to put the parameter name and the parameter value in two seperate arguments, so;
argument 2 : -ovc
argument 3 : lavc
argument 4 : -sws
argument 5 : 0
argument 6 : -lavcopts
argument 7: vcodec=mpeg4:vbitrate=803:keyint=240:vqmin=2:vqmax =15

and so on, you get the picture.

In this way, all of the chunks get dished out and completed. But there is one _major_ problem. There is no way of specifying a unique output file name (that I can see anyway, hence posting it here - more eyes) so 'movie.avi' keeps getting overwritten :-(

Also, I don't think xgrid is 'aware' of this file getting written, so it doesn't gather it into the output folder you specify.

Anyway, I thought I'd post it here in case anyone else wanted to have a play with this extremely cool technology. My main reason for trying this really was simply 'because I can'
:-)

All the best, and good luck xgridding,

BJPirt

[Rev-O]

Okay, So I'm giving XGrid a whirl (I was bored), and have it installed on the 3 Macs in my wired network.

They can't seem to find each other. The two agents are "unable to connect" to the controller. The controller goes out on Rendezvous and endlessly spins its wheels.

Normally, I can transfer files between my Macs all I want. They're hooked up thru an Asante wired router.

Like I said, at first I was just bored and thought it would be mildly interesting to make a cluster. Now I'm aggravated that the flippin' thing doesn't flippin' work when it should flippin' work. And I'm sure I'm overlooking something flippin' obvious. And either I get it to work or I just vaporize the flippin' thing.

Suggestions or guidence?

[larkost]

parsec_kadets: Actually, video compression could be accelerated by something like XGrid. You are right in that with most codecs most frames do depend on frames preceding it, but there are "keyframes" that a free-standing, or at least mostly so. Then the "interstitial" frames that come after them are dependent on them. How often you have keyframes, an how truly independent of each other they really are varies wildly by codec (and settings). If you were to offload the data to another computer and be able to declare the first frame a keyframe (or do something to figure out where to best mark one). Then you could offload processing to other computers.

The big issues would be to figure out sane ways of doing this sort of thing, and the fact that what is sane for one codec will kill another... with the panoply of codecs that QuickTime has, this is a real issue.

[Waragainstsleep]

To run the Mandlebrot set test over two or more Macs, set the agent service going on all machines, and the controller service on one using the Xgrid prefpane. Use connect to server to mount all available drives on all three Macs. Open the Xgrid app on one Mac (shouldn't matter which one, but some configs seem to work better than others, so try another if the first doesn't work right) You only need one client (Xgrid app) running to test. Then connect to the controller service and run the job.
All 3 Macs should show up as available agents now.

A few points:
Turn off all firewalls;
Set them all to always accept tasks in the Xgrid pane;
Turn off passwords (just one less thing to worry about for testing);

[andrejbl]

For those with a few CPU cycles to spare might be interested in this. Basically, a researcher at Stanford is using XGrid to harness volunteer's computers to speed up his own calculations.

As you would expect from Apple, the client software is easy to setup up for this. As a bonus, you can set up an XGrid screensaver that shows a tachometer with the current cluster power in GHz (35 GHz at the moment). Very cool.

Anyway, a clever application of XGrid over the internet. Apple thinks enough of this researcher's work that they have asked him to present his work at WWDC.

Andre

[Thain Esh Kelch]

Yes!! Now I can play UT2k4 on my bondi-blue! it just needs to get a giga-bit ethernet card so that it can talk with my G5-cluster in the closet!!