[Cipher13]

So, I'm on the G4/400, and it's converting a few gigs worth of AIFF files to AAC. Bit of a task, it'll take some time... and I'm not a very patient person when it comes to inanimate objects.

In the meantime, the iMac 600, the iBook 900, and all the old-world machines are just sitting here doing nothing.

I know I could set up something like this, but damned if I wanna have to do it myself... it'd take way too long.

I want to be able to distribute tasks across my network.

If the iBook is 90% idle, as it is now, I want my G4 to be able to take that 90% (or 85%, whatever) idle time and USE it. Send the data across the network, crunch it, and send it back.

I know in some cases it wouldn't be worth it due to the network latency, but this could be very cool for people with networks.

To have this built straight into OSX would seriously rock.

But yeah. Haven't put much thought into it. Just mental masturbation.

[drmbb2]

XGrid

[ginoledesma]

Having such a feature would be something else, although it will require a lot of thought and effort. Essentially, what's needed is a framework for distributed computing, and apps could be written using the framework to assign jobs to different idle computers that participate in it. However, one would have to think how things are handled in a network where several computers want to "give out tasks." There could be an option to make each computer specify which server to accept jobs from and such, but like I said, it will require some work.

[jaiqua]

Look to the software you are using and see if they support network rendering, or computation. I've been using it for years with programs like Combustion, Maya and so on, really brings down the time involved.

[Cipher13]

Yeah, I've used render slaves with Bryce and Maya and whatnot, and have distributed BLAST tasks and stuff, but i'm talking about a system-level implementation, whereby processes could automatically be shared, for any application - for example, FCP, iTunes, Cleaner, whatever...

[Jens Peter]

It does sound like a good idea, usefull too.
But from apples point of view, I think there are too few users that would use that option with ex. iTunes or some other program.
And from a developers point of view, is it then possible to do at all? I mean, if a program is written to run on one machine, can the OS then grab some of it's data and send it to some other machine to get processed ?

[Turias]

Why are you on a G4/400 when you have an unused iMac 600 and iBook 900 nearby?

[Millennium]

Originally posted by Cipher13:
Yeah, I've used render slaves with Bryce and Maya and whatnot, and have distributed BLAST tasks and stuff, but i'm talking about a system-level implementation, whereby processes could automatically be shared, for any application - for example, FCP, iTunes, Cleaner, whatever...

Not likely to be possible.

Not all algorithms benefit from a distributed approach. That's why the apps you've seen have all been written specifically to support distributed computing; they do things which could benefit from it.

Recoding audio is unlikely to benefit much from this approach. The best you could go would be to automatically give each machine a list of files to process, have each one work on those files, and possibly spread the load out so that the lists stay roughly the same length (if one machine is slower than the others, for example).

[Cipher13]

Originally posted by Millennium:
Not likely to be possible.

Not all algorithms benefit from a distributed approach. That's why the apps you've seen have all been written specifically to support distributed computing; they do things which could benefit from it.

Recoding audio is unlikely to benefit much from this approach. The best you could go would be to automatically give each machine a list of files to process, have each one work on those files, and possibly spread the load out so that the lists stay roughly the same length (if one machine is slower than the others, for example).

Not recording audio, processing it.

I know what you're saying - but most mass-processing applications would benefit, even if it's in the manner you say, which is entirely logical (send off a few files to be processed on another machine, send them back; though the toll in network traffic could get very high there).

I'm sure this is possible, and hell, I'd rather Apple spend some R&D cash on this than which candy-flavoured UI people will want to lick next.

Originally posted by Turias:
Why are you on a G4/400 when you have an unused iMac 600 and iBook 900 nearby?

The G4 is my primary machine.

17" screen, 5 hard drives, most RAM, etc.

[Earth Mk. II]

One way, and probably the simplest way, to do it might be to have a framework where an application could export some object code to a client computer and that object would be treated much like an extension or Mach-O bundle, and executed with data sent to the client over the network.

But I think that's just asking for security holes, and a given application would probably still have to be written to support it. So it's probably not that good of an idea, and with very few benefits for the SW designer.

In fact, I'm having a hard time conceptualizing a way this could safely be done on a system wide level... probably not impossible though.

just thinking out loud...

[parsec_kadets]

Originally posted by Millennium:
Not likely to be possible.

Not all algorithms benefit from a distributed approach. That's why the apps you've seen have all been written specifically to support distributed computing; they do things which could benefit from it.

Recoding audio is unlikely to benefit much from this approach. The best you could go would be to automatically give each machine a list of files to process, have each one work on those files, and possibly spread the load out so that the lists stay roughly the same length (if one machine is slower than the others, for example).

You're right about not all jobs being able to benefit from multiprocessing, and that each computer could take a file at a time and crunch on that. But, no need to build lists and try to guess how fast each computer could go. You just simply assign once of the computers to be a task master. The other computers come to it and say "give me a task." The task master doles out the data. Then when a task is completed that machine says "give me another task," until the task master says "there are no more tasks for you to do." Under this scheme, since the faster computers complete their tasks faster, they ask for tasks more often, meaning they do more work. That means the work load automatically balances itself out (to a degree, there's nothing to guarantee that the slowest computer won't get that 90 minute Philip Glass song).

[Misanthrope]

Originally posted by drmbb2:
XGrid

I'm fascinated by the fact that this has been highly overlooked.

It's in it's early stages, but goddamn is it promising.

Ciph: TRY THIS. If you can make the small sacrifice of using a CLI app to do the converting, the payoff may just be worth it.

[OwlBoy]

Yeah, it would be cool if APple could make it a library with a nice API. Then Apps could tap into it if they need it.

Just like Ativec.

Something like FFMPEG could use this library.

-Owl

[billybob]

Originally posted by Cipher13:
Not recording audio, processing

He said recoding, not recording. Big difference.

[Bi@tch]

Wouldn't it take almost the same amount of time to pass the data over to the next computer than to process it?

It seems to me that for audio converting this would be somewhat useless.

[entrox]

Unless you have ungodly fast interconnects between your nodes, a naive approach will actually destroy any performance you would gain from distributing tasks. A system-level implementation, which distributes processes or threads at a whim would be actually counter-productive without special precautions in the application since there is a huge difference between transporting data from one memory page to the other and transporting data over Ethernet.
The best way is still doing it by hand, because it isn't particularly difficult to implement. Discovering other nodes with Rendezvous and sending data back and forth is trivial. The real killer is having a design which can be actually distributed.

[kindbud]

^ ditto.

Keep in mind that those extra CPUs are at the end of a narrow dirt road.

If you can deliver a smallish 300KB data file that takes a short time to arrive at the CPU - yet will take a long time for the CPU to process (SETI, for example)...that would work fine.

If you wanted to distribute (via ethernet) 300KB WAV audio files to several CPUs - in order to convert them to MP3 format - it would take longer to transmit the data to the CPU than it takes the CPU to process that WAV into MP3. You would be better off by NOT using distributed computing.

[Millennium]

Originally posted by Cipher13:
Not recording audio, processing it.

Not recording, recoding. Transferring it from one format to another. I should have said converting; forgive my poor word choice.