Welcome to the MacNN Forums.

Graymalkin · Jun 28, 2004, 08:12 AM

Background
I ran across this article about using the open source Blender modeling and renderer with Apple's Xgrid. While this is indeed a very cool use for the technology I've found the particular implementation to be a little less than optimal in some situations. As I found and as is outlined in the article the custom plug-in system described is a heavy burden on not only the cluster's Controller but also the network as a whole.

Xgrid
Xgrid has two basic means to distribute tasks to the Agents in the cluster. The first method is to have the program locally available on each Agent and simply issue shell commands to each Agent telling it to run the program with particular parameters. This method of operation requires very low bandwidth to run but a fairly high level of management on the part of the Agents. The input files and programs must be distributed to each Agent in the cluster in order for them to carry out their assigned tasks.

The second method Xgrid employs is a plug-in based system. The plug-ins are really just application bundles that are sent to each Agent over the network and run in a temporary space with relatively few permissions on the systems. The plug-ins are nice because they are self-contained. The Agents don't need any real management before they're added to the cluster. The plug-in system is great for building ad-hoc or widely distributed clusters. The drawback to this is that in some situations the network overhead can reduce the effectiveness of the cluster. Large plug-ins can take longer to move around the network than the processing they're performing takes. The gains of clustering multiple systems is greatly reduced.

My solution for managed networks
I figured that there must be a lighter means to use Blender in a distributed fashion. In the plug-in method described in the article even rendering a few frames takes enormous amounts of network and disk bandwidth. Every plug-in bundle distributed to Agents is over 2MB in size with only relatively small Blender scene files. With the plug-in the Blender application is shipped to all of the Agents and given its processing instructions.

I set out then to use an Xgrid's XFeed distributed shell to do the job instead of a plug-in. I figured that a managed network the administrator could install any applications they wanted on the systems (lab, office, classroom, etc.). By having Blender on each Agent node the network would only be used to send the commands, distribute the scene files, and save the output files to a central location. Such a system reduces the resources needed by the Controller meaning even an old bondi iMac could do the job well.

My scheme's requirements

A couple Macs you have administrator rights to. I say rights because you shouldn't be doing stuff like this on systems you're not responsible for.
A central file server. Your scene files and output will go here. IMPORTANT: The share you use on the file server needs to be readable and writable to the user "nobody" from each Agent. I've only been able to get this working with NFS and AFS shares, Samba shares end up only accessible to the user that mounted them. I used a Linux based file server running NFS, netatalk, and Samba to test this system out.
Blender which you can download for free at the link above. This process will likely work equally as well with other programs with a little tweaking. I'm using Blender because the command line options are straight forward, I found some good demo files, and it's got excellent control of its output.
Some Blender (.blend) files you want to render out. While you can render still frames there's little point in doing so. I'm focusing on rendering frames of an animation to make into a movie in a distributed fashion.
Quicktime Pro.

Procedure
1. Install Xgrid and Blender onto each of the Macs you want to use in the cluster. I'm using Blender 2.33a for MacOS X and Xgrid Technology Preview 2.
2. Prepare a network share for use by the Agents. This share needs to be readable and writable by the user "nobody" which is an account with few privileges on the system so Xgrid tasks don't expose people to malicious code that could wreck their systems. As I mentioned I found NFS and AFS to work the best, I couldn't get Samba to work out at all.
3. Make a folder on the share called xgrid or something else memorable and inside of it put your Blender scene file. I used lostride which came out of a demo file I found on a Blender site. I take no credit at all for this file. All I did to it was tweak it to fit in with the rendering I wanted to do.
4. In Blender I changed the render settings to output a .png file with the extension added. I also set the output resolution to NTSC (720x480) and 30 frames per second. You can use any output format you'd like as long as each frame is rendered as an image files and not a movie file.
5. Make sure Blender and your network share have the same paths on all of your Agents. I've got Blender in /Applications/Blender for instance. Each user on the file server ought to be logged into the same account so each has equal access to the share. Any Agent without proper access is not going to be working correctly.
6. Once each Agent's got the network share mounted and Blender installed you're almost done. Start up an Xgrid Agent on each of the systems and a Controller on one of them. The particular settings and the usage of a password is entirely up to you. On my little demo run I didn't bother with a password and I set the Xgrid Agent to always run.
7. Open up the Xgrid application and select the Controller you started from the list. If you've never used Xgrid before play around with it a bit to learn its ins and outs. You're going to want to make an XFeed task. Make the task window look a bit like this:

Command: this is the command line path to the Blender application. This might need a little tweaking on your systems. Blender needs to be in the same spot on all of your Agents!
Argument 1: this tells Blender to run in the background i.e. run without its GUI enabled. This is basically a render-only mode.
Argument 2: this is the absolute pathname to your scene file. The networkshare will be whatever the name of your share is. Don't have spaces in the path because it will throw a monkey wrench in the works.
Argument 3: this tells Blender to render a frame on the scene.
Argument 4: this tells Blender what frame to render.

Blender will render the output to wherever the scene file resides (which is why "nobody" needs write access to the share). You can also set a relative location for the output to go inside of the scene file in the render settings. In the lostride example the output will appear in a folder named lostride in the same folder as the scene file. The output images are numbered 0001 through whatever frame you chose as the endpoint. If your choose some huge number that is higher than the number of frames the scene described Blender will probably throw up an error on you. I haven't tried it so I'm not positive.

Hopefully this process has gone smoothly, if it has you'll end up with a folder full of PNG files. You might be a little disappointed at this point since now you've got a bunch of frames of animation but no animation. This is where Quicktime comes in. Open up Quicktime and select File -> Open Image Sequence... and then browse to your network share where the output images are. Select the first image and Quicktime will do the rest. Quicktime will build the sequence into an image sequence movie. Save the movie to your local disk and enjoy the fruits of your labor! Once you've got your uncompressed movie you can export it into any format Quicktime supports which means you can use it in any application that supports Quicktime files like iMovie, Final Cut Pro/Express, or even just export it as a stand-alone animation.

While my system requires a bit more setup to make work it is a little friendlier on your network and cluster Controller. This is an open source Xgrid clone for Unix systems. It allows PCs running Linux or one of the BSDs to participate in Xgrid clusters. It is still in beta and needs a bit of work but the concept is sound. Apple did a nice job with Xgrid in that the architecture is open and anyone with a bit of effort can implement their own clients for the technology. Using a shell command to control Agents allows you to easily manage Agents running entirely different processor architectures and operating systems. If you've got some Linux Agents with Blender installed it is pretty easy to add them to your Blender Renderer. Make a symlink on your Linux and OSX systems in /usr/local/bin to wherever the real binary resides. That way you can give one command (/usr/local/bin/blender) and the Linux and OSX systems will work equally well. The Linux Xgrid system is still experimental and may or may not work properly but that doesn't have to stop anyone from building an OSX-only Xgrid cluster right now.

Here is a short animation I made using a small Xgrid cluster (two Macs), Blender, and Quicktime Pro.

Krypton · Jun 28, 2004, 03:21 PM

I won't pretend to have read your thesis... I mean post, but the animation was cool

suthercd · Jun 28, 2004, 03:41 PM

Would you compare the render time on one CPU vs. the two? Nice description of your process, thank you.

Craig

Graymalkin · Jun 28, 2004, 05:20 PM

Rendering with two nodes was almost twice as fast as rendering with one node. In this system the performance in CPU bound so performance scales linearly with each processor added to the cluster. This system will likely scale to as many systems as your Controller can control and file server can keep up with. A high throughput file server and fast network could allow for humongous speed increases over rendering with a single system. The performance gain of 3+ nodes in the cluster is theoretical at this point because I only had two systems available for testing.

I would guesstimate that it would take about four of my 867MHz Powerbooks to equal the speed of a single 2.5GHz G5. I figure that the G5 is about 25% better at rendering on Blender than my G4 due to the better floating point pipeline. The processor is also 2.8 times faster than mine. However Blender only launches a single thread so it won't receive any sort of gain on a DP system unless two independent processes are launched. I doubt Xgrid will launch multiple processes on DP systems.

The losteride scene I linked to takes my Powerbook about ten seconds per frame to render. At 30fps I'm looking at about 150 minutes to render a 30 second video clip. With a second 867MHz processor working on the job I could probably shave that down to 82 minutes. Three processors would only take about 45 minutes. A lab of 30 eMacs could probably chug through the 30 second animation in about 6 minutes under ideal circumstances. That lab could also render an hour long animation is about 12 hours.

The effort involved in building a makeshift render farm is relatively small compared to the increased productivity you could get out of it in my estimation. I think this solution can be good for managed environments (labs, home networks, offices) since it doesn't require heavy lifting on the part of the network or controller. Small tasks can be efficiently farmed out to the Xgrid cluster. Rendering ten frames of animation using a plug-in would take several times longer than using this XFeed method simply due to network overhead.

Sorry about the length of these posts but I wanted to make sure people could follow my procedure from start to finish and duplicate my results. I wanted to show people that with a couple of free tools they could produce some pretty awesome results without too much hassle.