[Warhaven]

My brother and I were talking about better ways for deploying images, updates, etc. As it is with NetRestore, Norton Ghost, and similar programs, you have a central server which deploys a single image to every client. For example, if you have a 6 GB image that you wish to deploy to 20 computers, your server has to dish out 120 GB of data. So, I thought, "Why not use the AJAX ideology and let the clients do the work? They're just going to sit there doing nothing anyway, except getting imaged. Let them do the work, and have the server do something more productive."

So, we thought this up:



Here's how it works:

You have a 6 GB image, and for sake of ease, a cluster of six clients who need the image. The server is told to expect six clients, and therefore chops up the image into 1 GB chunks. It then serves each client a 1 GB portion of the 6 GB image. Once all clients have reported that they've received and verified their 1GB chunk, the server is now free to perform other tasks (including serving the pieces to another cluster of 20 computers).

The cluster of clients now pass their bits amongst themselves, a la BitTorrent (peer-to-peer).

Now, at my school, each computer lab has its own switch, which then bridges the computer lab to the broader network. If I could do the aforementioned for each lab, then each lab could image it self independently within its own lan. It would be like giving every lab its own dedicated imaging server ... without the bottleneck of a single NIC pushing out the image.

Pretty sweet, eh? In the same time it would take you dish out an image to 20 computers using todays programs, you could dish out the same image to 400 computers.

[Warren Pease]

Sounds like overkill to me. Bit-torrent does this already (as you mention) and I don't see the need for an extra layer (what you propose) on top of it.

School networks have very fast networks and if you are sharing the image on many (6 x the number of labs) computers, the image should be passed around very quickly and in smaller chunks, so congestion may be less than pushing 1GB blocks.

Despite my experiences with Bit-torrent over the internet (very slow) I think that in this case, it's best to try to leverage existing solutions.

[mduell]

Where's the AJAX? Seems like you just threw that in to use a buzzword.

Why only put 1GB on each client? 6GB isn't very big these days, so why not put the entire image on each client?

Now you can conventionally netboot a lightweight (Linux?) environment with a bittorrent client, use bittorrent to download the image, and use dd or similar to write it all to disk.

[SirCastor]

Originally Posted by mduell 

Where's the AJAX? Seems like you just threw that in to use a buzzword.

I think the use of AJAX is more of a concept rather than an application. I think Warhaven means that the client machines should be sending out the requests for what they need rather than the server simply dishing out everything to everyone. A more appropriate term would have been (what was later used) Torrenting...

[Warhaven]

Originally Posted by mduell 

Why only put 1GB on each client? 6GB isn't very big these days, so why not put the entire image on each client?

That was just an example, since the example was only using six clients. In short, I'm only sending 6 GB of data as opposed to 36 GB of data over the network. Our images average about 20-30 GB when all is said in done after you've added in Office, Adobe Suite, Painter, Maya, 3DSMax, etc. Let me break down some of the math, as to why I was suggesting this, and to not only how long your server would be occupied, but to how much data is pushed over the general network:

30 GB image @ 300 computers = 9,216,000 MB of data to push out.

Under the HUGE assumption that you're doing this over a gigabit LAN, and that every single client has a gigabit NIC, it would take your server about 20 hours to push out the image to all 300 computers.

Let's put those 300 computers into ten 30-computer clusters with their own network (switch) bridge to the overall network, which incidentally acts as a local LAN for each of those clusters.

Each cluster as a whole would have only one image served to it. The server now only has to dish out that same 30 GB image ONCE to each cluster, not to every single client. 30 GB image @ 10 clusters = 307,200 MB of data to push out.

That'd be about 41 minutes of downtime for your server. Lowering your downtime by a factor of over 20 seems pretty advantageous to me, wouldn't you think?

Now you can conventionally netboot a lightweight (Linux?) environment with a bittorrent client, use bittorrent to download the image, and use dd or similar to write it all to disk.

That's sort of what I'm getting at. If you're familiar with NetRestore, it boots a striped down version of OS X loaded with ASR and such. This is where the AJAX ideology (not actually using javascript or anything, I'm just talking about the dogma behind AJAX) comes into play. The client in a cluster only request their tiny portion of the image from the server, instead of the entire image. Following that, each of the clients in a cluster would swap their pieces with each other until all the clients have their completed image. Each cluster would not only be autonomous, but the network traffic involved in the imaging process would be contained within each of the cluster's LAN. That is, you could image any number of computers on a campus network without causing a hit to the rest of the campus.

[olePigeon]

That's 20 hours assuming absolutely perfect conditions. It's probably be closer to 30 hours because no computer is going to be pushing gigabit speeds 100% of the time. You're going to have bottlenecks, interference (packet collisions, people using the network for other reasons, etc.).

Also, when not in the upper levels of education (i.e. college) where you also get lots of private funding, public K-12 schools are in a huge crunch. My school, for example, is in one of the more higher income areas of the Bay Area. Our LAN is only 10/100, some of it only 10 mbit. We rely on a lot of donated equipment from the PTA.

[olePigeon]

You could also just make a network appliance so you wouldn't have to change how the server and/or software operates (if you've invested thousands of dollars in LANdesk, Altiris, Norton Ghost, etc.) The appliance would act as a dummy client for your imaging system, it'd receive the image first. The appliance would then chop it up and send it out to the intended clients.

[Warhaven]

Oh, wait. N/m. It just donned on me that I'm comparing it to unicasting. Multicasting would be just as fast, if not faster.

Oh well. Still, I wonder how a peer-to-peer deployment would stack up against multicasting. At least you wouldn't have all the packet saturation on the network.

[olePigeon]

Apparently LANdesk does both Multicast and Peer Download. Altiris can do Multicast with some additional setup.

Good enough idea that they're already using it.