Yeah, that's going to be a pain. I also want to "downgrade". I have the install discs, but the hard part is extracting iMovie 2 off them. And nobody is going to have a mirror of the software, since it was a paid product.
For me, it's not the speed, but the stability. iMovie 3 is much more unstable.
iMovie 2 to me felt like a semi-fine-tuned product. Not great but good enough.
iMovie 3 feels like a product with a lot potential (eg. iLife integration, chapter markers, Ken Burns), but one that is in VERY early beta. Very unstable.