Movie are compressed in a way that implies that they are played back forwards. To decode one frame the software needs the frame before since only the differences to that frame are recorded. Every second or so a frame is fully encoded, so that errors in decoding don't propagate eternally. This frame is kalled keyframe. Now if you play the movie backwards the player does not yet have the frame that comes before the current one (it's not yet shown since it's playing backwards) so the player has to decode all frames from the previous keyframe to that frame. Of course that takes much more time than decoding just one, the next, frame when playing forwards.
If you need to loop back and forth you need to save the movie with a codec that does not encode one frame based on the previous one (I think Animation is such a codec), save using a keyframe per frame (both methods don't compress very well though), or prerender the backwards part and only loop forwards.