ok i had that problem too, but it magically got fixed, i did nothing about it.
also i noticed that my sleep/wake times got faster after remote access slowness got fixed.
try your web sharing/file sharing/apple talk settings.... disable all of those and zap your pram, might fix it.