What I've noticed is that there's always a delay when you access the contextual menu the first time, regardless of where you do it (Finder, third party apps, etc). But the succeeding times don't have this delay. Maybe its the build-up of the contextual menu items?