Have you tried Goo-gone? If it is a residue, goo gone will remove it. I swear that stuff can take paint off of a car...
edit: sorry, didn't see that you had already tried it. The InvisibleShield glue probably seeped into the anodizing on the handrests, which you won't be able to get out.