The best advice I can give on this topic is "if it's not broken, don't mess with it". In particular clearing VM files is definitely not OK. The contents of those files represent computer memory which you do not want to lose. Where did you get the idea that it was?
Cache files, on the other hand, exist to speed up system performance and they can usually be safely deleted because they'll just get rebuilt anyway. But consider my first rule...
Chris