So be careful when you do it! Aside from the complexity of opening a Mini (I'm assuming the Intel Mini is as complex to open up as a PPC Mini), it's not somthing that a normally coordinated person should be afraid of. As long as you take your time and proceed carefully, of course!
There's another reason to keep the factory RAM: you can use it to troubleshoot the computer if you run into problems later. Some computer problems come from weird sources, so if you can prove (to yourself) that a problem exists with factory RAM and your upgrade RAM, you've proven it's nothing to do with the RAM.