Leaving in the 512MB should help things. The additional capacity should more than make up for the increased average latency.
This assumes, of course, that you have two 1GB in the inner two slots of each card, and the 512MB on the outer two slots of each card.