The tower is probably noticeably faster. The HD is likely MUCH faster -- unless you put a 7200 rpm HD in the PB -- and the tower probably has a better video card, a speed increase that might be lost if you are using a large monitor. If you are doing video work, the extra HD space will also manifest itself as a faster experience over a nearly full PB HD.
The extra RAM will make a difference, too.
I am not sure if there is not some difference in the CPU per se -- tower chips tend to run hotter and have more cache, etc., over their contemporaries in the PBs (but the PB CPU may be a newer chip, not sure from your description).
If you wanted to make the PB closer to the tower in speed, add RAM, an internal 7200 rpm HD and a FW800 external scratch HD.