Thanks for the reply bergy. I figured it out though (after some time). Panther Help is great! Probably not the most gracefil way to go about things, but it worked for me.
1.) Delete the AddressBook folder in User\Library\Application Support - since I had already messed with Address Book on the Panther install, it created these files.
2.) From the backup of my Home folder, copy over the AddressBook folder. Same location in the source and target.
3.) Bam, everything including pix and groups are where I want them. Whew!
As for Mail, I didn't copy everything over like above:
1.) Imported Mail(boxes) from within Mail. Just pointed it to the backup.
2.) This created an Import folder. I had to mess around in the Finder and rearrange the mbox files, etc.--move them out of the Import folder that got created in Mail. Also, since I have a lot of sub-inboxes/folders for different kinds of mail, I had to place these ones in the right place. Not as difficult as it sounds, I just compared what the setup was like on my backup.
Again, not very graceful, but it worked. And surprisingly no screwy permissions. Hooray!