I had the same problem, but found help in a thread at the Apple support forum.
I had a circus ponies plug-in somewhere, and removing gave me rock solid stability in address book.
In your case this is could be the problem:
2. The personAtNumber crash
This one is usually triggered by an incoming phone call or SMS.
If you see a phrase like "personAtNumber" in the crashing thread of your crash report, determine who was just calling you and from what number. Then search your Address Book for that number and make sure it is in only one card. If the phone number appears in multiple cards, it can cause a crash.
The full thread :
http://discussions.info.apple.com/We....6@.599bfb98/0