Mail.app will do this when mailboxes get somewhere in the neighborhood of 700megs to 1 gig.
Only fix I've found is to download the demo of Mailsmith from
www.barebones.com, import the too big .mbox file to it, and export it as a group of smaller mboxes. Dividing them up into timespans (by year, or group of months, depending on the need) works well.
Hopefully this bad behavior is fixed in Tiger mail, but I haven't tested that yet.