Find out what modem is in your Mac. Then find the AT command list for your modem.
Maybe find an open source modem controller program so you can reuse the code to send messages to the modem. Next step will be sending audio through the mode (wiring it up as an output device then setting your mic as the input device.