Your system got confused and thinks it's sending audio to the optical digital out (where the volume control doesn't work).
Try sticking in and removing the headphone plug a couple of times.
If that doesn't help, and fiddling with SystemPreferences --> Sound doesn't fix it, open /Applications/Audio-MIDI Setup and change the output back to analog.