#1: Rogue Amoeba's Detour is exactly for routing specific apps to specific outputs.
However, it will only work if the optical and analog outputs are considered separate outputs. If they are designed to always output the same thing, then you can't route to them separately and you'll need to get some kind of secondary audio device.
As for #2, I highly doubt it. (I would love this, too.)
tooki