No need to drag the black pointer.. You can just click anywhere on the progress bar, and the song will instantly play from that point.
If you need to scrub back & forth tho, then maybe you are better off playing the file in QT, where you can shuffle back & forth with a scroll wheel equipped mouse.