I'm no good for a deep technical explanation, but I may be able to give a bit of insight:
Their claim of "not using pixels" is a little misleading... it all comes back to pixels at some point. What they're trying to get at is that what you're doing is not so much painting as it is warping the coordinate space the original pixels come from. As you play with their "brushes", you're essentially building a complicated function that says "the pixel at (x,y) should move to (x', y')"... except instead of translating or flipping every (x,y) by the same amount, the effects of this function vary with x and y.
You can play with this kind of math using products like Pacific Tech's Graphing Calculator (see
here) or Wolfram's Mathematica, among others. Googling "nonlinear coordinate transformation" or similar might help you find a good discussion of the subject.
Also, since Kai's Power GOO is pretty ancient, you can find much the same features (sans acid-trippy UI) in Photoshop 7 and newer (under Filter > Liquify).