It probably does work fine in practice. It just seems counterintuitive. If I'm turning a physical page, I swipe right to left to go forwards. And in software, with a hand-grab tool, if I drag it from right to left I expect to see the right hand side of the document - which is where the next page is, not the previous page.
Hang on. You seem to be agreeing with me. "Starting on the left and moving to the right moves the object from the left, to the right." Which should leave behind an object even further to the left - which is going backwards.
Maybe there's just a silly mistake in the video. He definitely says "to page backwards swipe from right to left." Or maybe there's just a mistake in my girlish brain that can't picture abstract manipulations correctly.