I tried the hdiutil command with your syntax and also with the syntax listed in the man pages, but in both cases I just got a copy of the original file. The utility also appended a .dmg extension to it (the man page says that it's supposed to append .toast to the end, so this seems odd).
The syntax in the man page is
hdiutil convert imagefile.dmg -format UDTO -o imagefile.iso