Just a guess, but in the Get Info window there’s the Open With… setting (which allows you to change the default Application to use to open that file). There’s also an option to Change All, which changes the default app for that filetype. It sounds like you might have, at some time, set that option to Appleworks for a .dmg file, and hit the Change All button. That could cause the symptoms you describe.
At any rate, under Jaguar, it’s either Disk Utility or Disk Copy (or something like that) which you can use to mount .dmg files. Find out which has the option to mount a disk image and try mounting one of these files from there. If it mounts properly, then go into Get Info (in the Finder) and set the Open With to that app, and chose the Change All option.
Best of luck.