I've not got much idea myself on how to do this, but you can certainly write a file system driver (at the BSD level) which would then work not only in the Finder, but in all other applications' open/save dialogues. In this case, the FS driver would not read/write a disk as such, but read/write a database.