[int69h]

I'm making pretty good headway porting IBM's JFS to OSX/Darwin and possibly FreeBSD. Right now I'm concentrating on the userland applications as I'll need a JFS volume to test a kernel module on. I realize Apple will never include it in the base distribution because of that "pesky" GPL. If Apple won't provide us with a reliable filesystem, a little hacking and IBM's goodwill will. I had originally considered porting the filesystem from AtheOS, but working on a moving target that deviates from the UNIX way of doing things didn't seem to be a good use of my limited time. Perhaps I will be able to do it in the future. Anyways, I'll be needing competent testers in the future, and if y'all are interested, I can post status updates.

[int69h]

I've managed to get the userland applications ported(I can create filesystems and fsck them anyways), and will turn my attention towards a kernel extension tomorrow.

localhost:~/Projects/jfsutils/output] root# uname -a
Darwin localhost 1.4 Darwin Kernel Version 1.4: Sun Aug 12 02:40:52 PDT 2001; root:xnu/xnu-177.1.obj~2/RELEASE_PPC Power Macintosh powerpc
[localhost:~/Projects/jfsutils/output] root# ./mkfs.jfs /dev/disk0s4
mkfs.jfs development version: $Name: $

Warning! All data on device /dev/disk0s4 will be lost!

Continue? (Y/N) y
/

Format completed successfully.

4326508 kilobytes total disk space.
[localhost:~/Projects/jfsutils/output] root# ./fsck.jfs /dev/disk0s4
sb_ptr = 0x5a0b0 agg_recptr = 0x55070 bmap_recptr = 0x5a160
In initial_processing
Block size in bytes: 4096
File system size in blocks: 1081627
Phase 1 - Check Blocks, Files/Directories, and Directory Entries.
Phase 2 - Count Links.
Phase 3 - Rescan for Duplicate Blocks and Verify Directory Tree.
Phase 4 - Report Problems.
Phase 5 - Check Connectivity.
Phase 6 - Perform Approved Corrections.
Phase 7 - Verify File/Directory Allocation Maps.
Phase 8 - Verify Disk Allocation Maps.
File system checked READ ONLY.
File system is clean.
[localhost:~/Projects/jfsutils/output] root#

[SJ]

Mate, if you can get a decent file system running for OSX, I will worship you forever!

Keep up the great work.

[Millennium]

How does JFS deal with metadata, anyway?

I've been thinking of trying to work with the AtheOS filesystem myself, though I'd probably have to fork the tree to add in the good stuff from HFS+ (chief among them, unique file ID's so that we can get aliases working).

[int69h]

I really need to study the JFS design, the Darwin VFS code, and Wilfredo Sanchez's USENIX paper before I can give you a positive answer on this. I haven't checked yet, but I assume Apple's technique for storing metadata on a UFS partition is a layer in the VFS stack. If this is the case, I can ignore JFS's metadata facilities and simply use Apple's methods for handling metadata. This isn't as bad as it sounds because if I'm understanding "How the Journaled File System handles the on-disk layout" correctly, metadata isn't journaled anyways. This also allows metadata to be archived without rewriting several more utilities.

I'm thinking that in order to keep divergence from the JFS for Linux codebase to a minimum, I will write wrapper functions around the required BSD functions that expose the required formal declaration and parameters that the JFS for Linux code expects. Anyone have any arguments for or against this decision?

Porting the userland was only about 2% to 5% of the work. Now comes the other 95% to 98%.

If you decide to go forward w/ AFS, I would recommend a fork. I had a pixmap window decorator that would have went along ways towards fixing the horrible interface in Kurt's GUI about 95% complete and scrapped it because of his reluctance to include any other developers in his pet project. Theo de Raadt of the GNU world indeed.

[SYN]

I'm no developer, but is there a reason for you choosing a port of JFS over XFS? As I understand it, XFS has extensive native support for meta-data, and is highly tuned for I/O performance. It's open-sourced too.

*very* excited to see a journaling filesystem coming to OSX.

Now if only Apple provided Fat32/NTFS drivers for OSX so I could just plug in hard drives to make large file transfers...

[ 09-15-2001: Message edited by: SYN ]

[int69h]

JFS also has native support for metadata; however, I will not be utilizing it. I intend to follow the approach Apple took w/ UFS and store the data in a seperate file. Although this is not as elegant as taking advantage of the native support, it will allow users to use existing tools like tar, stuffit, and zip to archive directories.

I chose JFS over XFS because it looked to be MUCH easier to port. IBM's code is some of the best documented Open Source code I've seen. SGI's is, ummm, like most other OSS code out there.

The one comprehensive benchmark of Linux filesystems I looked at did show XFS trouncing JFS in most every operation. I'm just now getting my feet wet in the Darwin kernel. Hopefully after JFS, I'll have the ability to attempt XFS.

I've registered over at Sourceforge. Hopefully I'll be able to get something posted up there soon.

[lgerbarg]

Originally posted by int69h:
<STRONG>I really need to study the JFS design, the Darwin VFS code, and Wilfredo Sanchez's USENIX paper before I can give you a positive answer on this. I haven't checked yet, but I assume Apple's technique for storing metadata on a UFS partition is a layer in the VFS stack. If this is the case, I can ignore JFS's metadata facilities and simply use Apple's methods for handling metadata. This isn't as bad as it sounds because if I'm understanding "How the Journaled File System handles the on-disk layout" correctly, metadata isn't journaled anyways. This also allows metadata to be archived without rewriting several more utilities.

I'm thinking that in order to keep divergence from the JFS for Linux codebase to a minimum, I will write wrapper functions around the required BSD functions that expose the required formal declaration and parameters that the JFS for Linux code expects. Anyone have any arguments for or against this decision?

Porting the userland was only about 2% to 5% of the work. Now comes the other 95% to 98%.

If you decide to go forward w/ AFS, I would recommend a fork. I had a pixmap window decorator that would have went along ways towards fixing the horrible interface in Kurt's GUI about 95% complete and scrapped it because of his reluctance to include any other developers in his pet project. Theo de Raadt of the GNU world indeed.</STRONG>

Fred's paper is very highlevel, it does not discuss anything specific that would be useful for this.

Just to give you a warning: The userland stuff is way, way, way easier than kernel stuff. When I did the intial import of softupdates it took me three days, and I kept hitting a weird edgecase I gave up on (I didn't have a second machine to debug it with, and gave up on printf style debugging). And that was way easier, the source bases are much closer (I should really get back to that once work slows down).

Metadata and extended attributes/metdata or different beasts. You can choose to just ignore extended attributes, or hook them into the attrlist code your choice, each has its benefits.

Also, just so you know, the *BSD and Linux VFSes are very different. Additionally there subtle differences in the locking semantics of Darwin's VFS and FreeBSDs VFS. You will also need to write new code to cope with the Universal Buffer Cache.

Good luck.

Louis

[int69h]

Reread Fred's paper last night and your quite right. I assume when you were talking about the "different beasts" you were talking about information stored in the inode like permissions, number of hard links, etc. versus the stuff that would be stored in the resource fork of HFS. If so, I understand that perfectly. If not, looks like that's something else I'll have to figure out.

I'm using this as a reason to learn more about Darwin kernel internals. If nothing comes out of the project, I've still gained something.

How are softupdates coming along? If you can manage to get them working, this project will be moot anyways.

I've setup a project over at sourceforge under jfs-osx. No web content yet, because I'm still waiting on my web monkey, but I did at least get some stuff in CVS. I contacted IBM about merging my changes to jfsutils into their JFS for Linux codebase, but unsurprisingly Steve wasn't interested. So rather than having all kinds of crufty preprocessor statements, I've decided to attack their code w/ sed and get rid of all the Linux specific stuff.

Is the FreeBSD VFS documented anywhere? From what I was able to gather from the freebsd-fs archives it isn't. Also does Darwin have an equivalent to OpenBSD's little endian to host byte order conversion routines? If not, I might have just found my first Darwin contribution.

Terry

[lgerbarg]

Originally posted by int69h:
<STRONG>Reread Fred's paper last night and your quite right. I assume when you were talking about the "different beasts" you were talking about information stored in the inode like permissions, number of hard links, etc. versus the stuff that would be stored in the resource fork of HFS. If so, I understand that perfectly. If not, looks like that's something else I'll have to figure out.

I'm using this as a reason to learn more about Darwin kernel internals. If nothing comes out of the project, I've still gained something.

How are softupdates coming along? If you can manage to get them working, this project will be moot anyways.

I've setup a project over at sourceforge under jfs-osx. No web content yet, because I'm still waiting on my web monkey, but I did at least get some stuff in CVS. I contacted IBM about merging my changes to jfsutils into their JFS for Linux codebase, but unsurprisingly Steve wasn't interested. So rather than having all kinds of crufty preprocessor statements, I've decided to attack their code w/ sed and get rid of all the Linux specific stuff.

Is the FreeBSD VFS documented anywhere? From what I was able to gather from the freebsd-fs archives it isn't. Also does Darwin have an equivalent to OpenBSD's little endian to host byte order conversion routines? If not, I might have just found my first Darwin contribution.

Terry</STRONG>

Yes, I did mean the data in the inode versus other extended data, though I think it also is important to note that extended metadata and forks are different. Forks are data, not metadata, the usual signifigance of this has two with extended attributes and metadata having more limited read/write semantics, as well as having atomic write characteristics (I am unsure if that is universally true). You can often fiddle with the semantics to make either one do whatever you need though.

The deal with OS X is we have some extended vfs interface, getattrlist and setattrlist. If you implement those Carbon will interact with the fs one way, if not it will interact with it a different way. I think forks are also accessed through this mechanism, I can't remember. The downside of doing this is that it requires more work, and divergence. The upside is if you do it just right Classic would be able to see it.

Soft updates have not been coming I just have not had the time, though it looks like I might in the near future.

I do not think the VFS in its current form is documented, though a number of books talk about the traditional 4.,4BSD-Lite2 vfs both are derived from.

We do have a whole bunch of endian macros (in OSByte.h maybe?). You should also take a look at what we do with rev_endian_fs. Basicly we write the FS metadata in a fixed manner, then we write the data in whatever the native format of the system that initialized disk did (I think we mark it in the superblock). We have code to transparently byte swap all the data reads if necessary, but usually we don't have to.

Louis