[kman42]

The following article is a great read:
http://www.macslash.com/articles/01/12/17/2126217.shtml

It brings up some major flaws in HFS and presents some niceties of the BeOS file system.

Does anyone know if there are any plans to replace HFS+ with something a bit more robust? The author also mentions that MS is moving Windows on to a MySQL database for metadata support. Since Apple originally ended the file extension fiasco of Windows and introduced metadata to the rest of us, I find it a little disconcerting that they are going back to file extensions. I realize this bit has been warmed over many times, so I am mainly curious about the file system and hopes for fixing it in the future.

Can HFS+ just be replaced by a better file system? I know you can choose UFS. Is the file system a plugin architecture that could be fixed with relatively little pain to the user? Is it Classic that is holding us back?

thanks,
kman

[starfleetX]

Compatability.

If you throw away HFS+, you throw away Classic and any Carbon apps that use the resource fork.

[TheBum]

Originally posted by starfleetX:
<STRONG>Compatability.

If you throw away HFS+, you throw away Classic and any Carbon apps that use the resource fork.</STRONG>

You also throw away any apps that depend on case insensitivity. For instance, under HFS+, if an application looks for myfile.dat, it will find MYFILE.DAT. Under UFS, it won't. Not that it's necessarily a bad thing, but it can be confusing to the average computer user.

[ 12-18-2001: Message edited by: TheBum ]

[xi_hyperon]

MS is going to use metadata now???? To quote my favorite movie, it seems that fate is not without a sense of irony.

[Nonsuch]

Originally posted by xi_hyperon:
<STRONG>MS is going to use metadata now???? To quote my favorite movie, it seems that fate is not without a sense of irony.</STRONG>

Given how many Windows users are still using older versions, and given how conservative most businesses are when it comes to change, I imagine it will take the better part of a decade for MS to migrate the Windows platform off dependence on extensions and onto a metadata-rich database. I think it's way too early to cede this race to Microsoft.

[boots]

http://www.mit.edu/people/wsanchez/papers/USENIX_2000/

A pretty good read that explains a lot of the difficulties associated with creating file system for the beast we call OS X.

Certainly gives a new appreciation for the size of the task it was to marry unix with the Mac GUI.

[Millennium]

One: it won't be a MySQL database that Windows uses; they're looking at their own SQL product.

Second, concerning robustness, I believe it's being worked on. They're working on a feature called softupdates, which, as I understand it, could add a journaling-like feature (though somewhat more advanced) to the filesystem without having to actually change the filesystem itself, so you could convert nondestructively. This would be Very Cool.

Metadata-wise, HFS+ can actually do most of the things that BeFS can, including extensible file attributes. There are two major things holding it back, though. First, Apple hasn't provided OS-level software to do it; nothing like a query language or easy calls to add attributes. This is one that they really need to do.

The second problem is more low-level. HFS+ is, technically speaking, database-like; perhaps even moreso than Be's file system. However, the structures that they use to actually store the data are very crude, compared to Be's. As a consequence, I/O is much slower, and can't be accessed by more than one process at a time. This is where BeFS' biggest performance gains come from; not so much the data and metadata, but rather the way in which they are stored.

BeFS wouldn't make a suitable replacement for HFS+, in the end, because of one critical HFS+ feature: filenames are just another piece of metadata in the HFS family. The real filename identifier, for HFS and related filesystems, is a number that the computer generates, though you never see it. The filename can change, but the number does not. This is why aliases do not break when you change the file's name or location, and I'm sure you'll agree that this one is absolutely critical. Be considered adding this to BeFS, but for reasons I'll never be able to fathom, decided againt it.

Perhaps, though, a new filesystem, keeping with the HFS+ advances but updating it with BeFS-like technology, could be devised. HFS-3 or HFS-NG, perhaps?

[lgerbarg]

Originally posted by Millennium:
<STRONG>One: it won't be a MySQL database that Windows uses; they're looking at their own SQL product.

Second, concerning robustness, I believe it's being worked on. They're working on a feature called softupdates, which, as I understand it, could add a journaling-like feature (though somewhat more advanced) to the filesystem without having to actually change the filesystem itself, so you could convert nondestructively. This would be Very Cool.

Metadata-wise, HFS+ can actually do most of the things that BeFS can, including extensible file attributes. There are two major things holding it back, though. First, Apple hasn't provided OS-level software to do it; nothing like a query language or easy calls to add attributes. This is one that they really need to do.

The second problem is more low-level. HFS+ is, technically speaking, database-like; perhaps even moreso than Be's file system. However, the structures that they use to actually store the data are very crude, compared to Be's. As a consequence, I/O is much slower, and can't be accessed by more than one process at a time. This is where BeFS' biggest performance gains come from; not so much the data and metadata, but rather the way in which they are stored.

BeFS wouldn't make a suitable replacement for HFS+, in the end, because of one critical HFS+ feature: filenames are just another piece of metadata in the HFS family. The real filename identifier, for HFS and related filesystems, is a number that the computer generates, though you never see it. The filename can change, but the number does not. This is why aliases do not break when you change the file's name or location, and I'm sure you'll agree that this one is absolutely critical. Be considered adding this to BeFS, but for reasons I'll never be able to fathom, decided againt it.

Perhaps, though, a new filesystem, keeping with the HFS+ advances but updating it with BeFS-like technology, could be devised. HFS-3 or HFS-NG, perhaps?</STRONG>

The article makes a few good points, but has some technical inaccuracies. Particularly, the issues involving multithreaded file access. HFS+ is singled-threaded with respect to B-tree access (which means things like file name lookups, etc). BeFS Is also single threaded with respect to its B-Tree entries, but it does a BTree per directory, which cuts down the contention (though makes it so that you have to do a lot more access to get stuff). Also, BeFS's journal is entirely serialized and single threaded, which means all metadata alterations bottleneck through it. To be fair, HFS does not have journalling, and most journalled filesystems serialize their journals (I belive the notable exception is XFS, which uses multiple timestamped journals in order to allow for concurrent journal access).

Also, OS X has a unified buffer cache, whereas BeOS does not. That is a significant boost for all file systems on Mac OS X over BeOS. Its also one of the things that Dominic Giampaolo seems to think is a serious performance issues for BeFS (he's the guy behind BFS, he wrote a very interesting book, "Practical File System Design with the Be File System").

On another sidenote, the softupdates stuff is more of something that some of us are doing in our spare time. It will most likely never be useful on HFS+, only UFS. Softupdates depends on the fact that you can define a stable serialization of data changes, and that you can calculate that at runtime. Unfortunately it is hideously complex to do that for B-trees, and when I talked to McKusick (who did the original implementation of softupdates for FreeBSD) he said that as far as he knew nobody had come up with a method that worked in all cases for B-trees. Since it must work in all cases or it is no good that is a big problem (there are some potential hideous workarounds I can think of, but I am not even sure they would work). Even if there is a solution the computations may be so significant that they end up slowing down the I/O, and they will almost certainly take massive amounts of RAM. B-tree based file systems have lots of benefits, but pervasive large b-trees do not play friendly with softupdates.

Not to be the bearer of bad news. I am a big, big fan of softupdates. I think they are better than journals, but they are not really amenable to HFS+ style filesystems.

As far as filenames being metadata, that is always true to some extent or another. On UFS the file is identified by a number (inode). Names point to inodes. The issue is that inode policy and reuse are different than how HFS+ works, but the basic concept is the same (On HFS+ there is no indirection, which makes things like hardlinks an issue).

Louis

[xi_hyperon]

Originally posted by Nonsuch:
<STRONG>

Given how many Windows users are still using older versions, and given how conservative most businesses are when it comes to change, I imagine it will take the better part of a decade for MS to migrate the Windows platform off dependence on extensions and onto a metadata-rich database. I think it's way too early to cede this race to Microsoft.</STRONG>

True, but the irony lies in the fact that MS is even going in this direction, while Apple is now going the opposite way. How long it takes them to make the transition is irrelevant to my point.

[kman42]

My central question remains: is there hope for the filesystem? I'm guessing that Louis can't give us much more info, but it sounds like there isn't a real concerted effort being made to move the filesystem forward. I guess this is probably a back-burner issue as Classic will prevent them from doing anything too dramatic any time soon. That being said, I think Classic will die a quicker death than most of us think. I'm already Classic-free at home and I only run Classic on my portable so I can do a few Photoshop things and run MacVector. I'm guessing that a year from now we might see Classic as an optional install on new machines (the CDs in the box, but not installed on the machine). Native apps really seem to be gaining momentum and I can imagine a glorious Classic-free world on many computers in a year's time. But did someone say that Carbon is also a problem...

kman

[boots]

I would have to agree. OS 9 (and classic) will be goner more sooner than later.

I bitched when word 6. supplanted word 5.x as the standard, then rejoiced when word 98 saved us from word 6. Not that word 98 was any where near as functioanl for *my* useas word 5.x....

When you think about it, the turn around on a major software product is only a couple of years. Two years per generation, and soon our old stuff doesn't even run on the hardware you need just to function at a normal pace. I have a couple of things from my pre-G3 (which wasn't really that long ago) that I really miss 'cause it won't run under the g3 architecture. OS 9 can probably be phased out in 2-4 years. G5 will probably still run legacy software (pre-os X) but G6 (whatever that will be) probably won't...that's only about 4 years or so. And it won't run the stuff 'cause it doesn't won't need to.

That's the problem with technology today...its obsolete before you take it out of the box.

[ 12-18-2001: Message edited by: boots ]

[cpt kangarooski]

starfleet--
Support of forks is NOT a factor. MacOS apps run great under FAT32, if properly handled. They have for about a decade. PCExchange isn't a difficult mechanism to extend to other filesystems.

Frankly, I'd like nothing better than to see Apple develop a new filesystem that was opened up and public so as to foster adoption, based on the strengths of XFS, NTFS, BFS and HFS+. I think they could do fairly well, really. Provided that they really tried to do something top notch, as opposed to crippling it, as they're doing with the metadata support at present.

[Hornet]

"it seems that fate is not without a sense of irony"

Similar to MGS1, where snake says "Seems that God is not without a sense of irony"

[moki]

HFS+ really isn't a "crappy" file system. About the only thing I'd really like to see is some form of journaling, which certainly is possible.

[kman42]

I don't really know how crappy it is. I just used to word to get people to participate. Nothing like a little inflamatory speech to get people involved.

kman

[sadie]

Can anyone explain (or point me to a good description) of journalling? It's a much-touted feature, but i can't find any pages that really explain what it is. And what are soft updates, and why are they better?

[dont.wanna.tell]

Salve,

Sorry I do not know any good source of information about journaling or softupdates (I even dont know what softupdates really are...

But I will try to explain what journaling means to me (hope its right, if not, please correct me.

So as far as I know Journaling means that whenever you try to write something to the hard drive you first write some sort of notification of it in a log file, and do another notification after you finished writing.

That way if something goes wrong, and the system crashes, it does know exactly whats wrong on the harddrive and where the errors lie. (no more Norton and Diskwarrior. ... {Thats probably the reason why we dont have it. }

So This is really a cute feature if you do like to ceep your data... &lt;gg&gt;

cu martin

[Jelle Monkmater]

Originally posted by sadie:
<STRONG>Can anyone explain (or point me to a good description) of journalling? It's a much-touted feature, but i can't find any pages that really explain what it is. And what are soft updates, and why are they better?</STRONG>

I found this one which, although a little propagandistic, is true enough: http://www-106.ibm.com/developerworks/library/jfs.html

[dfiler]

Originally posted by xi_hyperon:
<STRONG>

True, but the irony lies in the fact that MS is even going in this direction, while Apple is now going the opposite way. How long it takes them to make the transition is irrelevant to my point.</STRONG>

How long it takes doesn't matter? You should read up on some more history.

MS has had a multiforked file system for around 8 years now. They just haven't got around to making use of it. In my opinion, they never will finish and/or publisize the API for this functionality. Its far more likely that NTFS will be scrapped in favor of something else.

Microsoft actually does spend a lot on R&D. Just be aware that much of the work is never fully realized.

Mackido has a bit on this: http://www.mackido.com/Innovation/FileForks.html

[cpt kangarooski]

Well they never use it, but it's there, it's supported, and you can make multistreamed files anytime you feel like it. (although there's no built in method to support transferring it to FAT32 intact)

I'd love to see developers make use of the damn thing.

WRT HFS+, IIRC, it has a lousy sort of hack for dealing with file sizes over 4GB... it's not a native 64bit fs, or so I've heard. Although at this rate, I wonder if a 128bit fs would just be a better idea. Certainly Apple's on the verge of significant trouble with the number of bits allocated to system memory. I wonder if Connectix will write MODE64?

[dogzilla]

Originally posted by Millennium:
<STRONG>Perhaps, though, a new filesystem, keeping with the HFS+ advances but updating it with BeFS-like technology, could be devised. HFS-3 or HFS-NG, perhaps?</STRONG>

[Incr HFS]

[sadie]

Originally posted by Millennium:
<STRONG>The real filename identifier, for HFS and related filesystems, is a number that the computer generates, though you never see it. The filename can change, but the number does not. This is why aliases do not break when you change the file's name or location, and I'm sure you'll agree that this one is absolutely critical. Be considered adding this to BeFS, but for reasons I'll never be able to fathom, decided againt it.</STRONG>

Can anyone fathom why the rest of the computing industry doesn't recognise the huge difference this feature makes? In my programming, i've learned to ALWAYS provide long-term unique identifiers to things.

[cpt kangarooski]

Oh, that's easy, Sadie.

Long distance powerlines serve as receiving antennas for bogons eminating from the Sun. Mac power supplies tend to filter them out; cheaper PSU's used by most of the remainder of the industry let them pass, however, and this influences crappy design decisions on the part of MS, etc. It has something to do with the capacitors used (same reason why Macs are marginally less susceptable to brownouts than Wintels) from what I understand. The Woz was never much of an analog guy -- his early PSUs didn't always hold up well, but Rod Holt, who designed the very nice Apple II PSU, as well as some of the RF interface, was very good, and his influence is still felt.

Industrial-strength PSU's used in enterprise hardware have minimized this upon a small portion of the industry, but unfortunately, most of those guys don't have any impact on the microcomputer field. (e.g. Seymour Cray, who designed the distinctive Cray XMP also had heavy-duty bogon filters installed throughout installations)