Welcome to the MacNN Forums.

If this is your first visit, be sure to check out the FAQ by clicking the link above. You may have to register before you can post: click the register link above to proceed. To start viewing messages, select the forum that you want to visit from the selection below.

You are here: MacNN Forums > Software - Troubleshooting and Discussion > macOS > will tiger have a new file system?

will tiger have a new file system?
Thread Tools
Steve Bosell
Mac Enthusiast
Join Date: May 2001
Status: Offline
Reply With Quote
Jan 12, 2005, 01:48 AM
 
I heard the BeFS guy works for apple now, just wondering......
     
Disgruntled Head of C-3PO
Professional Poster
Join Date: Jul 2001
Location: In bits and pieces on Cloud City
Status: Offline
Reply With Quote
Jan 12, 2005, 01:57 AM
 
No, but he worked on Spotlight.
"Curse my metal body, I wasn't fast enough!"
     
larkost
Mac Elite
Join Date: Oct 1999
Location: San Jose, Ca
Status: Offline
Reply With Quote
Jan 12, 2005, 11:27 AM
 
What exactly would you want out of a new FileSystem? The real advantages of BeFS are all matched by Spotlight, and without the high complexity cost that BeFS imposed. Journaled HFS+ takes care of keeping the meta-information intact (and full journaling is simply too expensive). The current HFS+ implementation provides for hot zones and auto-de-fragmentation of the files that need it... And change notification has been implimented in the VFS layer... Other than a network distributed file system (a consumer version of XSan...), I really can't think of anything I would want in a primary FileSystem. (there are a few things I would like in the VFS layers, and one can always want more support for other file Systems... like write support for NTFS, but that is not likely).
     
Millennium
Clinically Insane
Join Date: Nov 1999
Status: Offline
Reply With Quote
Jan 12, 2005, 12:07 PM
 
Originally posted by larkost:
What exactly would you want out of a new FileSystem? The real advantages of BeFS are all matched by Spotlight, and without the high complexity cost that BeFS imposed.
Actually, that's not quite true. There's one major advantage of BeFS which HFS+ as we know it will never be able to match, because it's fundamental to the filesystem's structure. And no, this isn't a "we should keep everything in a database" rant.

The problem is how HFS+ keeps track of its files. This goes back to the MFS days, before multitasking or even multithreading were common in consumer operating systems. The design decisions they made then were optimized to ensure that a file could be looked up and opened as quickly as possible. It worked well, and to this day nobody beats HFS+ when it comes to this particular operation's speed.

However, that design decision had consequences. In particular, it forces the filesystem to be accessed in a single-threaded (and, by extension, single-tasked) manner. This is a major performance bottleneck, particularly for applications which work with many small files. While OSX is the fastest filesystem around at opening single files, it's the slowest filesystem in common use when dealing with multiple files. Consider that OSX bundles do precisely this, and you get a performance hit on every application as it is opened. It also hurts "project"-style applications, where many files are worked on simultaneously.

This is, of course, not something that many users will deal with on a regular basis. However, it is worth noting, because it cannot be fixed without replacing the filesystem itself. Apple has done a wonderful job at re-creating the advantages of many other filesystems, but this is one that they're not going to be able to work around.
You are in Soviet Russia. It is dark. Grue is likely to be eaten by YOU!
     
Janmansilver
Fresh-Faced Recruit
Join Date: May 2004
Location: Denmark
Status: Offline
Reply With Quote
Jan 15, 2005, 08:57 AM
 
So whats Apple going to do about it? To me it sounds as a serious issue - they have to come up with something - any ideas of what they are up to? Or what they should do?
     
Chinasaur
Senior User
Join Date: Dec 2002
Location: Out West Somewhere....
Status: Offline
Reply With Quote
Jan 15, 2005, 01:36 PM
 
Millenium,

Thanks for the info.

They changed so much with OS X..why didn't they change the FS?

I would have thought OS X used a *nix style FS.
iMac - Late 2015 iMac, 32GB RAM
MacBook - 2010 MacBook, 1TB SSD, 16GB RAM
     
SpaceMonkey
Addicted to MacNN
Join Date: Oct 2002
Location: Washington, DC
Status: Offline
Reply With Quote
Jan 15, 2005, 03:58 PM
 
Originally posted by Chinasaur:
Millenium,

Thanks for the info.

They changed so much with OS X..why didn't they change the FS?

I would have thought OS X used a *nix style FS.
At least initially, they needed to make sure that Classic (OS 9) would boot up on the same drive partitions that OS X was installed on. I suspect that a change in the file system won't come until after the Macintosh OS moves beyond the "X" moniker, at least.
     
Catfish_Man
Mac Elite
Join Date: Aug 2001
Status: Offline
Reply With Quote
Jan 15, 2005, 03:59 PM
 
Originally posted by Chinasaur:
Millenium,

Thanks for the info.

They changed so much with OS X..why didn't they change the FS?

I would have thought OS X used a *nix style FS.
Many many programs rely on the HFS/HFS+ APIs; any new file system will likely have to support a superset of HFS+'s capabilities so that the existing API can be grafted on to it. Personally, I'd be interested in seeing what ReiserFS 4 could do, as it seems to be designed to be quite extensible and full featured.
     
Millennium
Clinically Insane
Join Date: Nov 1999
Status: Offline
Reply With Quote
Jan 15, 2005, 04:52 PM
 
Originally posted by Chinasaur:
Millenium,

Thanks for the info.

They changed so much with OS X..why didn't they change the FS?
HFS+ has quite a few usability advantages over most Unix-style filesystems, but Apple's initial plans were to drop these entirely and transfer everone over to a Unix-style filesystem over time. The userbase, however, didn't stand for this, and so this plan has gone somewhat by the wayside.

As for why Apple didn't simply do this all at once when OSX was first released, the problem was compatibility. Classic doesn't work with most Unix-style filesystems, and it can't be made to boot from any of them, so they needed to keep HFS+ support for the time being. Also, they realized that users wouldn't be keen on doing a backup/format/reinstall just to upgrade to OSX.
I would have thought OS X used a *nix style FS.
Actually, there is one which is technically supported by OSX: UFS, the filesystem used in NeXTStep (OSX's predecessor) and several other OSX-based operating systems. You can't get to this option from the GUI anymore, though you could with the earliest versions of OSX. Not all apps work well with it, however.
You are in Soviet Russia. It is dark. Grue is likely to be eaten by YOU!
     
larkost
Mac Elite
Join Date: Oct 1999
Location: San Jose, Ca
Status: Offline
Reply With Quote
Jan 17, 2005, 10:44 AM
 
Millennium: I am not sure where you get the idea that HFS+ is slow on multiple file reads. The BTree system that HFS+ requires is only single-threaded when it comes to writing, not reading. And I am not aware of any filesystem implementation that is multi-threaded when it comes to writes (the implementation details of multithreaded writes are nasty).

I don't know if Apple's implementation of reads are multithreaded or not, but there are both sync- and asynchronous reads and write API's available. Earlier versions of MacOS were much more focused on synchronous only.

Oh... and MacOS 6 was multitasking... it was just shared multitasking (since that is a much more efficient use of a processor... but less efficient use of programmer time). So BeOS was nothing new on the multitasking side. It was novel in that it was very multi-threaded... excessively so in some places, and the Be team made some really creative things possible (like making multi-processing work on the 603 series, with no hardware support).

And I am not sure what this has to do with BeFS... since that was single threaded as far as I can remember. And in the second version they moved all of the meta-data out of the filesystem (since it was killing performance) and setup a system similar to what Apple is doing with Spotlight, but with less structure/more extensibility.
     
Millennium
Clinically Insane
Join Date: Nov 1999
Status: Offline
Reply With Quote
Jan 17, 2005, 11:20 AM
 
Originally posted by larkost:
Oh... and MacOS 6 was multitasking... it was just shared multitasking (since that is a much more efficient use of a processor... but less efficient use of programmer time).
You mean cooperative multitasking, I take it?

System 6 had cooperative multitasking as an option called MultiFinder. I remember running it on my souped-up Mac Plus (with a whopping four megabytes of RAM, no less). It worked like a dream, for the time. This option was then permanently integrated into System 7, and did not change much until Mac OS X.
And I am not sure what this has to do with BeFS... since that was single threaded as far as I can remember.
You might want to read Dominic Giampaolo's book on the subject, Practical Filesystem Design with the Be File System. He designed the Be file system, and is now working as one of the big filesystem guys at Apple. He studied HFS at length when creating Be's filesystem, and took inspiration from quite a few things.
And in the second version they moved all of the meta-data out of the filesystem (since it was killing performance) and setup a system similar to what Apple is doing with Spotlight, but with less structure/more extensibility.
I never argued that everything should be stored in a database. Frankly, I don't see the need for such a thing, as long as the metadata is still kept up to date properly. I'm only noting that Apple made a design decision which made sense many years ago but which we would consider unwise today.
You are in Soviet Russia. It is dark. Grue is likely to be eaten by YOU!
     
P
Moderator
Join Date: Apr 2000
Location: Gothenburg, Sweden
Status: Offline
Reply With Quote
Jan 17, 2005, 02:01 PM
 
To be fair: The main problem with OS 9 and above was not the actual multitasking, it was that the toolbox was not reentrant (and that the memory management sucked bigtime, but that's another rant). You can make preemptive threads using the MP libraries, and some programs did (notably MP3-players, IIRC, although the also used every trick in he book and then some to keep the music running smoothly), but those threads couldn't do much. Technically the system was like Windows 95 - preemptive threads possible together with cooperative threads, and a cooperative thread can still crash the entire OS. The difference was what you could do from those preemptive threads - in Win 95, you could use all of the Win32 API. In Mac OS 9, you could only use a very limited set of calls.

While there are certainly better filesystems out there, I don't think Apple is working on a new one now. If they were, they'd want to test it a lot and we'd have heard about it.
     
Angus_D
Addicted to MacNN
Join Date: Mar 2000
Location: London, UK
Status: Offline
Reply With Quote
Jan 17, 2005, 05:21 PM
 
Originally posted by Millennium:
You might want to read Dominic Giampaolo's book on the subject, Practical Filesystem Design with the Be File System. He designed the Be file system, and is now working as one of the big filesystem guys at Apple.
As I understand it he's primarily working on Spotlight. He may of course be touching HFS in doing so.
     
someone_else
Dedicated MacNNer
Join Date: Dec 2001
Location: Promised Land
Status: Offline
Reply With Quote
Jan 17, 2005, 05:59 PM
 
Originally posted by larkost:
And I am not sure what this has to do with BeFS... since that was single threaded as far as I can remember. And in the second version they moved all of the meta-data out of the filesystem (since it was killing performance) and setup a system similar to what Apple is doing with Spotlight, but with less structure/more extensibility. [/B]
Actually, if you read Giampaolo's book Practical Filesystem Design (using BeFS), you will find it's the other way around. The search meta-data was moved INTO the filesystem because storing it separately in a private Tracker DB was a huge speed issue and caused data coherency problems between things like dates stored in the DB and dates in the filesystem.

HFS Spotlight is similar, in that the Spotlight meta-data is stored in the filesystem -- from what I've heard they are using HFS+'s long unused ability to have > 2 forks associated with a file.
G5 2.5 DP/2GB RAM/NVidia 6800 Ultra
PowerBook Al 1Ghz/768MB RAM
6gb Blue iPod Mini
     
Catfish_Man
Mac Elite
Join Date: Aug 2001
Status: Offline
Reply With Quote
Jan 17, 2005, 06:34 PM
 
Originally posted by someone_else:
Actually, if you read Giampaolo's book Practical Filesystem Design (using BeFS), you will find it's the other way around. The search meta-data was moved INTO the filesystem because storing it separately in a private Tracker DB was a huge speed issue and caused data coherency problems between things like dates stored in the DB and dates in the filesystem.

HFS Spotlight is similar, in that the Spotlight meta-data is stored in the filesystem -- from what I've heard they are using HFS+'s long unused ability to have > 2 forks associated with a file.
... that doesn't really explain the .metadata files
     
nickm
Mac Enthusiast
Join Date: Dec 2001
Status: Offline
Reply With Quote
Jan 19, 2005, 02:48 AM
 
HFS Spotlight is similar, in that the Spotlight meta-data is stored in the filesystem -- from what I've heard they are using HFS+'s long unused ability to have > 2 forks associated with a file
Where did you hear this? (and will you have to kill me?)

I think this would be great, though some apps would have to be retooled. iCal, for instance, always writes new versions of calendars rather than modifying the existing files (try applying a color label to a calendar file and then editing it in iCal --- iCal will delete your labelled file and write a new one.

Backup, in particular, will require all new apps if you want metadata backed up. Still, I'm glad to see progress is being made on this front, rather letting legacy systems hold us back.
     
Millennium
Clinically Insane
Join Date: Nov 1999
Status: Offline
Reply With Quote
Jan 19, 2005, 01:50 PM
 
Originally posted by Angus_D:
As I understand it he's primarily working on Spotlight. He may of course be touching HFS in doing so.
Right now, that's probably where he's working. However, he was also responsible for adding journalling to HFS+. He was originally hired for filesystem work, and Spotlight is closely related to that.
You are in Soviet Russia. It is dark. Grue is likely to be eaten by YOU!
     
OmniX
Dedicated MacNNer
Join Date: Jun 2000
Status: Offline
Reply With Quote
Jan 19, 2005, 06:25 PM
 
Originally posted by larkost:
I really can't think of anything I would want in a primary FileSystem. (there are a few things I would like in the VFS layers, and one can always want more support for other file Systems... like write support for NTFS, but that is not likely).
I think the whole introduction of the concept real/useful metadata (as opposed to just those silly labels, which IIRC have been around (with an interruption from 10.0 - 10.2) since System 6.0.5) in Tiger will be huge, and people will only start to see the implications of it after the OS has been out for a while.

One immediate question I have is: will the Tiger Finder be able to MANIPULATE all this metadata at all, as opposed to just accessing it (via Spotlight)? For example, in today's Finder (and presumably still in Tiger), i can set the label for an MP3 file to 'RED'. Since Spotlight has a 'filter' to access MP3 metadata (i.e. ID3.x tags), can I use the Tiger finder to set/modify the various ID3 fields in a given MP3 file? Or a given MS Word file (Author, version number, etc.)? Or a JPEG file (XIFF or IPTC metadata)??

If this is the case -- i can (will be able to) use the Tiger Finder to modify and manage metadata -- that will be completely awesome, and something completely different. However, though I cannot claim to have a complete detailed or even conceptual understanding of Spotlight and the various features (used and unused) of HFS+, my impression is that the Tiger Finder will NOT be able to manipulate arbitrary metadata, because metadata management will still be left to the applications that create and handle the primary files. Thus iTunes will remain the only mechanism to manipulate/manage metadata for MP3/AAC audio files; Word for word files, Photoshop/Graphic Converter for JPEGS, et cetera....

So, Larkost: indeed I believe there is a lot left to wish for in a Mac file system in particular, and the management of data/metadata in general on our computing platforms. Tiger will be a nice advance, but I think a format/architecture for the storage and management of arbitrary metadata remains a ways off. Isn't this what MS is trying to achieve in WinFS? And isn't that goal so complex that MS (ignoring its general incompetency at the moment) had to drop WinFS for Longhorn?
I can't wait for the day when this is finally realized, in WinFS and/or Mac OS X 10.5. Best of luck to Dominic Giampaolo and his team. Until then, I'll be happy(ier) with Tiger/Spotlight.
     
OmniX
Dedicated MacNNer
Join Date: Jun 2000
Status: Offline
Reply With Quote
Jan 19, 2005, 06:31 PM
 
another question is: where does Spotlight store all of this metadata? There seems to have been some allusions to this question on this post, but I'm not sure if anyone has yet posted a definite answer. Will there be some single Spotlight DB file kept at the root level of a volume (similar at all to the desktopDB now??)?

as an aside, where is the 'label' information in Panther stored today? will this change at all in Tiger?
     
King Bob On The Cob
Mac Elite
Join Date: Apr 2002
Location: Illinois
Status: Offline
Reply With Quote
Jan 20, 2005, 11:15 PM
 
Originally posted by Catfish_Man:
... that doesn't really explain the .metadata files
Apple has to make them in case the file is moved to a place where Resource forks cannot be used, but where metadata should still be stored (like a network server)
     
nickm
Mac Enthusiast
Join Date: Dec 2001
Status: Offline
Reply With Quote
Jan 21, 2005, 09:58 AM
 
another question is: where does Spotlight store all of this metadata?
Spotlight definitely has a metadata store that is used for doing queries. One open question (to those with NDAs) is whether spotlight allows for attaching arbitrary metadata to files or not. If not, then all the metadata you can search must be supported either directly by the file format or by the HFS file system. Thus, there would be no way to add an "author" attribute to a file format that doesn't support it.

If it turns out you can author metadata, it will clearly go in the metadata store. The open question is whether it will also be attached to the file directly through the filesystem. Again, those who know the answer to this are under NDA.

as an aside, where is the 'label' information in Panther stored today?
Label information is a special HFS field.
     
Millennium
Clinically Insane
Join Date: Nov 1999
Status: Offline
Reply With Quote
Jan 21, 2005, 11:35 AM
 
Originally posted by nickm:
One open question (to those with NDAs) is whether spotlight allows for attaching arbitrary metadata to files or not.
As I understand it, Spotlight supports plug-ins so that third-party developers can add their own metadata to the supported types. This isn't under NDA, because it was presented at earlier conferences (WWDC, I believe).

In other words, Spotlight doesn't directly support arbitrary metadata, but plug-ins can extend the metadata support. A developer might, for example, write a plug-in to add Ogg Vorbis comments to Spotlight's search capabilities.

All this said, I don't know if plugins would make it possible to add metadata that isn't already part of the file format. That part, Apple hasn't released publicly yet.
You are in Soviet Russia. It is dark. Grue is likely to be eaten by YOU!
     
Angus_D
Addicted to MacNN
Join Date: Mar 2000
Location: London, UK
Status: Offline
Reply With Quote
Jan 21, 2005, 04:35 PM
 
Originally posted by Millennium:
All this said, I don't know if plugins would make it possible to add metadata that isn't already part of the file format. That part, Apple hasn't released publicly yet.
Sure they have. See http://developer.apple.com/macosx/tiger/spotlight.html. Basically you implement a function that given a file path fills out a dictionary of attributes about that file. I don't know what you mean by "metadata that is't already part of the file format" - you can't just make metadata up, really.
     
nickm
Mac Enthusiast
Join Date: Dec 2001
Status: Offline
Reply With Quote
Jan 21, 2005, 11:21 PM
 
Originally posted by Angus_D:
Sure they have. Basically you implement a function that given a file path fills out a dictionary of attributes about that file. I don't know what you mean by "metadata that is't already part of the file format" - you can't just make metadata up, really.
You aren't understanding me. All the plugin architecture allows you to do is translate metadata from a file-specific format to a general format suitable for querying by Spotlight. This means that if a file format doesn't support metadata, then you can't really write a plugin for it. Suppose I want to associate copyright information or keywords with a plain text file so that I can search for it later. This isn't "made up" metadata, but it also isn't something that can be stored in the file directly because plain text doesn't support metadata. There are other examples too. I may want to add a "project" field to a bunch of files: pdf, text, html, jpeg, whatever. Can spotlight do this?

As near as I can tell, the plugin architecture described in the article you referenced is "read only". Writing a plugin only allows spotlight to read the metadata from the file, not edit it. Just think of all the interesting apps that could be written if there was a uniform interface for adding keywords or other metadata to a file. As it is, if you want to write an app that can tag JPEG, MS Office, and PDF files, you have to know the internals of all those formats, which isn't a very scalable solution.
     
Agent69
Mac Elite
Join Date: Jun 2000
Status: Offline
Reply With Quote
Jan 24, 2005, 09:49 AM
 
If you want to read Dominic's book, you can download it from his website:

http://www.nobius.org/~dbg/index.html
Agent69
     
Millennium
Clinically Insane
Join Date: Nov 1999
Status: Offline
Reply With Quote
Jan 24, 2005, 11:11 AM
 
Originally posted by Angus_D:
Sure they have. See http://developer.apple.com/macosx/tiger/spotlight.html. Basically you implement a function that given a file path fills out a dictionary of attributes about that file.
Oh. OK, then; that works.
I don't know what you mean by "metadata that is't already part of the file format" - you can't just make metadata up, really.
That seems to be what a fair number of people want to do, though: set arbitrary "tags" on a file, like adding an "author" to a file format which doesn't store such things as part of the files.

I don't see much sense in it, but this seems to be what a lot of people think about when it comes to metadata. There's also been talk about wanting to edit it directly from the Finder.
You are in Soviet Russia. It is dark. Grue is likely to be eaten by YOU!
     
Angus_D
Addicted to MacNN
Join Date: Mar 2000
Location: London, UK
Status: Offline
Reply With Quote
Jan 24, 2005, 01:45 PM
 
Originally posted by Millennium:
That seems to be what a fair number of people want to do, though: set arbitrary "tags" on a file, like adding an "author" to a file format which doesn't store such things as part of the files.

I don't see much sense in it, but this seems to be what a lot of people think about when it comes to metadata. There's also been talk about wanting to edit it directly from the Finder.
Well, Finder comments are metadata, I guess.
     
Millennium
Clinically Insane
Join Date: Nov 1999
Status: Offline
Reply With Quote
Jan 24, 2005, 06:22 PM
 
Originally posted by Angus_D:
Well, Finder comments are metadata, I guess.
Sort of, but with a bit more structure. Labels would be a better example.

This doesn't look as though this type of thing would be directly supported by Spotlight, but I think I may have figured out a way to cheat with an add-on app. I need to work up a proof of concept, though.
You are in Soviet Russia. It is dark. Grue is likely to be eaten by YOU!
     
   
Thread Tools
 
Forum Links
Forum Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Top
Privacy Policy
All times are GMT -4. The time now is 06:32 AM.
All contents of these forums © 1995-2017 MacNN. All rights reserved.
Branding + Design: www.gesamtbild.com
vBulletin v.3.8.8 © 2000-2017, Jelsoft Enterprises Ltd.,