[besson3c]

I was just talking to a file system geek friend of mine about ZFS, since it seems that Apple will be included in Leopard, I thought I'd pass on what was said.


ZFS is owned by Sun, so therefore will be licensed to Apple, and is the most efficient file system out there. Its big claim to fame is that it does not require defragging like HFS or NTFS. Mac users talk about how OS X doesn't need defragging, although this is not really true. What can be said is that the performance improvement with defragging a ton of very small files is negligible, but it is significant for large files (e.g. video files). edit: stuff about defragmented seems to be false

BeOS style metadata wasn't necessarily new, it's just not something that was ever really used in the Mac or Windows to this extent. I'm not sure whether moving to ZFS would result in Apple changing how Spotlight works in moving to file metadata, and whether this would provide for restore points for Time Machine at the file system level.

Regardless, it seems pretty cool that Apple will be going from having one of the weakest file systems in the industry to one of the best. I wonder how they are going to assist in migrating all of the HFS disks to ZFS though...


Anybody know more about ZFS and its capabilities? This was only a relatively brief conversation, and I don't consider myself an expert on file systems, but I accept the high praise that my colleague offered ZFS.

[CatOne]

LOL.

"Its big claim to fame is that it does not require defragging like HFS or NTFS."

Naw, that's not its big claim to fame. Really, HFS+ needs very little defragmenting. OS X does many things to reduce the need to defragment -- enough that for MOST people it will NEVER be necessary to defragment.

Did a Google search, and lo and behold, the first link I came against was Sun's page on ZFS:

ZFS: the last word in file systems.

Why not have a read? ZFS is a much more scalable file system than HFS+.

Also a good Wikipedia entry:

ZFS - Wikipedia, the free encyclopedia

And John Siracusa is very good at making some points about ZFS and what it's good at... and how this may relate to 10.5:

http://arstechnica.com/staff/fatbits.ars/2006/8/15/4995

[TETENAL]

ZFS - Wikipedia, the free encyclopedia

[besson3c]

Digesting this slowly...

Maybe this guy was wrong or I misunderstood what was said about fragmentation, but reading the Wikipedia article I see some pretty neat stuff is included, such as the snapshots and vpools...

Still working on understanding all of this...

[bowwowman]

Having read the above linked info and more, suffice it to say that if it works the way it claims to, ZFS appears to a considerable advance in the file system technology and hopefully will be adopted soon by Apple, as well as other OS makers.

The real question for Mac users is how to migrate all of our data over to an OS that uses ZFS without losing any of it.

Surely Apple's programmers have been working on an elegant solution for quite a while now, so they can once again wow us with it at MWSF or whenever 10.5 comes out. I can't help but think this was/is one of the many "secret" features that Steve was referring to.......

And BTW, ZFS's biggest claims to fame are the pooled storage model, infinite scalability, and native redundancy of live data & device striping. Fragmentation is not even mentioned, nor does it need to be, AFAIK

[krove]

I think the biggest wins are the safety mechanisms inherent to ZFS: every block written has a 256-bit checksum that is verified. There are other mechanisms in place too, so go read about them. Hopefully, Apple can get ZFS in OS X to a place where it can be used on the boot disk. That would be awesome.

[TETENAL]

Originally Posted by krove 

Hopefully, Apple can get ZFS in OS X to a place where it can be used on the boot disk. That would be awesome.

It's impossible:

Wikipedia:
there is no ZFS boot support!

[- - e r i k - -]

Originally Posted by TETENAL 

It's impossible:

It is being worked on, which was what krove implied that Apple could help with. Duh.

[krove]

Indeed. I am aware that boot support is not availabe even for Solaris, yet. But they are working on it. As such, I see no reason such a capability could not be created by Apple, either.

[TETENAL]

If Sun can't get it to work – and they created the format – I don't see how Apple could.

[CharlesS]

Originally Posted by TETENAL 

If Sun can't get it to work – and they created the format – I don't see how Apple could.

There's a difference between "haven't yet" and "can't". Sheesh.

[OreoCookie]

HFS+ is not such a bad filesystem, although it's showing its age. But it's definitely not `one of the weakest filesystems' out there. The weakest filesystem in active use is clearly FAT16. ext2 is similarly `old', but still works quite well. On the other hand, the prospect of having zfs drives is groovy

[kman42]

What's the difficulty in making a file system bootable?

[Horsepoo!!!]

Originally Posted by kman42 

What's the difficulty in making a file system bootable?

Not entirely sure but if I had to guess I'd say the firmware has to understand how the file system works in order to figure out where to get the system files to boot the OS. Things could probably get tricky if, say, the data is encrypted or compressed (two features of ZFS). So the firmware has to not only know where to find the system data but has to also be able to decrypt or decompress the data.

[P]

HFS+ is decent. It's a little slow and it's not meant for servers but it's OK.

The downside to HFS+ is that the entire directory is in one file (with a backup), meaning that only one thread can modify the file directory at one time (there is only one big lock). Other file systems, including the grandfather of them all UFS, spread the directory data out to get better threading and to reduce wear on any single area of the disk.

[P]

Originally Posted by Horsepoo!!! 

Not entirely sure but if I had to guess I'd say the firmware has to understand how the file system works in order to figure out where to get the system files to boot the OS. Things could probably get tricky if, say, the data is encrypted or compressed (two features of ZFS). So the firmware has to not only know where to find the system data but has to also be able to decrypt or decompress the data.

Pretty much so, yes. The firmware/EFI/BIOS has to be able to locate the files needed to boot. Basically it looks at a certain block on the disk, the boot block, for the location of the file to start reading. This becomes tricky if you don't even know which disk the first file is on. Not impossible - especially not with EFI, which is quite capable - but tricky.

[OreoCookie]

Since Apple and Sun sell their own machines, they should be able to implement this feature into the firmware easily.

[kick52]

if we did move over to ZFS, we'd need to reformat our drives to the ZFS format, and move all our data. (To my understanding)

Without getting another HD and moving everything on there, formating the main drive, and moving everything back, you would have to reformat you drive, and loose all your data if you have no backup tools.

This looks like it would put off many users.

But, this most likely wont happen, as: ZFS isnt bootable, and Apple probably wouldnt completely replace HFS (They would give you a choice)

[OreoCookie]

Windows includes a tool that migrates a FAT partition to an NTFS partition. No need to reformat anything.

[CatOne]

Originally Posted by kick52 

if we did move over to ZFS, we'd need to reformat our drives to the ZFS format, and move all our data. (To my understanding)

Without getting another HD and moving everything on there, formating the main drive, and moving everything back, you would have to reformat you drive, and loose all your data if you have no backup tools.

This looks like it would put off many users.

But, this most likely wont happen, as: ZFS isnt bootable, and Apple probably wouldnt completely replace HFS (They would give you a choice)

That's pretty simple, though -- you just migrate when you get a new machine... make it part of the migration assistant.

I don't think ZFS is going to be the default file system for a while (i.e. not in 10.5... maybe not even in 10.6, if ever) so there will be, what, 3-5 years for this work to get done if it makes sense?

[Horsepoo!!!]

Originally Posted by CatOne 

That's pretty simple, though -- you just migrate when you get a new machine... make it part of the migration assistant.

I don't think ZFS is going to be the default file system for a while (i.e. not in 10.5... maybe not even in 10.6, if ever) so there will be, what, 3-5 years for this work to get done if it makes sense?

ZFS is one of those things that would make sense to become default in 10.5 because of OS features that would benefit from the file system features such as, say, Time Machine using ZFS snapshots for its backup mechanism.

Right now Apple is abstracting everything so that it doesn't matter what file system you're using, which I suppose is alright...the trade off, however, is a huge performance penalty. The same applies to File Vault. Apple could move File Vault to the file system level if ZFS was made the default file system...File Vault would then be much more flexible and much faster.

OS X could be a very fast OS if Apple chose to make it fast. But Apple's design decisions and poor planning has prevented this from happening.

[naphtali]

Originally Posted by Horsepoo!!! 

ZFS is one of those things that would make sense to become default in 10.5 because of OS features that would benefit from the file system features such as, say, Time Machine using ZFS snapshots for its backup mechanism.

Right now Apple is abstracting everything so that it doesn't matter what file system you're using, which I suppose is alright...the trade off, however, is a huge performance penalty. The same applies to File Vault. Apple could move File Vault to the file system level if ZFS was made the default file system...File Vault would then be much more flexible and much faster.

OS X could be a very fast OS if Apple chose to make it fast. But Apple's design decisions and poor planning has prevented this from happening.

Read somewhere that Time Machine does not work with ZFS (at least for now). Or that some metadata feature didn't work with ZFS. Though I could be wrong.

More interestingly, perhaps Apple could get around the current limitations by introducing Intel's Robson technology. Then all the boot files would be on a separate flash drive! Downside of course is that it won't work with current Macs till they figure out a way to boot off ZFS.

What are the downsides of ZFS?

The Sun article mentioned that it works on Sparc and x86 platforms, and that it is endian neutral. Does that mean it will work on older PPC G4 Macs?

And on the Wikipedia ZFS article - why doesn't Apple encounter the same licensing issues as Linux does - CDDL vs GNU PL? Sorry about the noobish questions - I am a noob so please be patient

And finally after reading John Siracusa's article I am quite bummed out about Time Machine being based on ZFS - he says that a small change to say a huge 800mb file will use up another 800mb for backup. That will be a potentially huge thing for Apple's creative markets - esp video, professional photography. Though maybe they could get around that by backing up the non-destructive changes (like Apeture) instead of the whole file?

Anyhow, exciting stuff!

[OreoCookie]

Originally Posted by naphtali 

Read somewhere that Time Machine does not work with ZFS (at least for now). Or that some metadata feature didn't work with ZFS. Though I could be wrong.

According to Apple, Time Machine is based on HFS+ (at least the early version they were showing to developers) -- which doesn't exclude the use of ZFS. ZFS has a lot of features that should make Time Machine much easier to implement.

Originally Posted by naphtali 

More interestingly, perhaps Apple could get around the current limitations by introducing Intel's Robson technology. Then all the boot files would be on a separate flash drive! Downside of course is that it won't work with current Macs till they figure out a way to boot off ZFS.

Robson technology has nothing to do with this.

Originally Posted by naphtali 

The Sun article mentioned that it works on Sparc and x86 platforms, and that it is endian neutral. Does that mean it will work on older PPC G4 Macs?

It just means that it has been ported to Sparc-based and x86-based systems. Since both cpu architectures are entirely different, I don't see any reason that prevents ZFS to run on PowerPC-based Macs.

Originally Posted by naphtali 

And on the Wikipedia ZFS article - why doesn't Apple encounter the same licensing issues as Linux does - CDDL vs GNU PL? Sorry about the noobish questions - I am a noob so please be patient

AFAIK Linux kernel developers don't allow code to be included in the kernel which hasn't been released under the GPL. Apple has a different attitude, they don't require

Originally Posted by naphtali 

And finally after reading John Siracusa's article I am quite bummed out about Time Machine being based on ZFS - he says that a small change to say a huge 800mb file will use up another 800mb for backup. That will be a potentially huge thing for Apple's creative markets - esp video, professional photography. Though maybe they could get around that by backing up the non-destructive changes (like Apeture) instead of the whole file?

Anyhow, exciting stuff!

That's (as far as I understand) how ZFS tracks changes. Also, the attitude is different: nowadays we have so much storage space that we can simply buy more harddrives to compensate for the loss in effective capacity.

However, if you create and delete a lot of big files, that will fill your backup drive rather quickly.

[ZnU]

Originally Posted by naphtali 

What are the downsides of ZFS?

Its checksumming feature introduces some processor overhead. Probably not a huge deal on today's multicore systems, though, and having confidence that your data is in good shape is certainly worth it.

Also, as noted above, ZFS isn't presently bootable, but this might change. (Probably not by the time Leopard ships.)

That's about it for downsides. Unless you count the fact that ZFS has so many features that explaining its significance to an average user is rather challenging. (But this isn't such a big deal, really; there are all sorts of ways Apple could build on ZFS to provide very useful and comprehensible features. Again, I wouldn't expect to see too much of this in Leopard, since ZFS probably won't be the standard file system there, but I'd expect to see it later.)

Originally Posted by naphtali 

And on the Wikipedia ZFS article - why doesn't Apple encounter the same licensing issues as Linux does - CDDL vs GNU PL? Sorry about the noobish questions - I am a noob so please be patient

While some people disagree, the Free Software Foundation says CDDL is GPL-compatable.

But, of course, the relevant parts of OS X aren't covered by the GPL anyway. If Apple needed to, they could just relicense the whole kernel under CDDL. They probably don't need to, if they're implementing ZFS as a kernel module, though. (The issue of how 'viral' licenses function when works are combined only at runtime is a matter of some dispute.)

Originally Posted by naphtali 

And finally after reading John Siracusa's article I am quite bummed out about Time Machine being based on ZFS - he says that a small change to say a huge 800mb file will use up another 800mb for backup. That will be a potentially huge thing for Apple's creative markets - esp video, professional photography. Though maybe they could get around that by backing up the non-destructive changes (like Apeture) instead of the whole file?

Right, this often isn't going to be a very large problem. In Aperture, when you adjust an image, the system just stores those tweaks (which are tiny) and applies them on the fly; it doesn't store another copy of the image data. Similarly, with video, most operations performed in a non-linear editor like Final Cut don't actually touch the original video files. Time Machine would mostly just be backing up versions of your project file, which is just a few megs.

People who work with a lot of large Photoshop documents are going to have some issues, though. Hopefully Time Machine will let you specify specific folders where you want it disabled or where you want it to keep fewer old versions of files.

[besson3c]

If the changes are going to be tracked at the file system level, couldn't the actual binary data that changed in between saves be logged? I believe this is how some of the older Unix file systems worked that did this, rather than simply keep a copy of each file. Doing the latter would be kind of a dumb way of doing this.

[naphtali]

Thanks for the information! Getting more interested in all this

Originally Posted by OreoCookie 

Robson technology has nothing to do with this.

What I meant was that since ZFS is not bootable in its current state, perhaps Apple could store Mac OS X's boot files off the main drive, and on a flash memory module like with Robson technology / Samsung's FlashOn drive (which they rejected). But then there's the thing about older Macs

Anyhow, I look foward to seeing this in the future (hopefully sooner) cos I'm quite sick of silent corruption.

[TETENAL]

Until it is bootable ZFS will be nothing more than a side note in Mac OS X. Sun worked over two years on bootability of ZFS and failed miserably so far, so it is a safe bet that Apple will never be able to get this to work. If you think you will use a ZFS volume soon you are seriously deluded.

[gorgonzola]

Re the ZFS booting issue, many people will probably need to use Time Machine with a dedicated external backup drive, so ZFS could be useful there regardless of whether you can boot to it. This seems like the most likely use of ZFS in Leopard to me (though I don't know much about it).

[ZnU]

Originally Posted by TETENAL 

Until it is bootable ZFS will be nothing more than a side note in Mac OS X. Sun worked over two years on bootability of ZFS and failed miserably so far, so it is a safe bet that Apple will never be able to get this to work. If you think you will use a ZFS volume soon you are seriously deluded.

Claiming ZFS won't be relevant until it's bootable is fairly odd. While snapshots, checksumming and flexible file system management would be useful even on single-disk systems, where ZFS really comes into its own is with multi-disk arrays, which often aren't used as boot volumes anyway. Maybe this isn't useful to you, but rest assured there are a lot of Apple customers who will find ZFS very useful in this role. Pretty much anyone doing video editing in any format more serious that MiniDV or HDV, for starters.

And the fact that Sun hasn't made ZFS bootable yet really doesn't tell you very much. It certainly doesn't tell you it's inherently impossible to make ZFS bootable. In fact there is code floating around for booting OpenSolaris from ZFS; it's just not release-quality yet.

Keep in mind, Sun sells comparatively few single-disk machines. If you've got an X4500 (the 4U Sun storage server with 48 500 GB hard drives in 4U of rack space, affectionately known as the Thumper), it's really not a huge deal if you have to only have 47 of those disks in your ZFS storage pool so you can boot from the 48th. Apple has more of a reason to make ZFS bootable (most of the systems they sell are single-disk machines), and so might invest more resources to get this up and running. (Probably not for 10.5.0, though.)

[krove]

Does it strike anyone else as strange that Sun set out to create this awesome file system and forgot or put off the ability to boot from this advanced, next-gen FS?

I'm not up to speed on the technical details behind this small gap in features, it does beg the question, what exactly is required for a file system to be bootable? For the longest time, booting off CDs was not possible (not until OS 8.5 or 9, I think). Intel Macs have GPT instead of APM partitioning schemes and this is important vis-à-vis EFI recognition of various disks.

Does anyone have some information behind what it takes to make a file system bootable? I would think that EFI would very easily accommodate various file systems more so than the old PC bios, but at this point, this is all just conjecture.

[ZnU]

Originally Posted by krove 

Does it strike anyone else as strange that Sun set out to create this awesome file system and forgot or put off the ability to boot from this advanced, next-gen FS?

Not so much; see my post above. While ZFS booting would be great, ZFS is extremely useful without this.

Originally Posted by krove 

Does anyone have some information behind what it takes to make a file system bootable? I would think that EFI would very easily accommodate various file systems more so than the old PC bios, but at this point, this is all just conjecture.

Basically, in order to natively boot from a file system, there needs to be code in firmware which understands how to read that file system, at least well enough to locate and load a single file. This is annoying with ZFS, because it's a lot more complicated than most other file systems.

This isn't quite the barrier that it might seem, though. There are hacks for booting from ZFS on OpenSolaris x86. The trick is basically to have a bootloader that has been modified to read from ZFS on a small UFS partition. As far as the machine's firmware is concerned, that's the boot partition, but as soon as the bootloader is up and running, it starts loading the operating system off a ZFS file system.

[CharlesS]

Couldn't they pull off ZFS booting via using some kind of wrapper partition like the HFS wrapper around HFS+?

[OreoCookie]

Originally Posted by besson3c 

If the changes are going to be tracked at the file system level, couldn't the actual binary data that changed in between saves be logged? I believe this is how some of the older Unix file systems worked that did this, rather than simply keep a copy of each file. Doing the latter would be kind of a dumb way of doing this.

As far as I understand, the old blocks are kept so that `only the changed data' is saved.

Originally Posted by naphtali 

What I meant was that since ZFS is not bootable in its current state, perhaps Apple could store Mac OS X's boot files off the main drive, and on a flash memory module like with Robson technology / Samsung's FlashOn drive (which they rejected). But then there's the thing about older Macs.

No, this has nothing to do with ZFS. See, the Flash drive would use ZFS, too, so there is really no difference if the data sits on a flash drive or a `real' harddrive.

Originally Posted by ZnU 

Claiming ZFS won't be relevant until it's bootable is fairly odd. While snapshots, checksumming and flexible file system management would be useful even on single-disk systems, where ZFS really comes into its own is with multi-disk arrays, which often aren't used as boot volumes anyway. Maybe this isn't useful to you, but rest assured there are a lot of Apple customers who will find ZFS very useful in this role. Pretty much anyone doing video editing in any format more serious that MiniDV or HDV, for starters.

Apple and Sun can simply use a boot partition and mount the rest right after the relevant drivers have loaded. FreeBSD for instance creates quite a few slices by default (the equivalent of volumes).

So it would be trivial to use all the features of ZFS IMHO -- after all, there is no necessity to use one filesystem for all files.

Originally Posted by ZnU 

And the fact that Sun hasn't made ZFS bootable yet really doesn't tell you very much. It certainly doesn't tell you it's inherently impossible to make ZFS bootable. In fact there is code floating around for booting OpenSolaris from ZFS; it's just not release-quality yet.

Yes, I agree. Sun is very good at producing high-quality code.


So even if ZFS won't be used in 10.5, its inclusion in the latest dev builds is a clear indication that Apple is seriously interested in it.