[besson3c]

Nothing to do with licensing, but politics, power, future proofing, insurance, etc.

Oracle has basically written the book on how to destroy a good product. This might be rife with inaccuracies, but as best as I can piece this together this seems to be the history of ZFS, in chronological order (although these may not be in exact sequence):

• Initially required commercial Solaris license

• OpenSolaris introduced at some point, ZFS available there, OpenSolaris thought to be what Fedora is to Redhat

• Solaris 10 (and possibly earlier versions, I don't know) made free to use, including patch downloads

• Sometime prior to the takeover Apple starts becoming interested

• Oracle takeover

• Oracle also offers a Linux product, with this and btrfs making the future of Solaris beyond the upcoming version 11 and ZFS a mystery. Plus, they're Oracle. A lot of apprehension seems to exist with MySQL as well, another property of Oracle's

• Free Solaris 10 no longer a viable option, as patches are unavailable to free customers

• OpenSolaris killed, the only option for free Solaris users is to fork over $1000/year for a Solaris license even if they need nothing more than the privilege of being able to download patches, or switch to a non-Solaris based OS supporting ZFS

• After all of this time, information on the Oracle site is still hairy and there are many broken links. For instance, other than the license agreement there is no explicit mention that if you download Solaris 10 that is easily accessible through the site that you won't be able to download patches without an Oracle customer support identifier thingy. Not only that, but the Update Manager GUI won't even tell you this - just silent failure, and the command line updater a generic non-specific error message. Bliss! For some reason their online support requires Flash and doesn't work with Flash blockers at all, but I digress...

• Illumos group formed to help support non-Solaris implementations of ZFS (Nexenta, OpenIndiana, FreeBSD, etc.), and in the case of OpenIndiana, OpenSolaris itself which has morphed into OpenIndiana. OpenIndiana versions now use the Illumos kernel and make use of the free bits of OpenSolaris and ZFS. Apparently some of the OpenSolaris developers are on the OpenIndiana team, but I don't know whether we'll see ZFS and/or OpenSolaris/OpenIndiana OS future innovation, or just put on life support.

With Apple having connections to Oracle management, my theory is that they looked at what was coming down the pipeline and were rightfully nervous about all of this. Apple could have still did what OSes like FreeBSD have done, but if Oracle will eventually kill off Solaris/ZFS in favor of their Linux-based products, this would mean that Apple would have to either rely on community support (which would be an unknown entity) or assume responsibility for the file system. If this was a possible outcome, I can see why Apple would have thought that it it would make the most sense for them to control 100% of their own file system and destiny. Besides, Apple is always free to consult the ZFS code to check out how things were done as long as they don't copy it verbatim.

Anyway, if there was ever any doubt, Oracle kind of sucks.

[bstone]

What's ZFS?

[SSharon]

Originally Posted by bstone 

What's ZFS?

Let me google that for you

Come on, bstone, you know how the internet works.

[CharlesS]

Originally Posted by besson3c 

but if Oracle will eventually kill off Solaris/ZFS in favor of their Linux-based products, this would mean that Apple would have to either rely on community support (which would be an unknown entity) or assume responsibility for the file system. If this was a possible outcome, I can see why Apple would have thought that it it would make the most sense for them to control 100% of their own file system and destiny.

So the worst-case scenario is that Apple might have to develop ZFS on their own. So the safer solution is... to develop a new file system on their own?

[bstone]

Originally Posted by SSharon 

Let me google that for you

Come on, bstone, you know how the internet works.

In all my letters, I try harder than anything else to make myself clear. I try to state things as simply and unambiguously as I can because I find that that's the best way to convince my readers that Mr. S Sharon believes, in his elitist delirium, that power, politics, and privilege should prevail over the rule of law. I want to share this with you because I feel that he has insulted everyone with even the slightest moral commitment. S obviously has none or he wouldn't move ungrateful authoritarianism from the dodgy fringe into a realm of respectability. We must overcome the fears that beset us every day of our lives. We must overcome the fear that he will make individuals indifferent to the survival of their families. And to overcome these fears, we must report as best as possible the facts and circumstances surrounding S's rancorous theories.

I feel no more personal hatred for S than I might feel for a herd of wild animals or a cluster of poisonous reptiles. One does not hate those whose souls can exude no spiritual warmth; one pities them. By writing this letter, I am clearly sticking my head far above the parapet. The big danger is that S will retaliate against me. He'll most likely try to force me to kiss my freedom goodbye although another possibility is that his accomplices are unified under a common goal. That goal is to rule with an iron fist. More often than not, some people don't seem to mind that he likes to reopen wounds that seem scarcely healed. What an antihumanist world we live in!

S used to maintain that we can stop racialism merely by permitting government officials entrée into private homes to search for cankered junkies. When he realized that no one was falling for that claptrap, he changed his tune to say that I'm too presumptuous to reveal the constant tension between centripetal and centrifugal forces of dialogized heteroglossia resulting from his catch-phrases. S is really a cruel liar, and shame on anyone who believes him. What he seems to be forgetting is that it requires surprisingly little imagination to envision a future in which he is free to desecrate religious objects. Let me try to explain what I mean by that in a single sentence: Once you understand his subliminal psywar campaigns, you have a responsibility to do something about them. To know, to understand, and not to act, is an egregious sin of omission. It is the sin of silence. It is the sin of letting S destroy the natural beauty of our parks and forests. Many the things I've talked about in this letter are obvious. We all know they're true. But still it's necessary for us to say them because Mr. S Sharon's formula for philistinism is more flagitious than ever.

[P]

ZFS is a filesystem, originally developed by Sun, that Apple played with on and off as the future FS for OS X. Since Lion has now arrived with absolutely zero news of a new FS, discussion about ZFS is starting up again.

The basic story that leaked was that Apple wanted the code under a different license. What I don't get about that is that ZFS was released under the same license (CDDL) as Dtrace, and Apple uses Dtrace. Perhaps they're thinking that if Dtrace died, that would be sad but they could rip it out without killing the entire OS, but a filesystem is fundamental to the OS.

I'm not so sure about your explanation, besson: Knowing Apple, they'd fork it and make a new OS if Oracle pulled support.

[turtle777]

Originally Posted by CharlesS 

So the worst-case scenario is that Apple might have to develop ZFS on their own. So the safer solution is... to develop a new file system on their own?

No, the worst-case scenario would have been that Apple started implementing ZFS, and then it would gradually blow up over the next years (due to various reasons - politics, licensing, support etc), suddenly leaving them with no alternative and a broken file system.

-t

[besson3c]

Originally Posted by CharlesS 

So the worst-case scenario is that Apple might have to develop ZFS on their own. So the safer solution is... to develop a new file system on their own?

Well, that's a good question for you as a developer...

Couldn't Apple just look at all of the ZFS code and use a substantial amount of this same code in their own file system? How much would they have to rework it to be in good legal standing, and how would Oracle ever know if Apple used some of their code verbatim, and how much does the CDDL permit? Does a port from one language or framework to another count as enough of a rework to put a company in good legal standing?

[besson3c]

Originally Posted by P 

I'm not so sure about your explanation, besson: Knowing Apple, they'd fork it and make a new OS if Oracle pulled support.

But this is a lot of responsibility to assume and years and years of code to take over written by people long gone, and who knows how much documentation exists?

It just seems pretty Apple-like to want more control.

[besson3c]

Originally Posted by turtle777 

No, the worst-case scenario would have been that Apple started implementing ZFS, and then it would gradually blow up over the next years (due to various reasons - politics, licensing, support etc), suddenly leaving them with no alternative and a broken file system.

-t

I wouldn't say broken, Apple would have every last byte of non-proprietary code (and there is no proprietary code that is absolutely necessary to run ZFS), but it might be fairly unmanageable.

Then again, Apple assumed responsibility over KHTML and made it theirs. I don't know if this is a fair comparison though, given the complexities of each. KHTML is probably complex, but it's much higher level stuff.

[CharlesS]

Originally Posted by turtle777 

No, the worst-case scenario would have been that Apple started implementing ZFS, and then it would gradually blow up over the next years (due to various reasons - politics, licensing, support etc), suddenly leaving them with no alternative and a broken file system.

Which would leave Apple on their own developing it — which is exactly where they'll be writing a new one. Assuming a complete collapse of ZFS which resulted in Apple being the sole remaining entity doing any development or offering any support on it, it would be identical to a home-brew file system, except that they would have started here:

|-------------------------------------------------------|
                                                                                            ^

instead of here:

|-------------------------------------------------------|
^

It's all water under the bridge at this point, of course, so it's not worth grousing about anymore. I would have been quite pleased if they'd adopted ZFS though.

[besson3c]

Originally Posted by CharlesS 

It's all water under the bridge at this point, of course, so it's not worth grousing about anymore. I would have been quite pleased if they'd adopted ZFS though.

Maybe, but what we end up getting may have a significant amount of ZFS or even btrfs code or inspired code, so maybe the start-from-scratch premise is not quite accurate?

[besson3c]

Also, just to do the straw man thing and preempt an argument: yes, regular users will benefit a great deal from a modern OS X file system, it will make a world of difference to us all, even if we don't understand why or how.

The snapshot thing along is an absolute game changer. Other benefits:

- ZFS send/receive (iCloud backups?)
- the option of establishing fast SSD based read/write buffers (better performance)
- being able to throw your copy of DiskWarrior into the trash
- Some/all Macs being shipped with multiple hard drives without expecting users to know anything about RAID (protection from physical SATA HD failure)
- a better Time Machine
- a better Lion Versions
- routine application/OS install rollback options
- file system level compression (faster reads)
- encryption (a better/revamped Filevault)
- intelligent incremental backup of large files such as VM images

[CharlesS]

Originally Posted by besson3c 

Maybe, but what we end up getting may have a significant amount of ZFS or even btrfs code or inspired code, so maybe the start-from-scratch premise is not quite accurate?

Just because they learned from prior work doesn't mean it's not from scratch. Everything is built upon prior work to some extent.

I'm convinced that had they stuck with ZFS, the new file system would have probably have been available by Snow Leopard, and would definitely be there in Lion.

Originally Posted by besson3c 

Also, just to do the straw man thing and preempt an argument: yes, regular users will benefit a great deal from a modern OS X file system, it will make a world of difference to us all, even if we don't understand why or how.

The snapshot thing along is an absolute game changer.

If nothing else, it would allow you to let Time Machine backup your VMWare/Parallels images.

[mduell]

The ability to download patches for Solaris is orthogonal to Apple licensing ZFS.

We've been using ZFS (on FreeBSD) more and more at work lately, and the features/performance is fantastic. Complete 200GB database backups in 30 seconds via snapshots, 5GB incrementals created and transfered to a remote machine in 5 minutes (zfs send and snapshot diffs).

On the desktop, Time Machine performance would be incredible with ZFS.

[besson3c]

Originally Posted by mduell 

The ability to download patches for Solaris is orthogonal to Apple licensing ZFS.

Agreed, I just wanted to provide a complete picture of the mess that is Oracle.

We've been using ZFS (on FreeBSD) more and more at work lately, and the features/performance is fantastic. Complete 200GB database backups in 30 seconds via snapshots, 5GB incrementals created and transfered to a remote machine in 5 minutes (zfs send and snapshot diffs).

On the desktop, Time Machine performance would be incredible with ZFS.

Very cool to hear, it's good to know that I'm not the only ZFS fanatic in my circle of associates! Most people seem to appreciate it, but the fact you can't get it on Linux is a major hurdle. If Apple had implemented it in OS X, who knows, maybe their whole server strategy would have been different, and certainly their Snow Leopard/Lion feature pages filled with things that leverage ZFS to some extent.

I mean, 30 second snapshots of 200GB databases? Say no more...

[Waragainstsleep]

How much code is involved in a filesystem? I always figured given how fundamental it is and how fast it has to do things that it would be reasonably lightweight in terms of coding.

[besson3c]

Originally Posted by Waragainstsleep 

How much code is involved in a filesystem? I always figured given how fundamental it is and how fast it has to do things that it would be reasonably lightweight in terms of coding.

That I don't know, but I think a modern file system like ZFS would have many tentacles in several parts of the OS. Into the kernel is a given, but also into the boot loader, userland tools responsible for creating snapshots and managing pools and stuff, share a pool across the network (iSCSI and NFS support is built right into ZFS), modular feature add-ons, metadata support, etc.

An older file system like HFS+ would probably amount to relatively few lines of code, but that's because it provides virtually nothing in the way of userland features short of its metadata support and repair tool (fsck).

[P]

Originally Posted by besson3c 

The snapshot thing along is an absolute game changer. Other benefits:

- ZFS send/receive (iCloud backups?)

Which is a filesystem snapshot that you then send over the network. I don't see how that makes sense for iCloud - you wouldn't backup the entire disk to iCloud.

Originally Posted by besson3c 

- the option of establishing fast SSD based read/write buffers (better performance)

Would be great, but not really related to the filesystem. There are other ways to do it, including the way that Intel does it - using the RAID1 hardware in the chipset. Just get us a driver for that and I'll be happy.

Originally Posted by besson3c 

- being able to throw your copy of DiskWarrior into the trash

Yeah, no. Every filesystem ever fails. This is actually a downside - there will be a lack of tools until the FS has been out a while.

Originally Posted by besson3c 

- Some/all Macs being shipped with multiple hard drives without expecting users to know anything about RAID (protection from physical SATA HD failure)

They are busy deleting the optical drive. Does it seem likely that they will ship new Macs with multiple hard drives? No, Apple's solution to guard against drive failure is Time Machine.

Originally Posted by besson3c 

- a better Time Machine
- a better Lion Versions

Better how? Cleaner, yes. Easier to implement, certainly. Better for the end user? Not sure. It would be awesome to have ZFS before those things happened, because it would be easier for Apple to make and test them, but now they exist and reimplementing them would just be more work.

Originally Posted by besson3c 

- routine application/OS install rollback options

Can be done through Time Machine, but OK, fair point. A System Restore feature wouldn't be out of place.

Originally Posted by besson3c 

- file system level compression (faster reads)

Do we want that? Some SSDs have compression in the firmware. For regular HDDs, it's not really important. Compression is for network transfers these days.

Originally Posted by besson3c 

- encryption (a better/revamped Filevault)

Lion's Filevault is a big step forward here. How is ZFS better than that`?

Originally Posted by besson3c 

- intelligent incremental backup of large files such as VM images

Yes, this would be nice.

All in all, these amount to things to remove Apple's hacks and maybe add some slight performance improvements. Nice and all, but nothing to convince me that ZFS is the best possible choice. I still think that an HFS3 could be better.

[Spheric Harlot]

Originally Posted by bstone 

In all my letters, I try harder than anything else to make myself clear. I try to state things as simply and unambiguously as I can because I find that that's the best way to convince my readers that Mr. S Sharon believes, in his elitist delirium, that power, politics, and privilege should prevail over the rule of law. I want to share this with you because I feel that he has insulted everyone with even the slightest moral commitment. S obviously has none or he wouldn't move ungrateful authoritarianism from the dodgy fringe into a realm of respectability. We must overcome the fears that beset us every day of our lives. We must overcome the fear that he will make individuals indifferent to the survival of their families. And to overcome these fears, we must report as best as possible the facts and circumstances surrounding S's rancorous theories.

I feel no more personal hatred for S than I might feel for a herd of wild animals or a cluster of poisonous reptiles. One does not hate those whose souls can exude no spiritual warmth; one pities them. By writing this letter, I am clearly sticking my head far above the parapet. The big danger is that S will retaliate against me. He'll most likely try to force me to kiss my freedom goodbye although another possibility is that his accomplices are unified under a common goal. That goal is to rule with an iron fist. More often than not, some people don't seem to mind that he likes to reopen wounds that seem scarcely healed. What an antihumanist world we live in!

S used to maintain that we can stop racialism merely by permitting government officials entrée into private homes to search for cankered junkies. When he realized that no one was falling for that claptrap, he changed his tune to say that I'm too presumptuous to reveal the constant tension between centripetal and centrifugal forces of dialogized heteroglossia resulting from his catch-phrases. S is really a cruel liar, and shame on anyone who believes him. What he seems to be forgetting is that it requires surprisingly little imagination to envision a future in which he is free to desecrate religious objects. Let me try to explain what I mean by that in a single sentence: Once you understand his subliminal psywar campaigns, you have a responsibility to do something about them. To know, to understand, and not to act, is an egregious sin of omission. It is the sin of silence. It is the sin of letting S destroy the natural beauty of our parks and forests. Many the things I've talked about in this letter are obvious. We all know they're true. But still it's necessary for us to say them because Mr. S Sharon's formula for philistinism is more flagitious than ever.

zetaboards, okCupid, or 8bc.org? (because Google works well for everyone who is not you)

Or cobbled together from memory and acid?

In any case, it's fairly obvious that you're completely off your rocker.

[CharlesS]

edit: figured it out! He used a random complaint letter generator.

[besson3c]

Originally Posted by P 

Which is a filesystem snapshot that you then send over the network. I don't see how that makes sense for iCloud - you wouldn't backup the entire disk to iCloud.

In ZFS I'm assuming the whole partition structure would change, because in ZFS you don't have to set a quota for that partition unless you want to. So, your home directory could be a pool, your applications directory another, etc. Home directory snapshots could be sent to iCloud as backups.

Would be great, but not really related to the filesystem. There are other ways to do it, including the way that Intel does it - using the RAID1 hardware in the chipset. Just get us a driver for that and I'll be happy.

I don't know of any other file system that provides this functionality, perhaps we are not on the same page as to what I meant here? What is your impression of what I was referring to here?

Yeah, no. Every filesystem ever fails. This is actually a downside - there will be a lack of tools until the FS has been out a while.

Chances are much, much, much less that with the CoW mirror/raid sets that the file system will become corrupt. You'll lose disks, but not the file system itself, and when you do the included scrubber should be more than sufficient rather than having to rely on third party products.

The fact that so many Mac users have relied on a third party product to band-aid Apple's file system is kind of a joke, agreed?

They are busy deleting the optical drive. Does it seem likely that they will ship new Macs with multiple hard drives? No, Apple's solution to guard against drive failure is Time Machine.

That's not a solution to drive failure, that's a solution to data loss from drive failure.

Unlike SATA drives and optical drives, flash drives can be very tiny. This is the big difference.

Better how? Cleaner, yes. Easier to implement, certainly. Better for the end user? Not sure. It would be awesome to have ZFS before those things happened, because it would be easier for Apple to make and test them, but now they exist and reimplementing them would just be more work.

Better for the end user because they would be considerably faster, go back in time further since snapshots consume less space, and performed more frequently behind the scenes since this would be far less intrusive since they would just take a few seconds rather than minutes.

Do we want that? Some SSDs have compression in the firmware. For regular HDDs, it's not really important. Compression is for network transfers these days.

Why not? The setting is configurable, but it would allow far more storage on a single disk, particularly making all of those little text files take up a fraction of the space they do, and it would also the reduce I/O
workload and stress the CPU more.

Lion's Filevault is a big step forward here. How is ZFS better than that`?

It may not be, it it is lower level which may provide Apple with some non-enduser-specific advantages relating to maintenance.

All in all, these amount to things to remove Apple's hacks and maybe add some slight performance improvements. Nice and all, but nothing to convince me that ZFS is the best possible choice. I still think that an HFS3 could be better.

I'm not saying that ZFS is the best possible choice, I'm just using it as the poster child for what a modern file system would bring to the table, because I think it has set the bar.

Would you have disagreed if I characterized all of this as what a generic modern file system would bring to the table? To me there is no strong rationale that makes a compelling case as to why OS X would not benefit form a modern filesystem.

[P]

Originally Posted by besson3c 

In ZFS I'm assuming the whole partition structure would change, because in ZFS you don't have to set a quota for that partition unless you want to. So, your home directory could be a pool, your applications directory another, etc. Home directory snapshots could be sent to iCloud as backups.

Yes, but how is that better than just sending the files? That the metadata is stored "naturally"? There isn't a lot of metadata in the home folder that is very relevant any more. Any other bonus?

Originally Posted by besson3c 

(SSD Cache)
I don't know of any other file system that provides this functionality, perhaps we are not on the same page as to what I meant here? What is your impression of what I was referring to here?

It's not in the filesystem per se, but I don't see the reason that it should be. The current disk cache in RAM (part of the VM system) is independent of the filesystem. The hardware disk cache is independent of the filesystem.

The two options I mentioned are ReadyBoost, which is part of Windows Vista and on and support SSDs from Windows 7, and Intel's Smart Response Technology (SRT). SRT uses RAID1, aka mirroring. It is a write-through cache, all stores are sent to both HDD and SSD, so it is only used as a read cache, but that could certainly change. Since it's RAID-based, I imagine that it filters at block level. Readyboost seems to be file-level. There is also a similar feature in DragonBSD, but I don't know anything about it.

Originally Posted by besson3c 

Chances are much, much, much less that with the CoW mirror/raid sets that the file system will become corrupt. You'll lose disks, but not the file system itself, and when you do the included scrubber should be more than sufficient rather than having to rely on third party products.

The fact that so many Mac users have relied on a third party product to band-aid Apple's file system is kind of a joke, agreed?

RAID is awesome, but it assumes multiple disks, and that just won't happen on most of Apple's product line. HFS+ isn't particularly fragile now that it is journalled, but of course it could be much better.

Originally Posted by besson3c 

(Time Machine and drive failure) That's not a solution to drive failure, that's a solution to data loss from drive failure.

Splitting hairs. If your drive fails, you buy a new one and restore from your backup. What would you do in the ZFS case - restore the filesystem on a broken drive?

Originally Posted by besson3c 

Unlike SATA drives and optical drives, flash drives can be very tiny. This is the big difference.

Fair enough, but the MBA is too small to even fit the regular-sized 2.5" SSD, and those flash chips do take up some space. With a clever controller you could do a RAID5 inside the drive, but you would still be in trouble if the controller dies.

Originally Posted by besson3c 

(Time Machine) Better for the end user because they would be considerably faster, go back in time further since snapshots consume less space, and performed more frequently behind the scenes since this would be far less intrusive since they would just take a few seconds rather than minutes.

Faster is good, but would they really be? Yes you can copy on the blocklevel, but how many files are changed on the blocklevel rather than the entire file being rewritten? Other than VM files, which are excluded from the backups by default anyway.

Originally Posted by besson3c 

(Compression)
Why not? The setting is configurable, but it would allow far more storage on a single disk, particularly making all of those little text files take up a fraction of the space they do, and it would also the reduce I/O workload and stress the CPU more.

A lot of files are already compressed. Will ZFS use less than a complete block for a file? If not, all those tiny text files won't shrink at all.

Originally Posted by besson3c 

I'm not saying that ZFS is the best possible choice, I'm just using it as the poster child for what a modern file system would bring to the table, because I think it has set the bar.

Would you have disagreed if I characterized all of this as what a generic modern file system would bring to the table? To me there is no strong rationale that makes a compelling case as to why OS X would not benefit form a modern filesystem.

I absolutely agree that the one big hole in OS X now is the lack of a modern filesystem. A filesystem that doesn't support threading is a joke when every computer they sell supports at least 4 threads, and we're going to bump in to the max size soon enough (it's 16TB if you keep using 4K blocks). What I'm saying is that making filesystems is HARD. ZFS would have been awesome, but if they can't have that (and honestly, Apple had sunk quite some work into it when they killed it, so they had a good reason), we don't need to have everything that ZFS has. We need more than 32-bit filesystems, we need better speed, and yes we probably should have the encryption at that level. Snapshots are an awesome idea, and if the patent gods allow it, lets use it, but they're not required. SSD caching really should be in another layer, either the VM stack or hardware (I think the next version of the ATA-standard has something of that type brewing as well). All the RAID stuff is not necessary for Apple's use case. Probably a bunch of other features are like this as well.

[besson3c]

Originally Posted by P 

Yes, but how is that better than just sending the files? That the metadata is stored "naturally"? There isn't a lot of metadata in the home folder that is very relevant any more. Any other bonus?

Yes. Very, very tiny incremental backup snapshots.

It's not in the filesystem per se, but I don't see the reason that it should be. The current disk cache in RAM (part of the VM system) is independent of the filesystem. The hardware disk cache is independent of the filesystem.

The two options I mentioned are ReadyBoost, which is part of Windows Vista and on and support SSDs from Windows 7, and Intel's Smart Response Technology (SRT). SRT uses RAID1, aka mirroring. It is a write-through cache, all stores are sent to both HDD and SSD, so it is only used as a read cache, but that could certainly change. Since it's RAID-based, I imagine that it filters at block level. Readyboost seems to be file-level. There is also a similar feature in DragonBSD, but I don't know anything about it.

The ZFS read/write cache (L2ARC and ZIL) allow you to dedicate any number of disks in your pool of any size and any type to function as read and write buffers.

RAID is awesome, but it assumes multiple disks, and that just won't happen on most of Apple's product line. HFS+ isn't particularly fragile now that it is journalled, but of course it could be much better.

Why wouldn't it happen? The number one point of hardware failure seems to be hard disk based, so to me the question is more when rather then if.

HFS+ is still quite fragile and prone to corruption. In fact, I needed to run Disk Warrior on my wife's laptop just recently, and I've had to run it for other people too post journal introduction. I think if you took a survey you'd discover that there is still a need for DiskWarrior.

Splitting hairs. If your drive fails, you buy a new one and restore from your backup. What would you do in the ZFS case - restore the filesystem on a broken drive?

No, the broken drive would be automatically detached from the pool, OS X would throw up a nice prompt saying "hey a-hole, you need to replace this drive", but because of this redundancy the user would still be able to work and do stuff without having to worry about permanent damage/loss. I envision Apple coming up with some way to make replacing damaged drives easy without having to open the case (perhaps similar to the old Powerbook battery mechanism). You would't even have to power down to do this.

Fair enough, but the MBA is too small to even fit the regular-sized 2.5" SSD, and those flash chips do take up some space. With a clever controller you could do a RAID5 inside the drive, but you would still be in trouble if the controller dies.

There is no need for a RAID controller in ZFS, again the file system I happen to be using as poster child for modern file systems.

I don't know exactly what the future holds, but I don't see any technical disadvantage over Apple allowing you to build the capacity you need by plugging in a number of thumbnail sized drives, say in 6 slots or something (a raid 5 set + mirror).

This is another compelling reason for a file system like ZFS actually. With our current system we have:

- a single giant drive that stores everything
- when it dies, you hopefully have a backup
- when you run out of space you have to upgrade to a larger drive and transfer all your paint over somehow

With what I'm describing the following would be possible:

- Say, 6 very tiny removable drives that can be of whatever size you want
- When one dies everything continues on, you are alerted
- When you want to upgrade you simply replace the drives with larger ones one at a time, using the mirror to copy the data back to the drive. You can do this on a live system

Yes I realize that there are many issues with making this grandma friendly and yadda yadda, but I'm just trying to suggest that there is a ton of potential here, and what we are used to today kind of sucks.

Faster is good, but would they really be? Yes you can copy on the blocklevel, but how many files are changed on the blocklevel rather than the entire file being rewritten? Other than VM files, which are excluded from the backups by default anyway.

I'm not sure what you are asking here? Are block level snapshots faster? Yes. Do they take up less space than complete file backups? Yes, considerably less.

A lot of files are already compressed. Will ZFS use less than a complete block for a file? If not, all those tiny text files won't shrink at all.

The savings and benefits would vary and be context sensitive, you are right, but having this handled at a low level so that this just works™ and you can configure how aggressive you want to compress certain parts of your file system is pretty cool.

I absolutely agree that the one big hole in OS X now is the lack of a modern filesystem. A filesystem that doesn't support threading is a joke when every computer they sell supports at least 4 threads, and we're going to bump in to the max size soon enough (it's 16TB if you keep using 4K blocks). What I'm saying is that making filesystems is HARD. ZFS would have been awesome, but if they can't have that (and honestly, Apple had sunk quite some work into it when they killed it, so they had a good reason), we don't need to have everything that ZFS has. We need more than 32-bit filesystems, we need better speed, and yes we probably should have the encryption at that level. Snapshots are an awesome idea, and if the patent gods allow it, lets use it, but they're not required. SSD caching really should be in another layer, either the VM stack or hardware (I think the next version of the ATA-standard has something of that type brewing as well). All the RAID stuff is not necessary for Apple's use case. Probably a bunch of other features are like this as well.

I say that if Apple is going to put the time into something as low level as the file system that they go all out and do it right, making it even better than ZFS. One of the other nice features of ZFS that we haven't discussed is the ability to apply filesystem feature updates to live file systems. So, Apple could launch this with a subset of the features we have been talking about, but they ought to make their file system *ready* for all of the goodness that is in ZFS, and they ought to think about building computers and software around these pretty significant features.

[P]

Originally Posted by besson3c 

The ZFS read/write cache (L2ARC and ZIL) allow you to dedicate any number of disks in your pool of any size and any type to function as read and write buffers.

So it's basically ReadyBoost then - except ReadyBoost has a limit on the number of drives you can add, I think. I would prefer something in the SATA controller, like SRT is. No need to burden the CPU further.

Originally Posted by besson3c 

Why wouldn't it happen? The number one point of hardware failure seems to be hard disk based, so to me the question is more when rather then if.

Multiple HDDs are simply too big. As for any flash-based storage - we get to that below.

Originally Posted by besson3c 

HFS+ is still quite fragile and prone to corruption. In fact, I needed to run Disk Warrior on my wife's laptop just recently, and I've had to run it for other people too post journal introduction. I think if you took a survey you'd discover that there is still a need for DiskWarrior.

I was listening to a podcast yesterday where Siracusa was saying more or less the same thing. Maybe I've been lucky, or maybe there's some common software out there that I don't use that triggers this.

Originally Posted by besson3c 

No, the broken drive would be automatically detached from the pool, OS X would throw up a nice prompt saying "hey a-hole, you need to replace this drive", but because of this redundancy the user would still be able to work and do stuff without having to worry about permanent damage/loss. I envision Apple coming up with some way to make replacing damaged drives easy without having to open the case (perhaps similar to the old Powerbook battery mechanism). You would't even have to power down to do this.

This is simply not how they work. You can't even replace the entire HDD yourself without disassembling the entire computer. I think that the way forward is in flash. All SSDs have some extra area that can be reused as the flash cells wear out. This means that they guard against failure in the flash chips, but not in the controller. Right now there are a lot of early failures on stuff like this, but there is no reason for a controller to die any earlier than a CPU. Once testing has improved to CPU levels, this means that the hardware should be more reliable. Not fail-proof, but no worse than anything else.

Also note that when flash dies, it can still be read (just no written) for a few more months. This makes the parity checking in RAID irrelevant for this failure.

Originally Posted by besson3c 

There is no need for a RAID controller in ZFS, again the file system I happen to be using as poster child for modern file systems.

Which is just another way of saying that it is RAID in software.

Originally Posted by besson3c 

I don't know exactly what the future holds, but I don't see any technical disadvantage over Apple allowing you to build the capacity you need by plugging in a number of thumbnail sized drives, say in 6 slots or something (a raid 5 set + mirror).

What you seem to be proposing is a lot of little flash drives, each with a connection of its own to the PCH and thus visible to the OS. It's just a lot of extra connections to a chip that already has a bunch of them, all so you can move the function from a dedicated chip to the CPU. Why would you want that?

Originally Posted by besson3c 

This is another compelling reason for a file system like ZFS actually. With our current system we have:

- a single giant drive that stores everything
- when it dies, you hopefully have a backup
- when you run out of space you have to upgrade to a larger drive and transfer all your paint over somehow

With what I'm describing the following would be possible:

- Say, 6 very tiny removable drives that can be of whatever size you want
- When one dies everything continues on, you are alerted
- When you want to upgrade you simply replace the drives with larger ones one at a time, using the mirror to copy the data back to the drive. You can do this on a live system

Yes I realize that there are many issues with making this grandma friendly and yadda yadda, but I'm just trying to suggest that there is a ton of potential here, and what we are used to today kind of sucks.

There is no reason to expect flash drives to up and die in the same manner that spinning discs do. They shouldn't die any more often than RAM chips do. Even if we want to do what you propose, that can better be handled in the SSD controller. After receiving that warning message, you would then make a fresh backup and then replace the entire drive.

Originally Posted by besson3c 

I'm not sure what you are asking here? Are block level snapshots faster? Yes. Do they take up less space than complete file backups? Yes, considerably less.

No, what I'm asking is how useful they would be for the regular files that change every day. They would be very useful for something like a big VM file or a database (which is what it was likely designed for), but other files that are loaded wholesale into an application are not changed on a block level - the entire file is changed.

Originally Posted by besson3c 

I say that if Apple is going to put the time into something as low level as the file system that they go all out and do it right, making it even better than ZFS. One of the other nice features of ZFS that we haven't discussed is the ability to apply filesystem feature updates to live file systems. So, Apple could launch this with a subset of the features we have been talking about, but they ought to make their file system *ready* for all of the goodness that is in ZFS, and they ought to think about building computers and software around these pretty significant features.

The problem I see is that ZFS is a memory hog that also likes its CPU time. That may not be a problem on a big Sun server, but it is an issue on an MBA, and even more so if it is going to be used on an iPhone. It doesn't need things like software RAID. It needs to be faster, threadsafe, and have a higher capacity ceiling, andt needs to include all the late hacks to HFS+ (journalling, hard-linked files, Spotlight, FileVault) to create a better foundation for the future. ZFS is a reasonable, but far from perfect, match for that. If Apple went for ZFS, they would end up with HFS+ on the iOS devices forever more - and that's where they make their money right now.

[besson3c]

Originally Posted by P 

So it's basically ReadyBoost then - except ReadyBoost has a limit on the number of drives you can add, I think. I would prefer something in the SATA controller, like SRT is. No need to burden the CPU further.

I would have to know more about ReadyBoost to confirm this, but it sounds like another difference is that this is a more software based solution, just as the fact that you no longer need a RAID controller is more a software focused approach.

Multiple HDDs are simply too big. As for any flash-based storage - we get to that below.

I've never been advocating multiple SATA drives.

This is simply not how they work. You can't even replace the entire HDD yourself without disassembling the entire computer. I think that the way forward is in flash. All SSDs have some extra area that can be reused as the flash cells wear out. This means that they guard against failure in the flash chips, but not in the controller. Right now there are a lot of early failures on stuff like this, but there is no reason for a controller to die any earlier than a CPU. Once testing has improved to CPU levels, this means that the hardware should be more reliable. Not fail-proof, but no worse than anything else.

I'm not following how this relates to your first sentence here?

Also note that when flash dies, it can still be read (just no written) for a few more months. This makes the parity checking in RAID irrelevant for this failure.

But when it is marked dead it can be detached from the pool.

Which is just another way of saying that it is RAID in software.

Yes. ZFS has fixed the RAID-5 hole, which is one of the big reasons why RAID-5/6/7 has always required hardware historically.

What you seem to be proposing is a lot of little flash drives, each with a connection of its own to the PCH and thus visible to the OS. It's just a lot of extra connections to a chip that already has a bunch of them, all so you can move the function from a dedicated chip to the CPU. Why would you want that?

What would the disadvantage be?

There is no reason to expect flash drives to up and die in the same manner that spinning discs do. They shouldn't die any more often than RAM chips do. Even if we want to do what you propose, that can better be handled in the SSD controller. After receiving that warning message, you would then make a fresh backup and then replace the entire drive.

You're right that Flash drives will not die like SATA drives have, but what about in setups that incorporate both SSD/Flash and SATA drives - SATA for more long term storage?

No, what I'm asking is how useful they would be for the regular files that change every day. They would be very useful for something like a big VM file or a database (which is what it was likely designed for), but other files that are loaded wholesale into an application are not changed on a block level - the entire file is changed.

I'm still not sure I'm following you... You mean new files that didn't exist prior?

The problem I see is that ZFS is a memory hog that also likes its CPU time. That may not be a problem on a big Sun server, but it is an issue on an MBA, and even more so if it is going to be used on an iPhone. It doesn't need things like software RAID. It needs to be faster, threadsafe, and have a higher capacity ceiling, andt needs to include all the late hacks to HFS+ (journalling, hard-linked files, Spotlight, FileVault) to create a better foundation for the future. ZFS is a reasonable, but far from perfect, match for that. If Apple went for ZFS, they would end up with HFS+ on the iOS devices forever more - and that's where they make their money right now.

I disagree. ZFS does like its memory, but for specific things, namely the CoW involved in handling large RAID setups and large pools of data. For 16 GB of data on your iPhone this would be gravy.

[besson3c]

I just thought of another major benefit in another thread - the ability to back peddle to previous OS versions. Here is what I wrote:

Once Apple's file system has block level snapshots what would be most ideal is if all of the OS guts be stored in their own pool. I use the term "pool" borrowing this term from a ZFS based file system where a pool is like a partition only it doesn't need to have a set quota as to how much disk space it can consume from your overall disk. You'd have a pool for your home directory, the OS guts, your application folder, and everything else that warrants a pool.

Before each OS update a snapshot is taken of the OS pool, which of an under 5 gig OS would take a few seconds. The result would be that you can safely back peddle to any OS version you want at any time... Want to run OS X 10.7.0? No problem... 10.6.3? No problem...

This whole back peddle of OS thing would be awesome, and something that has never easily been possible without a whole lot of extra work. Not only would this provide users with a safety net, but it would make support far easier for when you want to take a screenshot of the way something looked in a previous OS, or you want to see exactly what your user is seeing who is running an earlier version of the OS.

Obviously there are problems with reverting to an earlier OS on a live system, but if you can temporarily boot into your recovery partition, do the snapshot revert, then get back to work, that would be incredible! Add to this the whole Lion restore of how you last left everything: beyond incredible!

[P]

Originally Posted by besson3c 

I'm not following how this relates to your first sentence here?

"they" means Apple here.

Originally Posted by besson3c 

But when it is marked dead it can be detached from the pool.

I know how RAID works. My point was that when flash cells fail, you have time to make a backup after the fact.

Originally Posted by besson3c 

What would the disadvantage be?

It requires one controller chip per flash chip, instead of one per 8 or 10 or 16 or whatever. Each controller requires a connection (SATA or other) to the PCH, instead of just one. All of this costs money and - more importantly, in Apple's case - space inside the box.

Originally Posted by besson3c 

You're right that Flash drives will not die like SATA drives have, but what about in setups that incorporate both SSD/Flash and SATA drives - SATA for more long term storage?

What Apple device would have that? The MP and maybe, possibly, the iMac. That is a teeny tiny part of all the devices that Apple makes - and the MP can use a regular hardware RAID5.

Originally Posted by besson3c 

I'm still not sure I'm following you... You mean new files that didn't exist prior?

No, I mean a program like Excel. Say that you have an Excel sheet in which you enter one line of data every day. That doesn't mean that the file grows slowly from the end - the entire file changes every day, because that's how the file format works.

Originally Posted by besson3c 

I disagree. ZFS does like its memory, but for specific things, namely the CoW involved in handling large RAID setups and large pools of data. For 16 GB of data on your iPhone this would be gravy.

Sun recommends at least 1 GB RAM for good ZFS performance. No iOS device on the market today has that.

[Spheric Harlot]

It's "backpedal". It comes from bicycling, not from hawking goods ("peddling").

[besson3c]

Originally Posted by P 

"they" means Apple here.

I'm too lazy to go back and reread what you wrote now, sorry! Maybe later

I know how RAID works. My point was that when flash cells fail, you have time to make a backup after the fact.

So what would the difference be?

It requires one controller chip per flash chip, instead of one per 8 or 10 or 16 or whatever. Each controller requires a connection (SATA or other) to the PCH, instead of just one. All of this costs money and - more importantly, in Apple's case - space inside the box.

I see what you mean... This is a fair point. Still, I can see Apple building their desktop computers this way, that is pretty low dangling fruit when the space problem has been rectified. I can see Apple finding a way to accommodate this now though, they always seem to find creative ways to cram new ganja into a tiny space

What Apple device would have that? The MP and maybe, possibly, the iMac. That is a teeny tiny part of all the devices that Apple makes - and the MP can use a regular hardware RAID5.

I don't know, I guess it's kind of silly to speculate how the space factor would play out, but the potential is there if not for the present for the future.

The other factor to consider with a software based RAID solution over hardware is the ease of OS awareness. What was cool about the Sun workstations was that Solaris was aware of stuff that was going on in the hardware. It would be awesome if Apple could build in a notification of a drive failure into the OS without having to sort out these sorts of details with the hardware vendor. This also applies to the state of affairs today with a single drive. All we have is SMART notifications which seem to be very reliable, but the SMART status needs to be polled, and AFAIK there are different hardware thresholds which make this complicated.

No, I mean a program like Excel. Say that you have an Excel sheet in which you enter one line of data every day. That doesn't mean that the file grows slowly from the end - the entire file changes every day, because that's how the file format works.

I think that entire file changes because new blocks get tacked on and removed in random sequence, but there are probably several individual blocks that remain consistent. I don't know how ZFS deals with keeping blocks in sequence.

Sun recommends at least 1 GB RAM for good ZFS performance. No iOS device on the market today has that.

Sun catered towards their Solaris 10 market, and to a much lesser extent OpenSolaris, and neither were designed to run on anything but servers.

ZFS will make use of RAM as a buffer (another entirely separate layer of buffer separate from the physical hardware which I neglected to mention earlier), as well as for distributing checksums to different drives and stuff.

However, if you don't need or want a buffer, or you aren't distributing data to a fleet of drives as ZFS is most commonly used, there is no reason why ZFS couldn't run in considerably less RAM. Even if this the case today, it can be with optimization for other use cases - perhaps tweaking for a smaller footprint rather than faster performance. At the end of the day though, the things that ZFS needs RAM for are very specific and manageable.

[besson3c]

Originally Posted by Spheric Harlot 

It's "backpedal". It comes from bicycling, not from hawking goods ("peddling").

Thanks, you're right

[Spheric Harlot]

(sorry, no offense — just a pet peeve)

[besson3c]

Originally Posted by Spheric Harlot 

(sorry, no offense — just a pet peeve)

None taken, thanks for pointing it out...

[mduell]

Originally Posted by P 

Sun recommends at least 1 GB RAM for good ZFS performance. No iOS device on the market today has that.

Sun isn't thinking about pools smaller than terabytes when they make that recommendation. No iOS device has even 0.1TB.

[besson3c]

Originally Posted by mduell 

Sun isn't thinking about pools smaller than terabytes when they make that recommendation. No iOS device has even 0.1TB.

Exactly, and in an IOS device you could just add another drive as a backup mirror to give you two drives in total, which probably requires a lot less RAM than a RAID-Z/5 deal. With the limited space it has, you might only be able to fit a second drive in there.

I know it's kind of crazy to think about this sort of redundancy in your iPhone, but maybe for the iPad post iOS 5 where data starts to live primarily on that device and is not shuttled to it from your Mac, maybe thinking of a spare drive isn't the craziest idea in the world even considering Flash drives being far less damage prone than SATA drives?

What is the failure rate of Flash drives anyway?

If it is crazy small, maybe mirrors are pointless, but I still like the idea of upgrading to the capacity you want with low cost, small sized trios of Flash drives. It sort of takes the philosophy of CPUs into account: instead of making single core CPUs with the fastest clock speed as possible get your performance by adding more cores, only in this case this would come in the form of not waiting for the HD vendors to up the ante on how large a single drive can be, but instead getting your capacity from the combined space of two drives in a three drive RAID-5 set.


I might be going overboard here, but at any rate I think that a modern file system will at least stimulate some conversation on how Apple best build their machines.

[OreoCookie]

Originally Posted by P 

So it's basically ReadyBoost then - except ReadyBoost has a limit on the number of drives you can add, I think.

The appeal of ZFS is not single technologies, but that all of them are integrated in one package. ZFS can use SSDs as a `cache' in much more sophisticated scenarios than ReadyBoost ever could. Obviously, this doesn't matter for the class of machines we're thinking of. On the other hand, even though you can replicate most of the functionality of ZFS by other means (ReadyBoost, hardware/software RAIDs, etc.), since ZFS offers that in an integrated package which is aware of anything in between the physical devices and the files, I would still expect that you can potentially achieve better performance and flexibility.

Originally Posted by P 

The problem I see is that ZFS is a memory hog that also likes its CPU time. That may not be a problem on a big Sun server, but it is an issue on an MBA, and even more so if it is going to be used on an iPhone.

I don't think it's a problem for non-iPods, -iPads and -iPhones, especially since it's a problem that will be ameliorated over time. ZFS may be problematic for this iDevices, though, I agree with you on that.

[P]

Originally Posted by besson3c 

So what would the difference be?

It means that you don't need the parity part of RAID. All you need is an online spare, ie an extra chip. This decreases the amount of math the controller (or the CPU, if it's in software) needs to do.

Originally Posted by besson3c 

I see what you mean... This is a fair point. Still, I can see Apple building their desktop computers this way, that is pretty low dangling fruit when the space problem has been rectified. I can see Apple finding a way to accommodate this now though, they always seem to find creative ways to cram new ganja into a tiny space

I guess you could do something with multiple 2.5" drives, but to put multiple 2.5" drives instead of single 3.5" drive would be very expensive.

I'm fairly certain that Apple is moving towards all flash, all the time, and eventually removing even the 3.5" HDD in the iMac. Although that is at least 5 years away, that is the sort of perspective you need to have when designing a file system. HFS+ has been around for what, 13 years now? The original HFS also lived for 13 years (so it's just about due for replacement!).

Originally Posted by besson3c 

The other factor to consider with a software based RAID solution over hardware is the ease of OS awareness. What was cool about the Sun workstations was that Solaris was aware of stuff that was going on in the hardware. It would be awesome if Apple could build in a notification of a drive failure into the OS without having to sort out these sorts of details with the hardware vendor. This also applies to the state of affairs today with a single drive. All we have is SMART notifications which seem to be very reliable, but the SMART status needs to be polled, and AFAIK there are different hardware thresholds which make this complicated.

Making the OS aware of something is interesting, but Apple has been moving in the other direction lately (remember that the G5s used to control the cooling from the OS?). A dedicated RAID controller also lets you know when a drive dies.

SMART is a crapshoot at best (when it says that the drive is failing, it's certainly failing, but it may not warn you early enough). There is certainly more work to do there.

Originally Posted by besson3c 

Sun catered towards their Solaris 10 market, and to a much lesser extent OpenSolaris, and neither were designed to run on anything but servers.

Sun used to have a big workstation market, but OK, same thing here I guess.

Originally Posted by besson3c 

ZFS will make use of RAM as a buffer (another entirely separate layer of buffer separate from the physical hardware which I neglected to mention earlier), as well as for distributing checksums to different drives and stuff.

However, if you don't need or want a buffer, or you aren't distributing data to a fleet of drives as ZFS is most commonly used, there is no reason why ZFS couldn't run in considerably less RAM. Even if this the case today, it can be with optimization for other use cases - perhaps tweaking for a smaller footprint rather than faster performance. At the end of the day though, the things that ZFS needs RAM for are very specific and manageable.

I wonder about how small you could make that footprint. Remember that HFS could run in 128K and still leave room for the rest of the OS, and HFS+ retains a lot (too much) of that. Whatever replacement we get, it's going to expand from that. A big server filesystem doesn't seem like the best bet.

[kikkoman]

Originally Posted by besson3c 

With what I'm describing the following would be possible:

- Say, 6 very tiny removable drives that can be of whatever size you want
- When one dies everything continues on, you are alerted
- When you want to upgrade you simply replace the drives with larger ones one at a time, using the mirror to copy the data back to the drive. You can do this on a live system

Yes I realize that there are many issues with making this grandma friendly and yadda yadda, but I'm just trying to suggest that there is a ton of potential here, and what we are used to today kind of sucks.

That's pretty much how a Drobo works.

[besson3c]

Originally Posted by kikkoman 

That's pretty much how a Drobo works.

Exactly, this would be the same sort of thing except with a non-priorietary RAID system