[Super Mario]

Apple to use Sun's ZFS in Leopard | The Register

What are the benefits apart from security, snapshots (Time Machine) and support for very large volumes in the very far future?

[OreoCookie]

It's a new, powerful filesystem that is very future-proof. You can easily add new drives to a volume without the need to reformat. It also allows the user to easily setup failure-proof disk arrays. I'm not sure how much of this will be used right away, but these are just a few possibilities. For me as a mobile Mac user, the RAID features will probably not be relevant for a while, but this is a big plus for Pro Mac users.

[Brass]

You can grow and shrink file systems dynamically, on the fly. Previously most file system could grow on the fly, but shrinking on the fly is something new (so far as I know).

Built in RAID at the file system level, avoids the need for separate utilities for managing disk redundancy.

(Yay... Camino finally does spell checking in web browser windows in 1.5. About time!!!)

[Brass]

oh yeah... it's supposed to be virtually fault-proof... self detecting, self repairing. Not just journalling, but avoiding the need for journalling (at least in the traditional journalling sense).

[Big Mac]

Sounds cool. I hope it's at least on par with HFS as far as metadata is concerned, though.

[Super Mario]

Its Wikipedia article says that its background processes slow the file system down.

[besson3c]

I also read something about how it compresses/decompresses data on the fly, making it a better file system for intensive I/O.

[Super Mario]

Originally Posted by besson3c 

I also read something about how it compresses/decompresses data on the fly, making it a better file system for intensive I/O.

If you read the Wikipedia article it also says that when ZFS does that there is a speed penalty. It is very slow at that. Apple might have fixed it knowing them. Or it will be fixed incrementally.

[besson3c]

Originally Posted by Super Mario 

If you read the Wikipedia article it also says that when ZFS does that there is a speed penalty. It is very slow at that. Apple might have fixed it knowing them. Or it will be fixed incrementally.

True, it would push the load off to the CPU, but this might be nice in OS X Server in environments that are I/O bound (Exchange would be a good example of this if there was a Unix version, as it is extremely I/O taxing)

[C.A.T.S. CEO]

Awesome, but what will be its naming convention? (ie Mac OS Extended = HFS)

Mac OS Z?

EDIT: Does this mean that the new boot disk format will be ZFS and not HFS+?

[chabig]

Originally Posted by C.A.T.S. CEO 

EDIT: Does this mean that the new boot disk format will be ZFS and not HFS+?

Yes, that's exactly what it means.

[Super Mario]

Originally Posted by besson3c 

True, it would push the load off to the CPU, but this might be nice in OS X Server in environments that are I/O bound (Exchange would be a good example of this if there was a Unix version, as it is extremely I/O taxing)

We'll have to see if the performance issue has been addressed. No Linux version is forthcoming so nah nah

[Thinine]

Remember, this is a very new FS (launched in 2004), so it's likely that there's still room for optimization. But many of the performance "problems" are offset by gains in performance in other areas (copy on write, read ahead, variable block sizing).

[OreoCookie]

That's not sure. According to the project's homepage, there is no boot support … yet. But this doesn't have to be a problem (see my post above). And perhaps Apple surprises us announcing that it will add boot support to ZFS, who knows

Regarding the extra load for cpus, I don't think this will be a big deal. A few years ago, perhaps, but cpu power is growing constantly. A few years back, it took 30-40 % of a P3 450's cpu power to saturate one 100 MBit ethernet interface. Now, it's a fraction of that. Plus, features (such as RAID-Z) will only be taxing, if you actually use them.

[Super Mario]

What does Microsoft use to make system snapshots?

[CharlesS]

Originally Posted by Big Mac 

Sounds cool. I hope it's at least on par with HFS as far as metadata is concerned, though.

It should be. From what I can find on Wikipedia, ZFS supports both alternate data streams and extended file attributes, which should make it possible to emulate the resource fork and all the rest of the legacy HFS+ metadata.

The one thing I'm not sure about is the ability to reference files by ID, so that things like FSRefs and aliases can keep pointing to a file even if it moves in the file hierarchy. Does anyone with more info on ZFS know if something like this could be implemented?

Of course, I guess that once this makes it into a developer preview, we'll know. Thing is, once that happens I won't be able to tell you about it anymore...

[Brass]

Note that Sun doesn't really have any idea if Apple is going to make this the default FS in Leopard any more than we do, as per this Sun staff Blog: Marc Hamilton's Weblog : Weblog

What Sun do know is the same as what we know:

1. Apple ARE including ZFS in Leopard, at least as an option for non-boot volumes.

2. Apple are going to make ZFS the default FS in Mac OS X one day (well, we can deduce this logically, even if we can't be certain).

3. Apple are planning to announce all this at WWDC next week.

Nobody knows if they are really going to make it the default OS for Leopard or if they're going to wait for the next major release (or even later). I think, the sooner the better, but I'm not holding my breath.

[Brass]

BTW, benchmarks of ZFS show it to be much faster than most other common file systems, for in many tests.

[Peter]

roll on WWDC.

[Super Mario]

Originally Posted by Brass 

BTW, benchmarks of ZFS show it to be much faster than most other common file systems, for in many tests.

I can only find benches against ext2, ffs and ufs.

[P]

If I understand correctly, you can't boot from ZFS because the code to mount a ZFS volume is big enough that you can't fit it in the firmware on Sun's machines. Either you optimize the code, which is what Sun seems to be working on, or you add more memory. It could be that Apple has more space to do that in their EFI implementation, but I doubt it.

Saying that ZFS is the default filesystem in Leopard is overstating it - that's not what Schwartz said. More likely, he meant that ZFS support is in Leopard - which we already knew - and that it's the future filesystem of choice for Mac OS X, as in 10.6 or thereabouts.

[P]

Originally Posted by Super Mario 

I can only find benches against ext2, ffs and ufs.

You can safely assume that all of them are faster than HFS+. Speed is not exactly HFS strong suit.

[OreoCookie]

My information wasn't up to date, in the meantime, boot support has been added to ZFS

[chris v]

Dumb question, from a causal user:

If 10.5 does indeed ship with this new file system, is it likely that we're all going to have to backup, wipe, reformat & re-install everything, or will it convert from HFS+ while retaining the data on the drive?

[P]

Well, it will certainly support letting you stay with HFS+...

If it will let you convert the FS without reformatting is not known. Apple has not supported doing that with any other filesystem transition, but such tools exist for other OSes - notably, MS has converters from various FAT versions to NTFS and in Win98 also supported FAT16->FAT32. As I said, I don't think that ZFS will be the default in Leopard but rather in a later version, so I don't think that this will be an issue yet.

[Eriamjh]

This is kinda cool. Check out this link.

ZFS--the last word in file systems.
The breakthrough file system in Solaris 10 delivers virtually unlimited capacity, provable data integrity, and near-zero administration.
Most system administrators take the limitations of current file systems in stride. After all, file systems are what they are: vulnerable to silent data corruption, brutal to manage, and excruciatingly slow.

ZFS, the dynamic new file system in Sun's Solaris 10 Operating System (Solaris OS), will make you forget everything you thought you knew about file systems. ZFS will be available on all Solaris 10 OS-supported platforms, and all existing applications will run with it. Moreover, ZFS complements Sun's storage management portfolio, including the Sun StorEdge QFS software, which is ideal for sharing business data.

"If you're willing to take on the entire software stack, there's a lot of innovation possible."

Jeff Bonwick
Distinguished Engineer
Chief Architect of ZFS
Sun Microsystems, Inc.
"We've rethought everything and rearchitected it," says Jeff Bonwick, Sun distinguished engineer and chief architect of ZFS. "We've thrown away 20 years of old technology that was based on assumptions no longer true today."

ZFS is supported on both SPARC and x86 platforms. More important, ZFS is endian-neutral. You can easily move disks from a SPARC server to an x86 server. Neither architecture pays a byte-swapping tax due to Sun's patent-pending "adaptive endian-ness" technology, which is unique to ZFS. And you don't have to worry about migration. Sun continues to support the UFS file system.

ZFS meets the needs of a file system for everything from desktops to data centers. Designed with the administrator in mind, ZFS is the only self-healing, self-managing general-purpose file system. It offers:

Simple administration
ZFS automates and consolidates complicated storage administration concepts, reducing administrative overhead by 80 percent.
Provable data integrity
ZFS protects all data with 64-bit checksums that detect and correct silent data corruption.
Unlimited scalability
As the world's first 128-bit file system, ZFS offers 16 billion billion times the capacity of 32- or 64-bit systems.
Blazing performance
ZFS is based on a transactional object model that removes most of the traditional constraints on the order of issuing I/Os, which results in huge performance gains.
All About Solaris 10 on Sun.com

The Solaris 10 OS, the upcoming release of the industry's leading UNIX platform, integrates powerful new capabilities to deliver extreme levels of performance, availability, and security. ZFS in the Solaris 10 OS complements Sun's existing storage management portfolio, including the Sun StorEdge QFS software, which is ideal for accessing business data in a shared environment. In addition to the ZFS file system reviewed in this article, the Solaris 10 OS includes these revolutionary technologies:

N1 Grid Containers technology delivers a breakthrough approach to system virtualization and resource utilization.

DTrace is a comprehensive dynamic tracing framework that concisely answers questions about system behavior.

Solaris 10 also includes technology that enables running a range of Linux applications at near-native speeds.

The Predictive Self-Healing capabilities in the Solaris 10 OS enable Sun systems to accurately predict component failure and fix what's wrong.
ZFS achieves its impressive performance through a number of techniques:

Dynamic striping across all devices to maximize throughput
Copy-on-write design makes most disk writes sequential
Multiple block sizes, automatically chosen to match workload
Explicit I/O priority with deadline scheduling
Globally optimal I/O sorting and aggregation
Multiple independent prefetch streams with automatic length and stride detection
Unlimited, instantaneous read/write snapshots
Parallel, constant-time directory operations
Just as it dramatically eases the suffering of system administrators, ZFS offers relief for your company's bottom line. Because ZFS is built on top of virtual storage pools (unlike traditional file systems that require a separate volume manager), creating and deleting file systems is much less complex. Not only does this eliminate the need to pay for volume manager licenses and allow for single support contracts, it lowers administration costs and increases storage utilization.

ZFS appears to applications as a standard POSIX file system--no porting is required. But to administrators, it presents a pooled storage model that eliminates the antique concept of volumes, as well as all of the related partition management, provisioning, and file system sizing problems. Thousands--even millions--of file systems can all draw from ZFS' common storage pool, each one consuming only as much space as it needs. The combined I/O bandwidth of all of the devices in that storage pool is always available to each file system.

Get Into the Pool
With traditional volumes, storage is fragmented and stranded. With ZFS' common storage pool, there are no partitions to manage. The combined I/O bandwidth of all of the devices in a storage pool is always available to each file system.


Delivering Near-Zero Administration
Two of Sun's goals with ZFS were to do away with many complicated storage administration concepts and automate many common administrative chores.
For example, creating a storage pool, growing a pool, or adding or removing a file system can all be done with a single, simple command--instead of the multistep process (format, newfs, edit/etc/vfstab, and so on) found in traditional file systems and volume managers.

Consider this case: To create a pool, to create three file systems, and then to grow the pool--5 logical steps--5 simple ZFS commands are required, as opposed to 28 steps with a traditional file system and volume manager.

Moreover, these commands are all constant-time and complete in just a few seconds. Traditional file systems and volumes often take hours to configure. In the case above, ZFS reduces the time required to complete the tasks from 40 minutes to under 10 seconds.

ZFS' command-line interface drastically simplifies administration. It is task-oriented, letting administrators express the tasks they want to accomplish instead of having to memorize or look up cryptic commands.

"You don't have to worry about the details of what's going on with your disks, your storage, or your file systems," explains Bonwick. "You add disks to your storage pool, file systems consume space automatically as they need it, and administrators don't have to get involved.

"We wanted to design an integrated system from scratch. If you're willing to take on the entire software stack, there's a lot of innovation possible."

Taking the Guesswork Out of Data Integrity
Data can be corrupted in a number of ways, such as a system error or an unexpected power outage, but ZFS removes this fear of the unknown. ZFS prevents data corruption by keeping data self-consistent at all times. All operations are transactional. This not only maintains consistency but also removes almost all of the constraints on I/O order and allows changes to succeed or fail as a whole.
All operations are also copy-on-write. Live data is never overwritten. ZFS writes data to a new block before changing the data pointers and committing the write. Copy-on-write provides several benefits:

Always-valid on-disk state
Consistent, reliable backups
Data rollback to known point in time
"We validate the entire I/O stack, start to finish, no guesswork involved. It's all provable data integrity," says Bonwick.

Administrators will never again have to run laborious recovery procedures, such as fsck, even if the system is shut down in an unclean fashion. In fact, Solaris Kernel engineers Bill Moore and Matt Ahrens have subjected ZFS to more than a million forced, violent crashes in the course of their testing. Not once has ZFS lost data integrity or leaked a single block.

Additionally, ZFS is the only file system that conducts end-to-end 64-bit checksums on all data to prevent silent data corruption. When any data is read, the checksum is verified to ensure that the data that the application wrote is what is returned.

"The cost of doing something like a checksum is no longer prohibitive. Burning a small percentage of the CPU to know that data is intact is a price that administrators would gladly pay," says Moore.

As part of Sun's quest to build truly self-healing systems (see the September 7 Sun.com feature), ZFS can self-heal data in a mirrored or RAID configuration. When one copy is damaged, ZFS detects it via the checksum and uses another copy to repair it.

No competing product can do this. Traditional mirrors can only handle total failure of a device. They don't have checksums, so they have no idea when a device returns bad data. So even though mirrors replicate data, they have no way to take advantage of it. By contrast, the end-to-end checksums in ZFS allow it to find and fix bad blocks--with nineteen nines certainty--automatically.

Creating Immense Capacity
Sun engineers wondered if the 64-bit capabilities of current file systems will continue to suffice over the next 10 to 20 years. Their answer was no. If Moore's Law holds, in 10 to 15 years people will need a 65th bit. As a 128-bit system, ZFS is designed to support more storage, more file systems, more snapshots, more directory entries, and more files than can possibly be created in the foreseeable future.
This scalability also means that storage can be dynamically added to or removed from storage pools without interrupting services, providing new levels of flexibility and availability for globally accessed application services.

To efficiently use all of this capacity, file systems grow and shrink automatically as users add or remove data. Administrators can set quotas to limit space consumption and reservations to guarantee future availability of space. ZFS also provides compression to reduce disk space and I/O bandwidth requirements.

Logically, the next question is if ZFS' 128 bits is enough. According to Bonwick, it has to be. "Populating 128-bit file systems would exceed the quantum limits of earth-based storage. You couldn't fill a 128-bit storage pool without boiling the oceans."

It looks like the best way to support Time Machine.

[Chuckit]

Originally Posted by Eriamjh 

It looks like the best way to support Time Machine.

Yet strangely, Time Machine has already been implemented in a completely unrelated way for in all the betas up to this point. I wonder if they'll change the implementation now.

[krove]

Time Machine's original implementation was shown to work at the file-level, which would prove resource-taxing on very large files that change (aka, media files, big photoshop files, etc).

ZFS's implementation, on the other hand, works at the block-level. This would be much more efficient for large files so that if only part of a file changed, only the block(s) containing that data would need to be re-written elsewhere and marked as the latest revision. This was my understanding anyway. Handling large files in this manner would probably be the biggest boon for Time Machine implemented in ZFS vis-à-vis technical elegance and efficiency.

On referencing files by ID, it wouldn't be very Mac-like if this feature were not present. Being able to move files around in the Finder while they are open is just expected on the Mac. It just works!

[krove]

Apple probably won't throw away the Time Machine work done for HFS+, but certainly they could easily add Time Machine support for ZFS, thus supporting HFS+ AND ZFS.

[krove]

Originally Posted by Brass 

1. Apple ARE including ZFS in Leopard, at least as an option for non-boot volumes.

2. Apple are going to make ZFS the default FS in Mac OS X one day (well, we can deduce this logically, even if we can't be certain).

3. Apple are planning to announce all this at WWDC next week.

All your base ARE belong to Apple.

Bad joke, but what is up with the "Apple are" business? Apple is singular!

[Chuckit]

Originally Posted by krove 

All your base ARE belong to Apple.

Bad joke, but what is up with the "Apple are" business? Apple is singular!

I smell a Brit.

[TheoCryst]

Brits refer to organizations/corporations/etc as plural, whereas Americans use the singular.

[besson3c]

Originally Posted by Brass 

You can grow and shrink file systems dynamically, on the fly. Previously most file system could grow on the fly, but shrinking on the fly is something new (so far as I know).

You can grow and shrink LVM partitions with LVM under Linux too, just FYI...

[OreoCookie]

Sure, besson, but LVM have been a different beast entirely. ZFS brings these features down to the filesystem level whereas LVM is volume manager (well, there are others, too). As far as I know, LVM can't do RAID1 or RAID5, either.

To be honest, the appeal of ZFS aren't single features (that might be possible with additional software and other filesystems), but the combination of features.

[Eriamjh]

ZFS might be a good way to support added flash memory for those fast-boot machines.

[Brass]

Originally Posted by krove 

All your base ARE belong to Apple.

Bad joke, but what is up with the "Apple are" business? Apple is singular!

Yeah... you got me there! I'm so ashamed of myself. I tend to think of a company as a group of people therefore often write about it/them as though it/they were plural. My humble apologies to the grammar police.

I believe that this is even considered correct grammar in the country in which I'm typing, but I can't remember for sure.

Are you from the American grammar police, or from the Interpol grammar police? 'cause if American, I really don't give a stuff.

[Brass]

Originally Posted by besson3c 

You can grow and shrink LVM partitions with LVM under Linux too, just FYI...

And this points out why ZFS is so good... it removes the need for volume management solutions (such as LVM) and manages it all within the file system layer alone.

[krove]

Originally Posted by Brass 

Yeah... you got me there! I'm so ashamed of myself. I tend to think of a company as a group of people therefore often write about it/them as though it/they were plural. My humble apologies to the grammar police.

I believe that this is even considered correct grammar in the country in which I'm typing, but I can't remember for sure.

Are you from the American grammar police, or from the Interpol grammar police? 'cause if American, I really don't give a stuff.

Sheesh. I didn't realize that my poking fun would hit such a nerve. I truly had never heard this usage before. Sorry to have offended you.

[besson3c]

Originally Posted by Brass 

And this points out why ZFS is so good... it removes the need for volume management solutions (such as LVM) and manages it all within the file system layer alone.

Definitely!

[Brass]

Originally Posted by krove 

Sheesh. I didn't realize that my poking fun would hit such a nerve. I truly had never heard this usage before. Sorry to have offended you.

Don't stress... I wan't actually offended. I was just commenting off-handedly, and purely all in good fun. (Note the ).

[nerd]

So no mention of ZFS in the Keynote?

[TheoCryst]

No mention, but that doesn't mean anything either way.

[OreoCookie]

I'll ask some friends tonight, they are on the WWDC. I guess Steve wanted the people to focus on the UI changes and the features that are closer to the surface.

[krove]

You'd think they would have at least put it on the website.

[Ado]

Looks like Apple have have borrowed more from Sun than just their file system...

[ginoledesma]

I'm eagerly waiting to hear from the devs at WWDC as well. If Leopard boots off ZFS at all, I'd be truly impressed.

[shinykaro]

Figures, as soon as I finish deciphering the Wikipedia entry.. I was actually starting to get excited about ZFS, but why would Apple bother with moving over if the inability to boot hasnt't been addressed? Wouldn't that unnecessarily complicate things? They're probably just waiting until it's boot ready.

Don't forget the little disclaimer on Apple's website in the Leopard section: "All features referenced in the Mac OS X Leopard website are subject to change."

[Chuckit]

Originally Posted by Ado 

Looks like Apple have have borrowed more from Sun than just their file system...

But Ben told us the Looking Glass was inoperable!

[workerbee]

According to Apple's Brian Croll, no ZFS in 10.5.0:
Apple Says No Sun File System For Leopard -- Mac OS X Leopard -- InformationWeek

Sorry if this is old news... haven't seen it in this thread.

[passmaster16]

Originally Posted by workerbee 

According to Apple's Brian Croll, no ZFS in 10.5.0:
Apple Says No Sun File System For Leopard -- Mac OS X Leopard -- InformationWeek

Sorry if this is old news... haven't seen it in this thread.

It appears that the initial information from the Apple executive about ZFS not being in Leopard was erroneous. See the editor comment in the same link above.


As to the news, it seems that Croll mispoke a couple of times when asked about ZFS in Leopard. Despite direct questions about Sun CEO Schwartz's claims that ZFS is there, Croll flatly denied the reports to two of our reporters in a 1:1 interview.

An Apple spokesperson called us Tuesday seeking to clarify Croll's statement. Croll was apparently supposed to indicate that ZFS would be available as a limited option, but not as the default file system."

We are now writing a separate story to note Apple's mis-statement and hopefully to reveal more about how ZFS would work in Leopard.

Michael Singer
InformationWeek - West Coast Editor