[besson3c]

I've been testing and evaluating ZFS over the last several days, and I've been thoroughly impressed. ZFS is one of those products where there is so much attention to little details that it is literally a joy to use. This is Sun's masterpiece!

I'm only testing ZFS in an OpenSolaris VM for now so I can't report really back on performance and resource consumption, but from what I can gather there are a lot of people who consider ZFS to be highly stable, some companies and universities using it on production servers, and as you know it is the default file system in OpenSolaris. That being said, here is why I've been so impressed with it...

ZFS seems to sort of combine LVM and hardware RAID into a single product. It even includes a mini-NFS and iSCSI client for very easy file system sharing, it will automount all of your file sytems at boot so that you don't have to muck around with the Solaris equivalent to fstab, and the interface is very thoughtful and easy-to-use with so many little thoughtful features and little touches that make life easy. In fact, it is so straight forward that it would be very easy to wrap a configuration GUI around, I think, and hence the subject of this message...

One of the biggest problems with consumer PCs are the fact that the machine is dependent on a single disk which are prone to going bad. Given the availability of high capacity 2.5" SATA disks and Flash increasing in capacity and reducing in size constantly, isn't it only a matter of time before it makes sense to ship computers (even laptops) with multiple disks? It is so incredibly easy to setup a mirror of your filesystem in ZFS, and with a hot spare in your storage pool you can automatically set it up to failover and notify you. Perhaps someday storage in a PC will be as simple as it is with a digital camera or something? I don't see how this sort of thing would be that far off...

The other incredibly nice feature of ZFS that is a huge net gain for almost all of us is its snapshot ability. Snapshots are block level, meaning that whenever a file is changed you don't need to keep a complete copy of that file, a mess of hard links and all of that sort of stuff you'd find in a solution such as Time Machine or many of the backup snapshot solutions we concoct with shell scripts. Taking snapshots is *incredibly fast*, incredibly easy to manage, and you can browse these snapshots without having to create new inodes and hard links - these are, as I understand it, simply very low level references to files, but not files themselves. Still, you can copy individual files from snapshots as you would copy any file, or restore an entire file system from a snapshot. This is so easy to do and takes up so little disk space that not only is this a great way to do backups, but it is also a great way to create yourself an "oh ****" safety net should you want to back out of an update that has gone bad, etc.

I could go on about the many features and nice aspects to using ZFS, features like self healing, zpool iostat, etc. It's pretty cool stuff, but I'll leave my gushing for another time. Like I said, I'm just speaking to the features here...

Still, I wanted to see what you guys thought about the future of modern file systems such as ZFS? I realize that it will be a long time before BTRFS is ready, perhaps even longer for this sort of thing to make its way to OS X and/or Windows, but now that the bar as been set I'm sure it is only a matter of time before the copycats emerge. However things end up, who owns what, licenses, product names, etc. This seems like a very big deal to me. It is definitely changing the way I think of my data, and I'm surprised how easy it is to setup a storage appliance like the one I'm testing.

OpenSolaris is actually pretty nice. There are a few Sun-made things included, but a lot of it seems pretty familiar to me and more Linux-like than previous versions. I would say that if you are looking for a storage solution that OpenSolaris shouldn't really be a deterrent. FreeBSD 8 is in RC3, and it will include full ZFS support (previous versions include partial support), so this is another option.

Any other ZFS users/enthusiasts here?

[CharlesS]

Originally Posted by CharlesS 

Please stop bringing up awesome features of ZFS. It’s depressing.

     

[besson3c]

Well, it should cheer you up to know that it is really easy to create a little storage appliance for yourself using OpenSolaris or FreeBSD 8. If you just wanted to make this available on a private network you wouldn't really have to worry about constant software updates and stuff like that. Like I said, NFS sharing is so incredibly simple that I honestly think you will be spending more time picking out hardware than you would learning enough about ZFS/Solaris/FreeBSD to use as an appliance. It would sort of be like buying a new router...

On the hardware front, finding cheap motherboards with 6 internal SATA ports is quite easy. I found this PCI SATA controller that allows me to connect 8 additional SATA drives. I'm looking at 1U and 2U servers, but if you could find a tower with enough drive bays (hot swap or not) you could build yourself a nice little storage appliance. I found a couple of 1U chassis that will store 16 2.5" disks, so going with 2.5" disks is an option too should you want to keep the size of this thing down, assuming this sort of thing is common in other form factors too.

Unless you wanted to use ZFS compression you wouldn't need a fast processor, video card, and more than a couple gig of RAM. You could build yourself a machine for *well* under $1000, the greatest costs would be your disks.

You cannot add disks on-the-fly to RaidZ pools and have that space available, but you can standard mirrored storage pools. So, you could just add disks in pairs and increase your storage that way. Or, (and this is what I will probably do), leave a few spare bays so that it would be possible to create new RaidZ pools and/or replace your pool with larger disks as needed. Most people seem to advocate doing the former, so long as it is not vital that everything doesn't have to be one giant disk.

The other benefit to these sorts of storage pools is that it will take much more to saturate the I/O of these disks, depending on the width of your stripe.



This probably sounds more complicated than it is. Trust me, it is actually *very* easy to create yourself a simple little storage setup.

[11011001]

Yup:
5 TBs on 8 drives on the last build of ZFS under 10.5.
A Solaris ZFS server at work.

[lexapro]

Zero Force Insertion

[kupan787]

Originally Posted by 11011001 

Yup:
5 TBs on 8 drives on the last build of ZFS under 10.5.

Mind sharing how you have this setup? I am looking to build a large "storage appliance" (as besson3c) called it. Currently I have a drobo, with 5 TB of disks plugged into it, which is hooked to an always on Mac Mini. Mainly it is used for TV Shows and Movies (running Plex on the mini as a HTPC), but the mini is on the gigiabit network, and provides a good dumping ground for everyone in the house.

But If I could get something running with ZFS, and then use the drobo as a backup of my ZFS "drive", I have the best of both worlds...all the features and speed of ZFS, and the redundancy and ease of the drobo.

[Paco500]

WAFL rip-off.

[besson3c]

Originally Posted by kupan787 

Mind sharing how you have this setup? I am looking to build a large "storage appliance" (as besson3c) called it. Currently I have a drobo, with 5 TB of disks plugged into it, which is hooked to an always on Mac Mini. Mainly it is used for TV Shows and Movies (running Plex on the mini as a HTPC), but the mini is on the gigiabit network, and provides a good dumping ground for everyone in the house.

But If I could get something running with ZFS, and then use the drobo as a backup of my ZFS "drive", I have the best of both worlds...all the features and speed of ZFS, and the redundancy and ease of the drobo.

If you don't mind me answering this, I know that one answer is a cheap home-brew PC. Like I said, motherboards that support 6 SATA drives are pretty standard, and there are many PCI cards that will allow for even more. 6 1 TB SATA drives = 5 TB of usable space in a Raid-Z configuration.

[kupan787]

Originally Posted by besson3c 

If you don't mind me answering this, I know that one answer is a cheap home-brew PC. Like I said, motherboards that support 6 SATA drives is pretty standard, and there are many PCI cards that will allow for even more. 6 1 TB SATA drives = 5 TB of usable space in a Raid-Z configuration.

I've read elsewhere that OpenSolaris can be picky when it comes to hardware. And I am really looking for something that is somewhat maintenance free once setup. Have you only played with this in a VM, or have you started picking out components for hardware? Do you know if ZFS expects all drives to be the same size, or can I mix and match?

[besson3c]

Originally Posted by kupan787 

I've read elsewhere that OpenSolaris can be picky when it comes to hardware. And I am really looking for something that is somewhat maintenance free once setup. Have you only played with this in a VM, or have you started picking out components for hardware? Do you know if ZFS expects all drives to be the same size, or can I mix and match?

I haven't yet purchased my hardware for it, but about the only piece of hardware I'm looking at that would require a non-standard driver is the 8 port SATA controller PCI card, and it is listed in the Solaris hardware compatibility list as compatible. Everything else (RAM, SATA drives, CPU, power supply, etc.) I expect to work. The extra 8 port SATA controller is also optional if you are happy with 6 drives.

I should take back what I said about 6 drives on a 6 port motherboard giving you 5 TB of data though. You'll need at least one drive not in the RAID-Z pool to boot the machine with.

You can mix and match sizes, I'm pretty sure, but the pool's performance will be limited by the slowest drive.

[11011001]

Originally Posted by kupan787 

Mind sharing how you have this setup? I am looking to build a large "storage appliance" (as besson3c) called it. Currently I have a drobo, with 5 TB of disks plugged into it, which is hooked to an always on Mac Mini. Mainly it is used for TV Shows and Movies (running Plex on the mini as a HTPC), but the mini is on the gigiabit network, and provides a good dumping ground for everyone in the house.

But If I could get something running with ZFS, and then use the drobo as a backup of my ZFS "drive", I have the best of both worlds...all the features and speed of ZFS, and the redundancy and ease of the drobo.

Under Mac OS X, I have ZFS running on 10.5 with the last version of ZFS before it was axed by Apple. You can still download the binaries. I don't think it's compatible with Snow Leopard. I might even channel-bond the iSCSI stuff for a 4 Gb connection.

As was pointed out, the best route is Solaris. At work, I have a 4U setup with SATA expansion cards. We use 4 drives per RAIDZ virtual device, 5 would be better with RAIDZ-2. This gives you the ability to upgrade storage in relatively inexpensive 4-5 disk chunks. Solaris is more picky with hardware than most, but if you double check compatibility of your cards you'll be fine.

For your setup, I'd get ether a cheap PC with room for lots of drives, or a small PC and an external multi-channel SATA enclosure with 4 or 8 drive slots.

If you wanted to keep your Mac Mini as the HTPC, you'd then export the ZFS storage to the mini via iSCSI. There is a really great and free iSCSI initiator for mac called GlobalSAN. You should get >80 MB/s with a good switch. To back this all up, you could then use ZFS's snapshot feature, and rsync everything from the snapshot to the Drobo on some schedule. Both the Drobo, and the ZFS iSCSI target would be connected to the Mac Mini. Boom, a cheap and solid pseudo mini-SAN!

What I outline is what I'll probably being doing at home soon. I want to migrate all of my ZFS stuff off of Mac OS 10.5 and onto Solaris. I might even use channel-bonding to create a 4 Gb link between my Mac Pro and the Solaris box so that I can use it for my home directory, or at least for photos.

[11011001]

Originally Posted by besson3c 

I haven't yet purchased my hardware for it, but about the only piece of hardware I'm looking at that would require a non-standard driver is the 8 port SATA controller PCI card, and it is listed in the Solaris hardware compatibility list as compatible. Everything else (RAM, SATA drives, CPU, power supply, etc.) I expect to work. The extra 8 port SATA controller is also optional if you are happy with 6 drives.

I should take back what I said about 6 drives on a 6 port motherboard giving you 5 TB of data though. You'll need at least one drive not in the RAID-Z pool to boot the machine with.

You can mix and match sizes, I'm pretty sure, but the pool's performance will be limited by the slowest drive.

Solaris can boot from ZFS! Or maybe it can't boot from RAIDZ? I've only done it with a mirrored pool.

[Cold Warrior]

Originally Posted by 11011001 

Solaris can boot from ZFS!

yes, the latest version of opensolaris uses zfs as the default, for boot too.

[besson3c]

Originally Posted by 11011001 

Under Mac OS X, I have ZFS running on 10.5 with the last version of ZFS before it was axed by Apple. You can still download the binaries. I don't think it's compatible with Snow Leopard. I might even channel-bond the iSCSI stuff for a 4 Gb connection.

As was pointed out, the best route is Solaris. At work, I have a 4U setup with SATA expansion cards. We use 4 drives per RAIDZ virtual device, 5 would be better with RAIDZ-2. This gives you the ability to upgrade storage in relatively inexpensive 4-5 disk chunks. Solaris is more picky with hardware than most, but if you double check compatibility of your cards you'll be fine.

For your setup, I'd get ether a cheap PC with room for lots of drives, or a small PC and an external multi-channel SATA enclosure with 4 or 8 drive slots.

If you wanted to keep your Mac Mini as the HTPC, you'd then export the ZFS storage to the mini via iSCSI. There is a really great and free iSCSI initiator for mac called GlobalSAN. You should get >80 MB/s with a good switch. To back this all up, you could then use ZFS's snapshot feature, and rsync everything from the snapshot to the Drobo on some schedule. Both the Drobo, and the ZFS iSCSI target would be connected to the Mac Mini. Boom, a cheap and solid pseudo mini-SAN!

What I outline is what I'll probably being doing at home soon. I want to migrate all of my ZFS stuff off of Mac OS 10.5 and onto Solaris. I might even use channel-bonding to create a 4 Gb link between my Mac Pro and the Solaris box so that I can use it for my home directory, or at least for photos.

What advantages do you see in using iSCSI over NFS?

[besson3c]

Originally Posted by 11011001 

Solaris can boot from ZFS! Or maybe it can't boot from RAIDZ? I've only done it with a mirrored pool.

It can't boot from RAIDZ.

[besson3c]

If any of you dislike OpenSolaris, FreeBSD is another option for using ZFS, particularly version 8 which is in RC3 right now.

[11011001]

Originally Posted by besson3c 

What advantages do you see in using iSCSI over NFS?

I tried Solaris running ZFS sharing via NFS to Mac OS 10.5. Even with gigabit the performance was very depressing. 10.6 was supposed to improve some of these problems, however I haven't done any testing with it.

In my tests, I have found iSCSI to have better performance. Furthermore, being able to use HFS+ natively with a ZFS volume is pretty neat, and has all sorts of benefits (for instance ACLs and permissions). Yes, NFS does support ACLs too. I haven't tested it, but I imagine Spotlight, and even Time Machine would work with the iSCSI/ZFS backed HFS volume (academically of course, since who needs Time Machine when we can use ZFS and snapshotting, and external backups?).

[nonhuman]

Looks like FreeBSD 8.0 final is (unofficially) available: FreeBSD 8.0 available (unofficially) | FreeBSD - the unknown Giant for anyone who wants to give ZFS a try but might not have been too keen on a pre-release OS.

[besson3c]

Originally Posted by nonhuman 

Looks like FreeBSD 8.0 final is (unofficially) available: FreeBSD 8.0 available (unofficially) | FreeBSD - the unknown Giant for anyone who wants to give ZFS a try but might not have been too keen on a pre-release OS.

I've been trying to find info on whether FBSD 8 root pool is ZFS now, or whether it still defaults to UFS. Do you happen to know?

[nonhuman]

Originally Posted by besson3c 

I've been trying to find info on whether FBSD 8 root pool is ZFS now, or whether it still defaults to UFS. Do you happen to know?

I think it is, but I'm not sure. :/

[besson3c]

Just checked, it isn't. The FreeBSD 8 installer still creates UFS boot partitions. You can install ZFS on root: Setting up a zfs-only system , ZFSOnRoot - FreeBSD Wiki , but not via the main system installer.

This is not a big deal to me, I was mostly just curious as to how committed the FBSD team was to phasing out UFS. It seems like they are taking the (wise), conservative approach. Perhaps it's a matter of supporting older, slower computers with less RAM too.

What you *can* do, and what is the recommended solution if you run FBSD as a VM, is to put all of the user, non-OS data that changes semi-frequently into its own partition. So, for instance, /home, /usr/local, /var/log, /var/db, /usr/local/www, /usr/ports, etc. This partition can be a ZFS partition. This way, your UFS partitions are not being written to on a daily basis. The reason why this is also a great solution for a VM is so that you can present the VM two drives: one for your user data, and the other for your static OS files. Since the latter would not change you could keep a backup of this image and have something you can restore from easily, and it would require the minimum amount of disk space for transferring elsewhere.

[nonhuman]

Originally Posted by besson3c 

It seems like they are taking the (wise), conservative approach.

The FreeBSD developers are taking a conservative approach?!? Perish the thought!

It's unfortunate that they don't even provide the option to use a ZFS partition in the installer, but it's really not that hard to do. I've done it a couple times with 7.0 on a couple different architectures.