[besson3c]

Hey Guys,


I'm looking for some perspectives on the storage options available today, and on the horizon...

Specifically, I'm thinking about something that would be a little handier and more future proof for my servers I run. Specifically, I run VMWare Server on Linux and several virtual machines all on the same hardware. However, the advice I'm looking for here is platform agnostic.

There are several different variables and technologies at this point:

- direct attach w. software or hardware RAID

- file systems such as ZFS and BTRFS and the cool stuff you can do with them

- NAS

- external RAID arrays

- iSCSI

- getting higher quality disks such as SCSI and solid state drives



In a perfect world, here is what I'd like:


- A storage device not tethered to my servers so that I can add more storage as needed without planning for downtime and running dd, and upgrade server hardware without having to reinvest in storage.

- Fast I/O suitable for running VMs. There are tricks that can be employed which will help one get by with SATA, but you still have to be careful not to do heavy disk operations on the host as to render the guests unresponsive. This is a nuisance, and of course makes hypothetically backing up complete VM disk images difficult.

- Disk redundancy, i.e. some sort of RAID configuration.

- Some sort of solution somewhere between Cletus the Slackjaw Yokel's Windows XP box and high level enterprise stuff. I like the idea of spreading out the I/O demand across several cheap disks, adding more disk as needed, replacing failed disks as necessary. I don't need crazy fast I/O, I just need something a little more than a single SATA disk. So, enterprise level technology is not what I'm after here, just convenience and decent performance.

- Something that will be relatively future proof and not cost me a fortune

- I'm using nearly 250 gig right now so my capacity needs are not tremendous, but it would cost me a lot more than what I'm paying now to run within a VPS provider such as Linode or Slicehost, and I'd rather not get into the position where my costs grow significantly just to add a modest amount of disk space. My storage needs don't warrant an expensive SAN such as the ones that are no doubt in use by these providers, one of my VMs is running Windows, and I like the control I have now and I like working with my server provider. So, I'm not really keen on moving my eight servers to VMs provided by one of these companies

- A way to backup all of this data (snapshots), preferably via the same overall design so that I have something I can test with and perhaps even fail over to in the event of an emergency. I know that offsite redundancy is the golden egg for many companies, but hey, this is a perfect world type wishlist! I use Amazon S3 for an offsite backup in addition to my current backup to backup my most important data, but to keep my costs to a minimum I'd prefer to just stay with rsync to multiple cheap disks.


I'm interested in learning more about iSCSI, and am fascinated with BTRFS and ZFS. Do any of you have any experience with any of this, have any general recommendations, thoughts, predictions, anything? I don't need to buy anything tomorrow, I'm just thinking that it would be nice to think ahead a little.

It seems like I'm kind of stuck in between not needing to invest heavily in storage like a big company would, yet I'm pushing the limits of run-of-the-mill consumer grade direct attach SATA type stuff - the kind of solution that would be great for using with Time Machine to store pictures of your kids.

Like I said, any thoughts are appreciated...

[OreoCookie]

I would go with a server (with hardware that you deem necessary, be it normal, run-of-the-mill pc components or more expensive server hardware) that runs OpenSolaris (or whatever OS you prefer) and uses ZFS. ZFS gives you the ability to make snapshots on the fly on the level of blocks, i. e. ZFS will only store the blocks that have changed as opposed to whole files (e. g. VMs). It also allows you to create RAID5 or RAID6 equivalents (1 or 2 drives for redundancy) and add more storage to drive pools without down time.

OpenSolaris integrates an NFS and a SMB client right into the ZFS tools. Whether you want to use your server this way, I don't know. There is an iSCSI target daemon for Solaris (and I'm sure for many other *nix OSes).

Regarding hardware, this is more difficult and mostly a matter of how much redundancy you want to/can pay for. You could get some not too old workstation and put in a lot of drives. I think you should use at least 4-5 drives for a RAIDZ/RAID5. Of course, if you want room to expand, you should have that.

I'm not sure you actually need SAS or so, how about looking into getting Raptors instead? They are workstation-class SATA drives. If you add SAS ports to your hardware, you could even mix and match, seeing how SAS is backwards compatible to SATA. ZFS needs some cpu power, but nothing modern processors can't handle.

[besson3c]

Thanks for this OreoCookie!

I will definitely keep my eye on ZFS, but for right now I'm not sure I trust it yet, it hasn't been considered production ready for very long on FreeBSD ([base] Revision 197218), it isn't production ready on Linux, and I'm unsure about whether I want to be first in line to use it by jumping to OpenSolaris just yet. I'm also wondering what its future holds due to its license and the development of BTRFS. I want to plan this just right so I utilize this storage system for many, many years to come. I need highly stable even more than I need this extra convenience, and I have also heard mixed reports on whether ZFS is ready for production use. I'll probably feel better about it when it is in greater use.

I'm not sure I need SAS either or whether multiple SATA drives would do the trick, but I'll look into this. It depends on price vs. performance, as always

[OreoCookie]

Some anecdotal evidence:
Regarding ZFS, our very conservative IT department (which is very good, because they keep us safe, just imagine what a few bad boys can do with two (!) 10 GBit lines to the internet* (and a few backup lines (at the very least a 1 GBit line), not sure if they have kept the dual 155 MBit line ) … They switched our whole storage architecture to ZFS a while ago. Except for the migration itself (copying TB of data just takes time) and the necessity to install firmware updates, the move went without a hickup. If they say ZFS production ready, that's good enough for me. Just to give you an idea: we're a big university with ~25k students with famous top-notch departments. The math and cs department rely heavily on Sun hardware (we have tons of Sun rays and pretty much all servers come from sun).

BTRFS is not very mature: I have a few friends (big Linux aficionados) who are trying BTRFS on their boxes and they've told of quite a few problems. It doesn't sound production ready.


Installing OpenSolaris is not more difficult than installing a Linux distro (if your hardware is supported). Linux and ZFS don't play along very well: the Linux people refuse to include it into the kernel (for religious reasons, they don't like the license under which ZFS has been released and criticize the `rampant layer violations' which make ZFS what it is; BTRFS, however, has to make similar layer violations).


* We have to share the bandwidth with the other big universities of Munich as well as the various research institutes. I'm connected via 100 MBit ethernet at work and I'm easily able to saturate it, though, so I don't mind sharing.

[besson3c]

Thanks again!

I do realize that BTRFS has a ways to go before it is considered mature, and that ZFS is the better of the two today.

The other problem though is that VMWare would have to be able to write its disk images to one of these ZFS file systems, and VMWare Server only works on Linux and Windows hosts. I don't think Linux supports mounting and writing to ZFS volumes yet.

[mduell]

I don't have anything resembling a comprehensive solution for you, but I do have some ideas:

DAS: SiI SteelVine based enclosures
Hardware RAID0/1/10+hotspare on (e)SATA. Requires Windows (or OS X) to manage, but then you can plug it into any dumb SATA port and it acts like 1 big drive. Load one up with 5 Raptors and you've got 600GB RAID10 with a hotspare for $1400.

iSCSI: Drobo Pro
Not the fastest or most stable turtle in the pond, but it's the affordable path to iSCSI and the expansion is good (lots of bays, no downtime).

[OreoCookie]

Originally Posted by besson3c 

The other problem though is that VMWare would have to be able to write its disk images to one of these ZFS file systems, and VMWare Server only works on Linux and Windows hosts. I don't think Linux supports mounting and writing to ZFS volumes yet.

I'm not sure about Windows, but Linux can write to it. But why do you have to access the filesystem directly? Isn't it enough to be able to access the files in one way or another? Isn't mounting via smb or nfs sufficient for your intents and purposes? To my knowledge, though, you can read and write to ZFS on Linux via Fuse (since the Linux kernel crew does not want to include ZFS in the kernel, you have to run it in user land). I'm not sure whether ZFS via FUSE is considered production ready yet, though. I'd go for a solution that's platform agnostic.

mduell's suggestion to have a look at the Drobo Pro is worth a thought: if I remember correctly, the Drobo Pro maxes out at 80~90 MB/s -- which is fast enough for most things, but there are faster solutions. And the best thing (and unique feature, I'm not aware of any other such solution) is that you can simply pop in new drives and your storage capacity increases while still being a fault tolerant solution. It's supposed to be an `it just works' solution, though.

[besson3c]

One problem with Drobo is that ext3 support is only in beta and there are no other supported file systems that I could use, although it looks very compelling in other ways. I'm not sure if you use Drobo with a different file system whether the feature set is simply limited somewhat, but clearly there is some software involved here making the product non-file system agnostic.

I guess mounting a volume via Samba/NFS/SSHfs hosted on a Solaris machine would work, yes, that's definitely an option worth exploring, although iSCSI might be even better. One way I could experiment with introducing ZFS into the picture is running Solaris as a VM and experimenting with setting up shares. This would at least allow me to play around with the functionality of ZFS. I've also been entertaining FreeNAS, I could play around with it in the same way.

Thanks for the ideas, and excuse the thinking outloud here!

[besson3c]

mduell: thanks for the DAS recommendation!

As I said, I'm really working my SATA drives configured with software RAID 1. Any advice on deciding whether to go with a hardware RAID controller, whether I can absorb the performance hit of RAID 5 with a hardware controller, or whether I should look at upgrading to SAS drives? I've yet to do any real research on hardware vs. software RAID in terms of performance.

[mduell]

Why bother with ext3? DroboPro natively supports VMware VMFS.

Hardware RAID spanks software RAID all day long. I'd avoid RAID5; one drive lost and you're either offline or performance in the crapper. Plus you don't really need the capacity. RAID10 is the way to go in the datacenter. 3ware makes good controllers if you're OK with an internal solution.

If you're living on one 7k SATA disk (I doubt your software RAID1 is interleaving the reads), then going to a stripe of 10k SATA disks should be a nice improvement (especially if they improve disk utilization significantly). 15k SAS is great when you need the IO performance (3000 IOPS in a 2U is a beautiful thing), but really runs up the array and drive price.

SteelVine is going to max out at about 200MBps and 10ms seeks (for empty arrays, whack both by a factor of 3 for nearly full arrays) ; DroboPro is going to max out at about 80MBps and 20ms seeks (same rule of thumb).

What's your disk access profile like?

[besson3c]

VMWare Server does not use VMFS, this is for ESX.

[mduell]

Why not upgrade to ESXi?

[besson3c]

I've thought about that, but ESXi is too bare bones without the other stuff they'll rope you into paying for. From their site:

VMware recommends using VMware vSphere Client to manage VMware ESXi. You may also query the VMware ESXi host with the vSphere Command-Line Interface 4.0 (vCLI). Note that the free version of VMware ESXi does not allow any configuration changes for the VMware ESXi host through the vCLI. Further, managing a VMware ESXi host with VMware vCenter Server requires a VMware vCenter Server Agent license for each host, which is included in all editions of VMware vSphere.

I might reevaluate this at the end of the server's life cycle.