[fishguy]

Hey,

I have a macpro and wish to take advantage of disk striping to increase performance. My plan is to get an external drive for the system/boot/apps and then stripe all 4 internal drives in the macpro for my "data drive". Then I need an external back-up and scratch disk. I need to replace a defective sonnet fusion anyway. I was going to get another 4-drive enclosure (I have a sonnet 4 port SATA card) and stripe all 4 together. However, I was going to partition the 4 external drives into one small partition as a scratch disk and a second larger partition as my "back-up disk".

Any comments on this approach? Alternate suggestions? I use photoshop, aperture and FCE.

I would assume that time machine would be able to back-up my "data drive" to my "back-up drive", even if they are both RAID 0?

Would anyone recommend SOFTRAID or would Mac drive utility be sufficient?

Also, what would you recommend as far as an external drive? Should I go for an FW800 drive? I was thinking of something that has built in RAID 1.

Thanks much!

[BLAZE_MkIV]

If your going to spend that much on performance your better off getting an SSD drive. Then use the normal HD for large/infrequently accessed files.

[OreoCookie]

To have a striped software RAID over 4 drives as backup is a recipe for disaster. I would definitely advise against it. To give you alternatives, how about you give us a rundown on the different types of data you have, what workflow you use, etc.

If, for instance, you work on movie projects, then you could archive old projects on different harddrives as the data is no longer actively worked on.

[fishguy]

In the last day I've thoughty and read about it and I now understand the issue of raid 0 as a backup. You are correct in that I have older movie material that can be archived. (images and movie material mostly, images need to be accessable, but 2-3 years of video does not).So I think I'll go for the largest external drive I can get and get a voyager Q and archive media to hard drives and store them. Any tips on how to safely store hard drives? I'm assxuming humidity is the main issue.

As far as ssd drive, they are simply too expensive and not enough capacity. I'd love one in a powerbook though. Now it looks like I need to buy a big external drive, 2 - 1TB seagates and a voyager Q and I'm good to go.

Ok, so If I stripe 3-4 drives in the macpro for speed and use redundant large external drives as a backup - time machine, then I'm assuming I don't need softraid, and can stick with disk utility?

[besson3c]

With 4 drives you would be much better off with a PCI RAID card + RAID 5, definitely not RAID 0. RAID 10 (2+ drives being mirrored) is another option, but you won't get as much total disk space that way.

For your boot drive, if you have a spare PCI slot you wouldn't mind giving up I'd suggest a PCI slot rafter. This rafter will allow you to attach a 2.5" drive to the rafter and connect this directly to your motherboard VIA a SATA cable. This would be much faster than USB, if performance is an issue. I'm pretty sure that OS X will be constantly accessing this drive, so this would be a better option than taking a performance hit in going USB or Firewire, and because you don't really need a big 2.5" drive just for housing the OS the cost of the 2.5" drive would probably be about the same as your USB drive + enclosure/power anyway. The rafter costs about $10.

As far as backup goes, perhaps a NAS would be good for you?

There are actually a *ton* of options here with all of this, so it's hard to really push any one idea at you. I would suggest acquainting yourself with all available options first.

If it were me, being the geeky type I'd be seriously thinking about building a FreeBSD or Solaris based storage appliance for creating myself a ZFS based disk array I'd access over a gigabit network.

[OreoCookie]

I second besson's comment: it's impossible to make any recommendation without knowing what kind of data you have. In all likelihood, you need a mixed solution anyway that takes the importance of the data into account.

Time Machine would almost certainly be one of several components, but not the whole story.

[mduell]

Buy the Apple RAID card if you're serious about RAID with performance. SSD(s) is probably a better option since you're after IO rather than throughput.

[besson3c]

What is so special about the Apple RAID card, mduell?

[mduell]

It works in the Mac Pro case without lame hacks.

It's probably just a rebadged LSI or similar chip.

[SierraDragon]

Duplicate post.

[SierraDragon]

Originally Posted by fishguy 

My plan is to get an external drive for the system/boot/apps and then stripe all 4 internal drives in the macpro for my "data drive". Then I need an external back-up and scratch disk.

Bad idea. You want system/boot/apps on a single internal drive, SSD preferable by a lot. Then RAID0 array 2, 3 or 4 internal drives (you can add a drive in the extra DVD slot) for scratch and the Aperture Library (I use 2 which works well). Note that anytime you go external with FW or USB (except eSATA) you are slowing throughput, so keep critical access for the Aperture Library and PS scratch on internal drives.

RAID0 is the most cost effective way to achieve speed and capacity without having more drives arrayed than are generally available in a MP box. Some folks focus on the increased failure risk of RAID0, but we need daily backup anyway so the increased risk of array failure becomes moot.

Note that RAID1 (half the capacity) is twice as expensive as RAID0 and with images work we do not need or even want real-time redundancy; real-time redundancy simply copies your mistakes. Better is to maximize capacity/value and back up to multiple locations at discrete intentional intervals.

I strongly recommend using Referenced Masters for the Aperture Library, not the default Managed Masters. Separately back up original images prior to importing into Aperture. Use of Referenced Masters allows the Aperture Library to always remain small enough to easily back up to single cheap 1 TB drives at multiple locations. Managed Masters make the Library size grow too much and leads to all kinds of other issues.

-Allen Wicks

[fishguy]

I'm sold on an ssd for boot/apps and will stripe remaining 3 drives as raid 0. I was way over thinking the back-up strategy and will keep it simple with multiple drives instead of raid 1. If i skip time machine as a backup software, what do you guys recommend, carboncopycloner or what?

SierraDragon, So you put your aperture library on the scratch disk instead of main data disk? Is a single physical drive sufficient for scratch or do you raid0 multiple drives? Is it a pain to go from my current managed masters library to referenced masters? My intention is to get a mbp soon this year for Aperture on the road, but i need a streategy that will allow me to "sync" what I do on the road with my main aperture library once back home.

Thanks all!

[besson3c]

Originally Posted by mduell 

It works in the Mac Pro case without lame hacks.

It's probably just a rebadged LSI or similar chip.

Apple prevents other RAID cards from working in the XServe as well? If so, man, could Apple/the XServe be any lamer? Damn...

[OreoCookie]

Originally Posted by fishguy 

I'm sold on an ssd for boot/apps and will stripe remaining 3 drives as raid 0. I was way over thinking the back-up strategy and will keep it simple with multiple drives instead of raid 1. If i skip time machine as a backup software, what do you guys recommend, carboncopycloner or what?

You shouldn't skip Time Machine. It should be very suitable to back up your data and apps.

Regarding cloning, clones are a very bad backup, you should avoid them. Period. What kind of backup you use, really depends on your data. In general, I would have a look at software like Synk. Unless you give us more details, though, we cannot make very specific recommendations!

Originally Posted by fishguy 

SierraDragon, So you put your aperture library on the scratch disk instead of main data disk? Is a single physical drive sufficient for scratch or do you raid0 multiple drives? Is it a pain to go from my current managed masters library to referenced masters? My intention is to get a mbp soon this year for Aperture on the road, but i need a streategy that will allow me to "sync" what I do on the road with my main aperture library once back home.

I don't think you need to manage files manually. If you keep your library as it is, all you need to do is copy/move your Aperture library bundle from one location to another. This moves all your files. If you want to open the library in the new location, just double click. In fact, if you are happy that Aperture manages your files (personally, I am), then there is no reason to manually manage your files, unless you run out of space.

There is no real way to sync libraries with a piece of software. In fact, this will probably damage your Aperture library! I would work with projects: export them as projects, move them to your desktop and start working on them on your main machine.

[Simon]

Originally Posted by fishguy 

If i skip time machine as a backup software, what do you guys recommend, carboncopycloner or what?

1.) Clones are not a backup strategy.
2.) Do not skip TM. It's a very simple yet effective backup solution with some simple version control. It's truly plug and forget.
3.) You can make a clone as an additional level of security beside a proper backup. But do not use third-party/commercial tools like CCC or SuperDuper. They have caused a lot of people a whole lot of trouble. Instead, use the free and reliable cloning tool built right into OS X.

/Applications/Utilities/Disk Utility > Restore

If you select 'erase destination' a clone of a bootable partition will remain bootable.

Disk Utility can also be accessed from the Utilities menu if you boot from the OS X install DVD. IOW Disk Utility is always available whenever you're running OS X.

[besson3c]

fishguy: you also have to ask yourself whether you a manual clone or an automatic clone. What Simon wrote for a manual clone method will work, but you can also automate this with the Disk Utility CLI utility and setting yourself up with a cronjob (or launchd job, if you feel like challenging yourself).

If this were me, I would still not go with a RAID 0 solution - even with a good backup scheme. In the event of a failure, not only may it require time to trace and realize you have one, but it also amount in downtime. I would suggest a more "high availability" scheme like RAID 5 or 6 that is a hybrid of what you want, I would think: high availability, redundancy, *and* making good use of disk space.

With a scheme like this, there is less need for a complete backup... The array would be *designed* for some failure and would deal with this gracefully, esp. with a RAID-6 scheme where you could have two simultaneous disk failures.

[mduell]

Originally Posted by besson3c 

Apple prevents other RAID cards from working in the XServe as well? If so, man, could Apple/the XServe be any lamer? Damn...

They work, but you have to run cables to the drives and find some way to connect them. They don't integrate like the Apple card does. Pretty standard even in non-Apple machines.

[besson3c]

Originally Posted by mduell 

They work, but you have to run cables to the drives and find some way to connect them. They don't integrate like the Apple card does. Pretty standard even in non-Apple machines.

I see. I guess the exception would be with motherboards with an included RAID controller?

[SierraDragon]

Originally Posted by fishguy 

...If i skip time machine as a backup software, what do you guys recommend, carboncopycloner or what?

Clone software is ok for the boot/apps drive but not for data. And in the images world for data personally I prefer scheduled manual backup rather than automated backup like Time Machine, because automated backups can back up our mistakes.

SierraDragon, So you put your aperture library on the scratch disk instead of main data disk? Is a single physical drive sufficient for scratch or do you raid0 multiple drives?

Personally I make a RAID0 array primarily for the Aperture Library and for PS scratch but I also put misc data on it as space permits, never filling the array more than 50% for speed. As for "a more "high availability" scheme like RAID 5 or 6 that is...: high availability, redundancy, *and* making good use of disk space" my feeling is that RAID5 or RAID6 sound good but (a) take a minimum of four drives and (b) do not particularly "make good use of disk space"; 50% better than RAID1 but still 75% the capacity of RAID0 (if one devotes 4 drives of a MP to RAID5, which I consider functionally unrealistic for most of us anyway). The same routine backup is a necessity whatever schemata is selected, and I consider the effort to rebuild from backup not that important because it only happens once every few years (most times drive failures are telegraphed by noise preceding failure and allowing replacement).

For me the primary need for the RAID array is to improve access speed to scratch and to the Aperture Library. I have not partitioned my 2 TB RAID0 array because every partition I have done in the past few decades has ultimately turned out to be inappropriate. However it might make sense to partition 200 GB of the array for scratch just to be able to easily erase scratch. Note that keeping drives underfilled is essential to maximize speed.

My setup is Drive 1 OS/apps. Drives 2/3 RAID0 for the Aperture Library and for PS scratch and some data. Drive 4 for on site backup of the Drives 2/3 RAID0 array (note that whereas for speed reasons a 2 TB RAID0 array should not contain more than 1 TB data, a 1 TB backup drive can be filled to 85% for backup purposes because speed is largely not relevant). When I can afford it I will add an SSD in the extra DVD drive slot for OS/apps and I will replace the backup Drive 4 with a 2 TB drive.

Is it a pain to go from my current managed masters library to referenced masters?

No, it is easy and strongly recommended. The fatal flaw with Managed Masters is that sooner or later the Aperture Library gets too large to function properly. Until that point Managed Masters work just fine, but IMO it is dumb to build a workflow that does not scale.

My intention is to get a MBP soon this year for Aperture on the road, but i need a strategy that will allow me to "sync" what I do on the road with my main aperture library once back home.

Many of us need that but despite years of requests Apple has so far failed to provide any synch mechanism. Hopefully by the time you get a MBP v3 Aperture will be out with a supported synch procedure. In the meantime there are many threads here discussing various not-fully-adequate ways to work around Apple's failure to provide synch.

-Allen Wicks

[besson3c]

Originally Posted by SierraDragon 

Personally I make a RAID0 array primarily for the Aperture Library and for PS scratch but I also put misc data on it as space permits, never filling the array more than 50% for speed. As for "a more "high availability" scheme like RAID 5 or 6 that is...: high availability, redundancy, *and* making good use of disk space" my feeling is that RAID5 or RAID6 sound good but (a) take a minimum of four drives and (b) do not particularly "make good use of disk space"; 50% better than RAID1 but still 75% the capacity of RAID0 (if one devotes 4 drives of a MP to RAID5, which I consider functionally unrealistic for most of us anyway). The same routine backup is a necessity whatever schemata is selected, and I consider the effort to rebuild from backup not that important because it only happens once every few years (most times drive failures are telegraphed by noise preceding failure and allowing replacement).

Sorry, I just don't like this idea.

First of all, a RAID 5 needs 3 drives technically, but yes, practically speaking you'll need 4 drives if you want the redundancy. It is true that you won't get the same amount of disk space, but who is saying that you need to confine yourself to 4 drives? There are SATA backplanes that will hold more than 4 drives (although I don't know if they'll fit into a Mac Pro), but more important, there are a whole host of ways to access more than 4 drives:

- eSATA
- PCI risers
- externally hosted NAS

I encourage the reader to think outside of the box a little bit, pardon the pun. A RAID-0 solution is a dangerous affair. If a drive fails, the whole array fails, leaving you to rebuild it. Drives fail pretty frequently, so why would anybody trouble themselves with this burden hoping that their backup is sound?

If you need serious storage, direct attach (DAS) is a limited option anyway, as since the drives are tethered to your Mac whenever you want to upgrade to a new Mac you are going to have to find some way to haul those same drives over to the new Mac, or else get your data off of all of this.

If it were me, I'd go for a separate storage appliance with a whole swath of drive bays that will give you room to grow, will allow you to easily replace drives with larger ones, and that are hot swappable so that you don't have to power down and crack open the case to replace a drive. This will eliminate having your storage tethered to your computer, and will also allow you to share this same storage with other computers if you are cool with using some sort of protocol that negotiates shared access (e.g. AFP, NFS, Samba, iSCSI, etc.)

For me the primary need for the RAID array is to improve access speed to scratch and to the Aperture Library. I have not partitioned my 2 TB RAID0 array because every partition I have done in the past few decades has ultimately turned out to be inappropriate. However it might make sense to partition 200 GB of the array for scratch just to be able to easily erase scratch. Note that keeping drives underfilled is essential to maximize speed.

A RAID array does not improve speed, it provides more disk bandwidth so that you don't saturate what you have. If you have enough I/O with your current setup so that your read and writes are not getting backlogged, a striped disk array is not going to gain you more speed, as each disk will still operate at 7200 RPM or whatever the speed of the slowest disk is. Since it is pretty easy to saturate a single SATA disk, yes, the net result will probably be faster speed, but it is still wise to think about this carefully as there is no relationship between adding more disks resulting in faster speeds. This will only work up to the point where you aren't saturating your available I/O.

[OreoCookie]

Originally Posted by SierraDragon 

And in the images world for data personally I prefer scheduled manual backup rather than automated backup like Time Machine, because automated backups can back up our mistakes.

You know you can do that with Time Machine as well, right? All you need to do is switch it off and then every time you want to make a backup, you initiate a backup from the menu item.

Originally Posted by SierraDragon 

No, it is easy and strongly recommended. The fatal flaw with Managed Masters is that sooner or later the Aperture Library gets too large to function properly. Until that point Managed Masters work just fine, but IMO it is dumb to build a workflow that does not scale.

What do you mean by `too large?' I haven't noticed any problems up until now.

Originally Posted by SierraDragon 

Many of us need that but despite years of requests Apple has so far failed to provide any synch mechanism. Hopefully by the time you get a MBP v3 Aperture will be out with a supported synch procedure. In the meantime there are many threads here discussing various not-fully-adequate ways to work around Apple's failure to provide synch.

I know of no satisfactory solution to this problem by any manufacturer. Certainly nothing easier than exporting projects manually.

[mduell]

I'm surprised the pros still don't have a good sync mechanism. For amateurs like me, Picasa + Dropbox provides awesome syncing (and versioning and cross platform and galleries and blah blah).

Originally Posted by SierraDragon 

Personally I make a RAID0 array primarily for the Aperture Library and for PS scratch but I also put misc data on it as space permits, never filling the array more than 50% for speed. As for [i]"a more "high availability" scheme like RAID 5 or 6 that is...: high availability,

Many RAID5/6 implementations at the consumer or even pro level do not offer high availability, only RAID1 consistently offers HA.

Originally Posted by besson3c 

If you need serious storage, direct attach (DAS) is a limited option anyway, as since the drives are tethered to your Mac whenever you want to upgrade to a new Mac you are going to have to find some way to haul those same drives over to the new Mac, or else get your data off of all of this.

If it were me, I'd go for a separate storage appliance with a whole swath of drive bays that will give you room to grow, will allow you to easily replace drives with larger ones, and that are hot swappable so that you don't have to power down and crack open the case to replace a drive. This will eliminate having your storage tethered to your computer, and will also allow you to share this same storage with other computers if you are cool with using some sort of protocol that negotiates shared access (e.g. AFP, NFS, Samba, iSCSI, etc.)

DAS is the only way to get decent speed (including latency/IOPS, not just bandwidth) without getting amazingly expensive.

Originally Posted by besson3c 

A RAID array does not improve speed, it provides more disk bandwidth so that you don't saturate what you have. If you have enough I/O with your current setup so that your read and writes are not getting backlogged, a striped disk array is not going to gain you more speed, as each disk will still operate at 7200 RPM or whatever the speed of the slowest disk is. Since it is pretty easy to saturate a single SATA disk, yes, the net result will probably be faster speed, but it is still wise to think about this carefully as there is no relationship between adding more disks resulting in faster speeds. This will only work up to the point where you aren't saturating your available I/O.

Latency doesn't change but IOPS also scales nearly linearly with the number of disks in a striped array. IOPS is often more significant than bandwidth/bulk transfer for apparent speed.

[besson3c]

Originally Posted by mduell 

Many RAID5/6 implementations at the consumer or even pro level do not offer high availability, only RAID1 consistently offers HA.

Why not? The variance is in the difficulty of replacing drives, I agree with you, but you still have the HA of at least a single drive being able to fail completely without skipping a beat.

DAS is the only way to get decent speed (including latency/IOPS, not just bandwidth) without getting amazingly expensive.

What about eSATA? I agree that one can saturate a gigabit network, and that ethernet bonding is a little complicated for many people, although it is also not amazingly expensive.

Latency doesn't change but IOPS also scales nearly linearly with the number of disks in a striped array. IOPS is often more significant than bandwidth/bulk transfer for apparent speed.

Right, but my point is that this linear relationship only works up until the point where there is no shortage of IOPS and the bottleneck becomes the speed of the individual drives. It's sort of like how adding multiple processor cores can increase the number of things you can do at a given time, but for a *single* task having that extra core is not going to buy you more speed. Is my logic flawed?

[Simon]

Originally Posted by besson3c 

What about eSATA?

Umm, eSATA is used for a lot of DAS.

[BLAZE_MkIV]

Which is my SANs are lots of ~30GB 10k scsi drives. limit latency and maximize the chance of the file being on multiple drives.

[mduell]

Originally Posted by besson3c 

Why not? The variance is in the difficulty of replacing drives, I agree with you, but you still have the HA of at least a single drive being able to fail completely without skipping a beat.

The cheaper controllers can't rebuild and provide data access at the same time with RAID5/6. With RAID1 they can.

Originally Posted by besson3c 

What about eSATA? I agree that one can saturate a gigabit network, and that ethernet bonding is a little complicated for many people, although it is also not amazingly expensive.

It's not about bandwidth. The the combination of ethernet and the network sharing software stack is where latency/IOPS performance suffers.

Originally Posted by besson3c 

Right, but my point is that this linear relationship only works up until the point where there is no shortage of IOPS and the bottleneck becomes the speed of the individual drives. It's sort of like how adding multiple processor cores can increase the number of things you can do at a given time, but for a *single* task having that extra core is not going to buy you more speed. Is my logic flawed?

But the files are always spread across multiple disks in the stripe. So in the analogy, the process is multithreaded.
If you're constrained by disk, the shortage can be bandwidth/bulk transfer, IOPS, or latency. Striping scales well for the first two but does nothing for latency. Latency is less often the issue.

[OreoCookie]

Originally Posted by mduell 

I'm surprised the pros still don't have a good sync mechanism. For amateurs like me, Picasa + Dropbox provides awesome syncing (and versioning and cross platform and galleries and blah blah).

Perhaps that's because there isn't enough bandwidth and online storage available. I'm nowhere near a pro (and I do use Dropbox), but one evening with my camera weighs in at 800 MB or so (only 25-30 % or so are keepers, though). That's a bit much to sync. Once storage with this capacity becomes available cheaply and I have enough bandwidth to actually sync with it in a reasonable time, I'll definitely use online storage as one of my backup options. (Right now, I use Dropbox to share data with colleagues.)

[mduell]

Originally Posted by OreoCookie 

Perhaps that's because there isn't enough bandwidth and online storage available. I'm nowhere near a pro (and I do use Dropbox), but one evening with my camera weighs in at 800 MB or so (only 25-30 % or so are keepers, though). That's a bit much to sync. Once storage with this capacity becomes available cheaply and I have enough bandwidth to actually sync with it in a reasonable time, I'll definitely use online storage as one of my backup options. (Right now, I use Dropbox to share data with colleagues.)

I was thinking sync between different machines in the same building; Dropbox's local block exchange works wonderfully there while the data trickles up to the cloud. Even with crappy home broadband like mine, 800MB is only 6 hours to sync. If you start editing right away that drops to 2 hours.

But the problem for the pros is the pro apps have their own little walled off blobs that don't work well with Dropbox and concurrent updates.

[SierraDragon]

Originally Posted by OreoCookie 

You know you can do that with Time Machine as well, right? All you need to do is switch it off and then every time you want to make a backup, you initiate a backup from the menu item.

Yes TM can be initiated manually. The only thing that I have against TM is the dozens of issues that folks running Aperture have reported with TM. Aperture and PS are my two most important apps.

...I know of no satisfactory solution to this problem [synching computers] by any manufacturer. Certainly nothing easier than exporting projects manually.

Filemaker databases have been seamlessly network synching for more than 20 years; I did it with Mac SEs and PhoneNet connectors in the 80s. FM is an Apple product and has been databasing images for more than a decade. All the SQL, Informix, etc. DBs have also networked images for decades, as have the multiuser versions of Extensis Portfolio and Canto Cumulus DAM apps.

...What do you mean by `too large?' I haven't noticed any problems up until now.

Correct, in the beginning a Managed Aperture Library works fine. But it does not scale well.

At 20 MB or more per image and 20 GB of images for a common pro shoot like a medium-sized wedding, a Managed Aperture Library can easily grow to 500 GB - 1+ TB size, at which point the drive throughput of a single 1-TB drive starts to slow or even shuts down. That means such a Managed Aperture Library must be on a RAID array of some sort, and that means that backup must also be on a RAID array of some sort, NAS, etc.

OTOH with a Referenced Aperture Library the Library easily remains small, lending simplicity to locating the Library, throughput speed, backup drives, etc.

-Allen Wicks

[OreoCookie]

Originally Posted by mduell 

I was thinking sync between different machines in the same building; Dropbox's local block exchange works wonderfully there while the data trickles up to the cloud. Even with crappy home broadband like mine, 800MB is only 6 hours to sync. If you start editing right away that drops to 2 hours.

That's not good enough to me: unless the uplink is fast enough so that it feels almost like transmitting files over a local network, I pass.

I am an avid user of Dropbox, but only for text files (git repositories to be exact) that are a few to a few tens of megabytes in size. In a few years, it may be ready to serve as online backup solution, but IMO it's not there yet.

Originally Posted by mduell 

But the problem for the pros is the pro apps have their own little walled off blobs that don't work well with Dropbox and concurrent updates.

I haven't tried using Dropbox in conjunction with another piece of software, but I can see quite a few problems that may lead to database corruption. Nothing that's not present when you deal with other shared files that may be used by several users.

[OreoCookie]

Originally Posted by SierraDragon 

Yes TM can be initiated manually. The only thing that I have against TM is the dozens of issues that folks running Aperture have reported with TM. Aperture and PS are my two most important apps.

Time Machine and Aperture have been running flawlessly side-by-side for me. There were initial problems regarding library corruption in 10.5.0, but those were quickly fixed.

Originally Posted by SierraDragon 

Filemaker databases have been seamlessly network synching for more than 20 years; I did it with Mac SEs and PhoneNet connectors in the 80s. FM is an Apple product and has been databasing images for more than a decade. All the SQL, Informix, etc. DBs have also networked images for decades, as have the multiuser versions of Extensis Portfolio and Canto Cumulus DAM apps.

Data base apps are different from the types of apps we're talking about now.

Originally Posted by SierraDragon 

Correct, in the beginning a Managed Aperture Library works fine. But it does not scale well.

At 20 MB or more per image and 20 GB of images for a common pro shoot like a medium-sized wedding, a Managed Aperture Library can easily grow to 500 GB - 1+ TB size, at which point the drive throughput of a single 1-TB drive starts to slow or even shuts down. That means such a Managed Aperture Library must be on a RAID array of some sort, and that means that backup must also be on a RAID array of some sort, NAS, etc.

OTOH with a Referenced Aperture Library the Library easily remains small, lending simplicity to locating the Library, throughput speed, backup drives, etc.

Sorry to say, but your conclusion is false. It's not Aperture that gets more inefficient, it's just that the image files are saved on slower and slower parts of the harddrive the more it fills up. It's got nothing to do with the database design as such.

In your case, I agree that it may be easier and cheaper to manage your files manually and just add drives to accommodate new projects. But I don't agree with your suggestion that everyone should manage files manually: as long as they comfortably fit on one harddrive, I see no reason to.

[SierraDragon]

Originally Posted by OreoCookie 

Data base apps are different from the types of apps we're talking about now.

Not really. It is Aperture's images database that needs networking, not some kind of multiuser concurrent adjustment of images that we are talking about. And even if we were, DAM apps like Cumulus and Portfolio that I mentioned do exactly that.

Sorry to say, but your conclusion is false. It's not Aperture that gets more inefficient, it's just that the image files are saved on slower and slower parts of the harddrive the more it fills up. It's got nothing to do with the database design as such.

Sorry, but you misinterpret my words. I said nothing about Aperture getting more inefficient or about the database design. I agree "it's just that the image files are saved on slower and slower parts of the harddrive the more it fills up. It's got nothing to do with the database design as such."

My conclusion (that you consider false) is that for pro volumes of images, Referenced Masters more easily allows for fast i/o throughput as the number of images in the database grows, not for reasons of Aperture efficiency, but for reasons of workflow efficiency and hard drives usage. A Managed Masters Library is fine for small quantities, but this whole post is about multi-drive setup which presumably involves more than amateur-level quantities of small files. Once one scales up to larger volumes and file sizes a Managed Masters workflow tends to fall apart.

In your case, I agree that it may be easier and cheaper to manage your files manually and just add drives to accommodate new projects. But I don't agree with your suggestion that everyone should manage files manually: as long as they comfortably fit on one harddrive, I see no reason to.

First off, any competent images workflow requires one to manage files manually. It is imperative that originals on CF be manually copied to a hard drive and backed up to another location as well. Once that original essential Master Original is living on an attached hard drive, Importing as a Referenced Masters Aperture Library is as automatic as it is to use a Managed-Masters workflow. Automating via Aperture from the original CF image prior to copying to a hard drive is very poor form.

I do agree that if one will forever only have a small volume of images and a single underfilled drive, Managed Masters work fine. The gist of this thread however seems to be about multiple drives and larger (more typical of pro app workflows) volumes. And, the need to manage files manually prior to Aperture usage applies regardless of volume.

-Allen Wicks

[OreoCookie]

Originally Posted by SierraDragon 

Not really. It is Aperture's images database that needs networking, not some kind of multiuser concurrent adjustment of images that we are talking about. And even if we were, DAM apps like Cumulus and Portfolio that I mentioned do exactly that.

They're plenty different from simple content management solutions with a lot more capabilities. I'm not saying it couldn't be done, it just hasn't been done. One possible ansatz -- which I don't think would work in the context of photography -- is something like Final Cut Server. It's a problem to tailor a software solution to the workflow of photography. If it were only about managing things (and you'd have to manage versions of files by hand!), then the solution is simple, you'd just add versions of files. If your management software also allows for editing, it's much easier to get conflicts.

Originally Posted by SierraDragon 

My conclusion (that you consider false) is that for pro volumes of images, Referenced Masters more easily allows for fast i/o throughput as the number of images in the database grows, not for reasons of Aperture efficiency, but for reasons of workflow efficiency and hard drives usage.

Just to make it clear: I don't consider that conclusion false.
I just said that as long as you don't run into io bottlenecks, there is no speed advantage of managing your files manually.

Originally Posted by SierraDragon 

First off, any competent images workflow requires one to manage files manually.

I disagree, there is nothing less pro or competent about using a pro software package's built-in capabilities. If you need to manage files because of space constraints, that's another thing, but to claim that in order to have a `competent workflow' you need to manage files manually is false.

Originally Posted by SierraDragon 

It is imperative that originals on CF be manually copied to a hard drive and backed up to another location as well. Once that original essential Master Original is living on an attached hard drive, Importing as a Referenced Masters Aperture Library is as automatic as it is to use a Managed-Masters workflow. Automating via Aperture from the original CF image prior to copying to a hard drive is very poor form.

Why?
I always delete the files on my memory cards manually after I have checked that all files have been copied and a backup has been made. Aperture offers file vaults to allow me to make additional backup of the files.

Originally Posted by SierraDragon 

I do agree that if one will forever only have a small volume of images and a single underfilled drive, Managed Masters work fine. The gist of this thread however seems to be about multiple drives and larger (more typical of pro app workflows) volumes. And, the need to manage files manually prior to Aperture usage applies regardless of volume.

What's a `pro app workflow,' I thought Aperture was a pro app
I can think of workflows even with large libraries where you don't necessarily manage all files manually. You could export old projects to other harddrives or move old projects as referenced files while letting Aperture automatically manage current projects. The `pro' tag is arbitrary.

[SierraDragon]

Originally Posted by OreoCookie 

They're plenty different from simple content management solutions with a lot more capabilities. I'm not saying it couldn't be done, it just hasn't been done. One possible ansatz -- which I don't think would work in the context of photography -- is something like Final Cut Server. It's a problem to tailor a software solution to the workflow of photography. If it were only about managing things (and you'd have to manage versions of files by hand!), then the solution is simple, you'd just add versions of files. If your management software also allows for editing, it's much easier to get conflicts.

I disagree, because we are only talking about synching the Library (while the app itself is not being run) like most databases have done forever. Apple has unfortunately elected not to provide that simple solution to us so far. Frankly it sucks that all us single-user two-computer photogs have to create workarounds for what has been database sop for decades.

I just said that as long as you don't run into io bottlenecks, there is no speed advantage of managing your files manually.

Agreed. However IMO it makes sense to plan ahead when establishing a heavy-duty images management workflow. Ergo start out with a Referenced Library from the beginning.

I disagree, there is nothing less pro or competent about using a pro software package's built-in capabilities. If you need to manage files because of space constraints, that's another thing, but to claim that in order to have a `competent workflow' you need to manage files manually is false.

Correct, we disagree. The fact is that allowing any app other than the Finder to have first shot at one's original not-yet-backed-up images is IMO unnecessarily risky behavior. Sure lots of folks do it, and Apple even describes the workflow in the manual, but it is still unnecessarily risky behavior that has bitten some photogs in the butt. [/QUOTE]

Why?
I always delete the files on my memory cards manually after I have checked that all files have been copied and a backup has been made. Aperture offers file vaults to allow me to make additional backup of the files.

Aperture vaults have failed. Deleting memory card files on the computer can also cause issues. Best is to reformat only in-camera after Finder-level backup. The lucky folks with cameras like the Nikon D3 that allow writing to 2 CF cards at the same time can create less rigorous workflows.

What's a `pro app workflow,' I thought Aperture was a pro app

Aperture is a pro app, but the Aperture workflow needed to optimize pro volumes of images is different than the Aperture workflow needed to optimize non-pro volumes of images. Of course a prolific amateur may shoot as many pix as a pro, thus requiring a pro Aperture workflow (multiple hard drives, RAID, etc.)

-Allen Wicks

[OreoCookie]

Originally Posted by SierraDragon 

I disagree, because we are only talking about synching the Library (while the app itself is not being run) like most databases have done forever. Apple has unfortunately elected not to provide that simple solution to us so far. Frankly it sucks that all us single-user two-computer photogs have to create workarounds for what has been database sop for decades.

I agree they could have offered a solution, but I wouldn't like it to be a simple sync tool. Way too restrictive.

Originally Posted by SierraDragon 

Agreed. However IMO it makes sense to plan ahead when establishing a heavy-duty images management workflow. Ergo start out with a Referenced Library from the beginning.

This is unnecessary thanks to the flexibility of Aperture: you can move projects (or even single pictures) later if necessary. It's quite easy and even if you have tons of projects shouldn't take longer than an afternoon.

I've heard many people argue that you shouldn't use a managed library, because you don't have `easy access' to the pictures and that the setup is not cross-app compatible. These people fail to see that the work is in the adjustments, the tagging, the albums, web galleries and books -- and I probably won't be able to migrate that easily anyway, because this is stored in Aperture's database. There is a lot of FUD going on about managed libraries.

Originally Posted by SierraDragon 

Correct, we disagree. The fact is that allowing any app other than the Finder to have first shot at one's original not-yet-backed-up images is IMO unnecessarily risky behavior. Sure lots of folks do it, and Apple even describes the workflow in the manual, but it is still unnecessarily risky behavior that has bitten some photogs in the butt.

Risky??
I've never found it to be risky. The Finder is no more or less prone to data loss than Aperture's import dialog. I verify that the photos have been imported and once everything checks out, I format the memory card in my camera.

The only images I've lost so far were from user error (= myself) and once a memory card gave up on me. I was able to restore most of the images via some recovery software, though.

Originally Posted by SierraDragon 

Aperture vaults have failed.

Any form of backup can fail, that's why you should have redundant backups. I have stored my images in three locations (four, if you could old DVDs, although I don't trust them anymore):
(1) Time Machine
(2) On-site Aperture vault
(3) Off-site Aperture vault
That's in addition to the original.

Originally Posted by SierraDragon 

Aperture is a pro app, but the Aperture workflow needed to optimize pro volumes of images is different than the Aperture workflow needed to optimize non-pro volumes of images. Of course a prolific amateur may shoot as many pix as a pro, thus requiring a pro Aperture workflow (multiple hard drives, RAID, etc.)

My objection is that you seem to think professional photographers need to take a lot of pictures -- more pictures than amateurs (as you write yourself). While this is true in many lines of commercial photography, the volume of a professional (≠ commercial) photographer need to be larger than that of an ambitious amateur.

The correct distinction is not pro vs. non-pro, but merely a question of how many photos you take. And even then, going with a referenced library only is not automatic. I guess my only disagreement with what you write is that `heavy duty images workflow' requires you to use a referenced library.