your digital shoe box and beyond?

Text and images Copyright Mark Graf, 2005, 2006. All rights reserved. Revised 12/2/2006

So you have 'gone digital' and are starting to pile up digital images on your hard drive. Things have changed quite a bit since throwing all of those slides in slide pages, and then into a binder or a file cabinet. Perhaps you even used a shoe box. There is a bit more to worry about with digital storage than insects eating the emulsion on your slides. Are you confident they will be around tomorrow? Backup and storage strategies are a hot topic in most of the forums discussing digital cameras. Interesting enough, I have heard more about hard drive failures in the past few years than over the entire time I have owned a computer.

Just like people have many ways of organizing their slides, there are just as many, if not more methods and preferences to keeping your bits and bytes of sunsets safe and secure. Which is best? It really comes down to personal preference, accessibility, how much you want to spend, how cautious (and/or paranoid) you want to be, and how much medication you want to avoid if you lose something. If you are not backing up your digital originals yet - you are basically playing Russian roulette with your pictures. Maybe the barrel has 12 chambers, maybe it has 1000, maybe it has 2. How much risk are you willing to take? How many memories and once-in-a-lifetime moments could you lose?

I have contemplated how I want to handle this, and will likely will continue to review and revise for years to come. Storage technology is changing all of the time. My collection of digital originals is still relatively small, but quickly growing such that management needs to be planned out. The key is, just like in defining a good slide filing system up front, to get a good digital filing system established you can grow with. How will it adapt to changing technology and media? Some questions to think about:

• If I do have a crash, how easy will it be to copy files from my backup media?
• Will my backup media be accessible in 5/10 years or will I have to recopy to another media?
• What if my house burned down or was burglarized?

Below is my current method of backing up images as of November 2005 (Updated Dec 2006). I provide some of my experience with the different devices and mediums I use and you can then decide if you think this might be a good match for you.

hard drives

All of my master image files are stored on hard drives. Quite simply, because this is the easiest, most flexible technology that allows you to move images around from place to place, and offers the most storage space in a single unit. Whether you use internal drives inside your computer case, or external drives connected via a USB2 or Firewire connection, it is up to you. I personally find external drives more flexible and convenient, although I currently use both.

There is an access speed difference between internal and external drives. For example, see the table below for the difference in speed to copy a file across my internal drives vs. from an internal to an external one. This test was conducted using DiskBench software, on my Dell 8300 PC 3 GHz with 3GB RAM.

While external drives have slower transfer speeds, on a file to file basis, you will not see that much difference when working with them in Photoshop. File Open times are still only a matter of a couple seconds. This of course depends on your PC configuration and how much memory you have.

Simpletech makes some very nice external hard drives. I purchased the one referenced here based upon many other photographers that have had good success with them. Hard drives are discussed all of the time in areas like Naturescapes.net's digital forum or Rob Galbraith's forums.

With hard drives however, backup is critical. Particularly because you can place so much data on them these days, one crash can cost you a lot. There are a few options you can use in backing up your hard drives - the most simple being buying two of every drive you plan to use. So as you add data to one drive, you use the other to back it up. There are a lot of software programs out now that address the issue of backup, because there is such a growing need for it. Most of them allow backups to be scheduled during the night automatically so you aren't bothered with it during the day. Two particular programs that I have found incredibly useful are Microsoft's SyncToy and Acronis TrueImage. SyncToy is currently FREE and is a very user friendly program that helps you in synchronizing data from one folder to another perhaps on another drive or network. It is best suited for backing up segments or folders of a drive. TrueImage makes a duplicate copy (a disk "image") of one drive and stores it in a single compressed file in another location. It also offers disaster recovery options in case an entire disk fails. I have found this programs makes migrating over to a new larger internal hard drive a piece of cake.

Another option in the type of hardware to use is a RAID array. RAID stands for Redundant Array of Independent Disks. Sometimes you will find "Independent" replaced with "Inexpensive" - I don't think either is set in stone, but these things certainly don't fall into my definition of inexpensive. I won't attempt to get into all the technical nitty-gritty of what RAID does other than offer my simple overview of how it works. You can do a Google search on RAID for some late night sleep aids. This is a group of hard disks (2 or more) that work together to automate the backup process for you. There are also many different types of arrays, like RAID1 or RAID5, which are probably the most common. They require a special RAID controller either inside your computer, or as an external unit like a Networked Attached Storage device (NAS). (more on this later) Some PCs sold these days have this as an optional feature, and perhaps may become standard in the future.

RAID1 for example, stands for a mirroring type array, meaning all data you place on one drive is automatically duplicated (ie. mirrored) on the other. If you have a 500 GB RAID1 array of 2 drives, you have 250 GB of usable space, and the other 250 GB is used for backup. You computer really only sees 1 250 GB drive. You need a 2 for 1 drive ratio for this type.

RAID5 uses three or more drives working together, but instead of your primary data being on a single drive like RAID1, it is actually scattered across all of the drives in the array (striping). So pieces of a single image file may exist on all of the drives in your array, and the controller assembles the file together when you access it. This offers the benefit of offering you more usable space vs. something like RAID1.

For example, suppose you have 4 250 GB drives. You could create a RAID1 array of 500 GB, which uses the other 500 GB as backup. Your computer would see a single 500 GB drive.

In RAID5, the array only uses 1 of the 4 drives for its "working" space, so you would have 750 GB usage available to you. The entire 750 GB is considered 'backed up' by the RAID5 array. Again, your computer would see a single 750 GB drive. RAID5 does not use another drive to backup your data, but keeps a "master log" of where it is putting everything in the array. If you have one of the drives fail, you replace it, and the array detects the drive is replaced, and uses that 'log' to rebuild all of the data on the new drive. This is a pretty simple explanation of how it works, it can become highly technical in the details. Like I mentioned, an internet search will turn up a lot more detailed information if you need it. Have your eyes glossed over yet?

So, what if you like this RAID concept, but don't currently have this feature in your current computer? A new group of devices are emerging called NAS (Networked Attached Storage). Don't you just love acronyms? These are standalone 'mini-computers' that often offer a RAID array inside them. They are connected to your PC by an ethernet cable, an ethernet card (a simple $30 add in card if you don't already have an ethernet port), or to a router or switch on your home network if you already have one. They act like a big external hard drive that has a built-in backup function. When you consider how much it would cost to buy 2 for every external hard drive plus its backup, the cost is actually quite comparable.

The RAID capability, and the fact that I like external devices for their flexibility, prompted me to look into the Buffalo Terastation. This is a set of 4-250 GB hard drives in a box basically the size of a small sub-woofer speaker. There is a total of 1 terabyte (1,000 GB) of hard drive space. You then have to decide how you want to configure this 1 TB of storage into a RAID array. Remember you lose some usable space to make room for the necessary backups. For a RAID1 array, you could have 500 GB usable space. For a RAID5, you have 750 GB. Why wouldn't you chose RAID5 all of the time? Well, RAID5 is a bit slower due to the time it takes the processor to figure out where it stored the bits of your file. There are tradeoffs with everything.

There are also tradeoffs in using NAS vs. an external drive connected via USB2 or Firewire. Network connections are still much slower than USB2 or Firewire, even while using Gigabit Ethernet. I conducted the same tests as above in transfer speeds to add NAS to the comparison table. I note the firmware version as the Terastation had some firmware improvements to address speed since its introduction. These tests are conducted on a Dell 8300 3GHz WinXP Pro PC with 3GB RAM. Update Dec 2006: NAS devices continue to improve with uses of faster hard drives and improvements in network efficiency. Since my Terastation was already filling up, it prompted me to look into another NAS from Infrant... The ReadyNAS NV+. I get speeds approaching my USB2 drive with this new device.

On a file to file basis, the speed difference amounts to approximately 8 seconds in a 70 MB (D2X 16bit TIFF) file open in Photoshop. The same file opens in Photoshop from the external USB drive in approximately 2 seconds. It takes 8-10 seconds for it to open from the NAS. This time to open a file in Photoshop isn't too bad to live with on a single file basis. If you are copying large amounts of files however, it can add up. But you do not have to be at your computer for file copies, it can be a start and leave it process, even if overnight. As mentioned above, the costs per BACKED UP gigabyte (BUGB) are comparable to external USB drives, in fact giving a slight edge to the NAS at the sacrifice of some speed. BUGB is my own invented term because it considers the cost of the storage media AND the media to back it up. The two really should go hand in hand.

So I am currently using my external and internal drives as my primary storage, and I back those up on a regular basis to the Terastation, which is also protected via the RAID5. So I essentially have double backups. I can set the backups to start automatically overnight. I will also store files I use as backups for prints and other miscellaneous items on the Terastation. The Terastation is also used to backup the music collection on my laptop. Overall, it has become a very functional part of my backup and storage system, and protected via the RAID5 array in case any one of the drives should fail.

optical media

So what about optical media such as CDs and DVDs? These are also an alternative in making backups, but I would hesitate to use them for primary storage simply because you cannot fit that many images on a disk. Given their storage per disk is limited, it may hold some advantages as far as 'having all your eggs in a basket. It is cumbersome and time consuming in changing from disk to disk. For backups, they do provide an alternative to hard disk storage. They are certainly less prone to mechanical failures and not as sensitive as magnetic media. DVDs in particular can be sensitive from moving from device to device, a disk burned on one drive may be unreadable on another.

The key to making backups on CD/DVD is to do them incrementally over time to spread out the inconvenience of swapping them in and out. For example, create a folder on your hard drive labeled BACKUP TO DISK and drop your images in there waiting for backup. When the folder approaches 4.5 GB or whatever the close limit of your media is, burn a disk of that folder, and then move the images from the BACKUP TO DISK folder to a permanent location on your hard drive. If you have a lot of images though, you will end up with quite a few disks, and burning & swapping them out is a time consuming process that requires you to be there. This is where hard drives offer a nice advantage, you can set them to backup and leave.

If you are going to backup to optical media, make sure it is one known for its archival properties. Do not go for bargain, no-name brand disks because you are asking for trouble to save a few bucks. One of the more well known brands for reliability long term storage of data is MAM-A, the North American equivalent of Mitsui. Their websites offer extensive information on the research they do on coatings and their longevity.

Optical media weighs in similar to the Buffalo NAS as far as cost per backed up gigabyte. Let's examine the cost per Gigabyte to backup an external USB drive for primary storage to a 4.7 GB DVD. Keep in mind, you will need approximately 53 DVDs to backup this drive. This will likely change in the near future as optical technologies evolve into higher densities per disk.

There are a lot of options available to the digital photographer in protecting their images. But none of them will work if you do not backup your files to begin with, and there is no better time to start than now if you haven't already. To summarize these options, the table below was created to show advantages and disadvantages of each medium based on the tables above and my subjective opinion. I added one additional option of an Internal RAID worth considering if you are buying a new PC, or saavy enough to build your own inside your current computer. Although I have no personal experience with this type of device, one that is connected inside your computer will logically run a lot faster than one connected via a network. In the future as new media becomes available, and hard drive capacities increase, I am sure the cost per BUGB will come down. So please consider the rating of cost to be relative as of Nov 2005 and the specific devices mentioned purely for example purposes.

No single medium is perfect in all areas. A lot of people end up using a combination of multiple media in an effort to cover all possible areas of data loss. What method you choose is ultimately up to your own system of filing your images, how much new computer hardware you want to invest in, and ultimately how many chambers you think are in that gun.

Feel free to email me on your opinions, thoughts, or suggestions on this subject. It is of interest to all photographers in the future. Below are some useful links that have helped me with software and other information regarding image backups. I will update this list from time to time as I encounter new information.