As I’ve already whined about on social media, I had a hard drive failure this past week. I’ve been pretty lucky when it comes to hard drives. Usually, they fill up and need to be replaced before anything catastrophic happens. Rarely do I consider just their physical age if they are passing regular drive checks and not filled up yet. But the events of this week have me rethinking that aspect. The drive that failed passed a physical check just 2 days prior. It wasn’t making any unusual noises or showing other warning symptoms.

A crash is certainly one way to test the integrities of your backup discipline as it did mine. After the initial panic of seeing the drive that held all of my developed photographs on it report zero data in the master directories, I had to go through a mental checklist of how diligent I’ve been in my backups.

And that answer was – mostly diligent.

Three Copies of All Data Strategy

It’s a generally accepted best-practice backup strategy to have 3 copies of all of your data somewhere. The primary, a local backup, and then an offsite/emergency backup. My local backup consists of a ReadyNAS NV+ that I purchased back in 2006. My offsite is in the form of cloud backup with a service called CrashPlan.

One hole in my discipline is that I don’t keep my NAS (Network Attached Storage) running 24/7. The thing is old, it’s noisy, and it’s an energy hog…but it still works. And if something still works, I don’t want to contribute to the growing e-waste problem just because something is old and outdated. So, I tend to only turn it on when I am ready to initiate a backup, and then shut it off. The last backup I happened to run was on December 10.

Cloud backup

I’ve been using CrashPlan for 9 years now. I remember researching cloud options at the time and discovered many of the services then would throttle your uploads if you went beyond a certain amount. So the service would drastically slow your initial seeding of the cloud with your data if you went beyond say 500 GB. I don’t know how common that is today, but something to be wary of if you are shopping around.

At the time there were some calculators for the different services – and for some of them that throttled your uploads, based on the size of my backup, it would take 6-8 months to upload everything (with a computer running 24/7)! CrashPlan was one that did not – so it was a good match for a large photo archive. Using the cloud for backup really depends on your internet upload speeds as well, as they tend to be much slower than download speeds. My current service has 10 Mbps upload, and it used to be 2 Mbps. The 10 Mbps has OK for maintaining my backup.

CrashPlan changed quite a bit when it was bought by Code42. They became much more business-oriented, and as a result for better or worse, their software seemed to cater to IT professionals rather than consumers. But, I already had a lot of data stored there, the pricing plans were still reasonable, so I just remained with it. Overall they have been reliable, which is probably what matters the most when it comes to a backup company – but it has its quirks and lacks some features.

Recovery

The recovery process/diagnosis was initially complicated with the fact that my entire computer wouldn’t boot after this internal data drive failed. (I have 6 drives in it – 4 HDDs + 2 SSDs) That led me down a rabbit hole of thinking maybe this was the final leg of my aging 2008 Mac Pro. After some trial and error, I was able to reboot with the internal drive completely removed. A sign something seriously went wrong with the electronics of this drive, with little warning.

A Connection to the Massive Flood in Thailand?

The drive was a 3 TB Seagate Barracuda drive purchased in 2013 that apparently has a notorious reputation that I only came to learn about after the failure. The backup service BackBlaze did a study on the failure rates of this particular drive, suggesting a potential tie to flooding in Thailand around that time. It certainly highlights how global supply chains can be disrupted (in both quality and supply issues) from more frequent and severe weather events.

My drive’s failure may have absolutely nothing to do with that situation as it was indeed nearly 8 years old, and an active internal drive, but it does make you wonder.

Death to Obiwan, Long Live Grogu

Knowing I had a gap in my local backup, I was relying on my cloud backup to be the savior for anything I’ve stored on that drive over the past month. I had to go through some techno-gymnastics with CrashPlan just to check what files I had in the cloud, so I started recovery from the NAS and would determine the NAS vs. cloud difference after. I just happened to have a new Western Digital 4 TB drive that I intended to use to replace an old 500 GB drive, but now it will replace the failed one.

Of course, I name my drives after Star Wars characters. I have a few random externals named Bossk, Greedo, and C-3PO. As it turns out (spoiler alert), Drive Obiwan died just about as fast as Jedi Obiwan did in being struck down by Darth Vader. Enter Obiwan’s replacement – one of the true saviors of 2020 fall entertainment season – Grogu. Grogu would take on Obiwan’s reincarnated soul partially stored on the Wookie NAS. 🙂

Needless to say, this resulted in some entertaining discussion with CrashPlan’s tech support in trying to straighten things out with the cloud. Discussions of Obiwan’s presence still existing in the cloud, but was dead, and I needed to download him into Grogu while recognizing that Grogu was really Obiwan. As it happens, I, fortunately, had a tech quite familiar with The Mandalorian,

Using Lightroom in Recovery

Ultimately I found about 26 GB of data difference between my NAS and my cloud backup. The most useful tool in determining that was Lightroom. It is quite easy in Lightroom to reassociate content from an old drive to a new one in your catalog.

Lightroom Tip: By using the structure on your hard drive of DRIVE NAME > “MASTER IMAGE FOLDER” > subfolders, you can make it easy to update all the photos and sub-folders within the MASTER IMAGE FOLDER if you happen to change the drive name (or get a new drive). Lightroom will grey out the old drive if it is removed or renamed indicating it doesn’t know where it is. Then all you have to do is click the old MASTER IMAGE FOLDER and point it to the new drive MASTER IMAGE FOLDER and Lightroom will update all the content within it.

This may be common knowledge for users of Lightroom, but it is particularly handy in a drive failure event. The next step was to find what images were still missing due to the gap in backup dates between my NAS and the cloud. This is where I wish CrashPlan had a SKIP FILE feature. If it had this, I could simply ask CrashPlan to download all the data from Obiwan to Grogu, skipping the files that were already recovered and matched. It makes it much easier than trying to look through sub-directories to find files that were updated after that December 10 backup date.

This is where Lightroom helped greatly. Lightroom will indicate it can’t find the associated photo in the catalog with a file on your storage drive by placing a question mark in the corner of the thumbnail. By using the menu LIBRARY > FIND ALL MISSING PHOTOS – Lightroom created a quick reference of all the files I needed to recover and download from the cloud backup.

Lessons Learned

For as long as this post is, it is really a summary of all the steps I had to take to get my computer back up and running, and to determine exactly what I needed to do to recover my photos. I can only hope someone may read and think about the potential holes in their own backup strategy as a result. Here are some of the concluding lessons I came away with;

• My NAS for recovery is slow but better than nothing. It took a full 16 hours to transfer all the photos from this local backup to the new hard drive. I made note of the transfer rate of 28 MBps. I am going to update to a USB3 drive as my primary photo backup as file transfer will be about 10x as fast and keep the NAS just for non-critical backup items. It will also be less of a hassle and easier to keep up to date as an always-connected external drive. I know people like NAS Raid setups for the redundancy aspect, but I already have that with the cloud.
• Keeping my Lightroom catalog on a separate drive from my primary photo storage helped. I didn’t need to be concerned with recovering the catalog (even though it is backed up also), and it was a useful tool in identifying missing photos.
• CrashPlan saved my butt for the 26 GB of photos that I missed backing up locally, but the service could use some more features in Recovery (eg. Skip file), or finding all files updated after ‘x’ date. The CrashPlan “synchronizing block information” is painfully slow which caused some of those techno-gymnastics I needed to do and is so far into the weeds I won’t bore you with details.
• The three copies philosophy is a REALLY good idea. I will strive to be a better disciple.
• Be wary of components purchased during times of large scale natural disasters in areas that are known as the major global supplier of such components.