Clone Wars, or How My Backups Ate My Photos
An occasional public admission of stupidity can be good for the soul, a bit of psychic self-flagellation. I’m sure you all know by now how obsessive I am about backups, so I’m sure you’ll enjoy a bit of schadenfreude related to this story of how I’ve been losing data for two months without even noticing, thanks to my backups. That’s right, my backups ate my data.
Down the Rabbit Hole of Data Loss — It all started when I received email from a PR person asking if I’d had a chance to check out Boinx Software’s You Gotta See This iOS app, which makes it easy to create a panorama with an iPhone or camera-equipped iPod touch. The app worked fine, but because of the extremely wide aspect ratio of the panoramas, the results didn’t ring my bells. In an effort to explain this to the PR person, I launched iPhoto to snag a recent panorama I’d made, along with a single image of roughly the same scene that I wanted to show her for comparison. But when iPhoto launched, the photos I wanted
weren’t there. And when I looked more closely, the last event showing was from two months ago. Uh oh.
Data loss always gives me the horrible sinking feeling in my stomach that I get when I know I’ve done something stupid and There Will Be Consequences. I realized quickly that the problem was that I had somehow been using an iPhoto Library package on a backup disk. In my panic, I initially blamed Time Machine because, well, I don’t trust it. But after my pulse settled down, I realized that Time Machine was not at fault, and that the blame instead lay with my backup strategy, aided and abetted by iPhoto (this was iPhoto ’09, but I suspect that iPhoto ’11 would also be vulnerable) and another nemesis, Spotlight.
My backup strategy, which I’ve previously been quite proud of, is threefold. First, I use Time Machine, sending archival backups to a partition on a second internal drive in my Mac Pro. Second, to provide offsite backups and as a secondary archival backup, I use CrashPlan+ to back up my entire home folder over the Internet to a hard drive attached to a friend’s Mac. Third, for quick recoveries and in case my main hard drive (named Zeus) dies a sudden death, Carbon Copy Cloner clones my startup drive to another partition (named DoppleZeus) on the second internal hard disk. That clone runs every night before my Mac goes to sleep.
Normally having DoppleZeus mounted isn’t a problem; the way applications generally default either to previous locations or default locations (like ~/Documents
) ensures that there’s no confusion between my main drive (Zeus) and the clone (DoppleZeus) when opening or saving files. Since I got the Mac Pro in early 2008, this approach has worked fine.
But I hadn’t counted on iPhoto. When you Option-launch iPhoto ’09 to switch between iPhoto libraries, iPhoto doesn’t give you a standard Open dialog like iTunes does. Instead, it displays a custom dialog that lists all the iPhoto Library packages it can find via Spotlight. Since DoppleZeus is an exact clone of Zeus, iPhoto displays the same libraries on both, differentiating only by a path that’s displayed at the bottom of the dialog. Apparently, at some point in August, I had switched iPhoto libraries to test something, and when I switched back, I accidentally chose the backup version on DoppleZeus. (Although iPhoto won’t see iPhoto libraries on a disk that’s in the Privacy list in the Spotlight preference pane, if you’ve
ever chosen a library on that disk before adding the disk to Spotlight’s Privacy list, iPhoto remembers the library’s location from then on in the com.apple.iPhoto.plist file in ~/Library/Preferences
, regardless of any Spotlight settings.)
I’m sure you can imagine what this means. Every time I imported photos, they were imported into the iPhoto Library on my DoppleZeus backup disk, and the following evening Carbon Copy Cloner deleted them as it followed my commands to keep DoppleZeus looking exactly like Zeus.
I hadn’t been doing much with iPhoto, so it took me two months to notice, meaning that I’d lost every imported photo since mid-August. And they were totally gone, with no chance of recovery. Since both Time Machine and CrashPlan back up Zeus and not DoppleZeus, there was no backup of those photos—they never existed on a drive that was itself backed up.
Before I explain how to avoid this problem, there is a happy ending. I take photos with two cameras: my iPhone 4 and a Canon PowerShot SD870. I use the iPhone when I haven’t anticipated a photo opportunity and failed to pocket the PowerShot SD870. Any time I think of it, I use the PowerShot, since it takes better photos. But since I’ve been busy this summer, it turns out that the only photos I’ve imported into iPhoto were those taken with the iPhone. All my “good” photos are still sitting on the PowerShot’s SD card. I feel a bit like my parents back in the 1980s, who would often forget to develop rolls of film for months. So, while I’m not happy that I lost all my iPhone photos since mid-August, it’s not that big of a
deal.
This problem is pretty specific, and I doubt most people switch iPhoto libraries as often as I do for testing, but be warned if you do use iPhoto’s Option-launch approach to switching. Or use Fat Cat Software’s iPhoto Library Manager, which requires that you specify the iPhoto libraries you want to switch among manually, rather than relying on Spotlight to find every last one.
An Ounce of Prevention — So how could I prevent this from happening again in the future, and, what’s the solution for those who also use Carbon Copy Cloner or Shirt Pocket’s SuperDuper to clone to another hard disk? My buddy Andrew Laurence gave me the hint I needed when he commented on Twitter that backups should be out of sight, out of mind, read-only, and far away.
Looking at my three-pronged backup strategy, CrashPlan’s backups meet all four criteria. Apart from the partition’s icon on my Desktop, my Time Machine backups are out of sight, and they’re essentially read-only because Time Machine manages the permissions on the folder containing the backups to prevent you from writing to it. (Time Machine fails the out-of-mind test because it’s disconcerting when it’s running, and it isn’t designed to meet the far-away test by allowing backups over the Internet.) My Carbon Copy Cloner setup is out of mind, since it runs only at night when I’m not at the Mac, but there’s no easy way to make it read-only (since Carbon Copy Cloner has to write to the disk), and I don’t want it to be far
away.
The trick is in the remaining goal—putting the Carbon Copy Cloner backup destination out of sight while I’m using my Mac. To achieve that, I can mount the DoppleZeus volume right before Carbon Copy Cloner’s backup runs, and unmount it afterward. Were it on its own disk, rather than sharing the disk with the Time Machine partition, a nice side-effect of this approach would be that the drive wouldn’t be spun up so much, and it might save a tiny bit of power.
In fact, Carbon Copy Cloner’s documentation gives a script for unmounting a volume after backup with a post-flight script, but it notes that a pre-flight script can’t mount a volume because Carbon Copy Cloner performs sanity checks for the existence of the destination volume. Instead, the documentation suggests mounting the drive in some other way and having Carbon Copy Cloner start the backup when it sees the drive appear.
There are undoubtedly a bazillion ways of mounting a disk at a particular time, but I chose to have Stairways Software’s Keyboard Maestro execute a short shell script at the appropriate time. The script is simply diskutil mount disk0s2
(you can find the disk identifier—the disk0s2
bit—with the command disktool -l
).
For the unmount script, I simply copied the script from the Carbon Copy Cloner documentation to a text file, saved it as UnmountDoppleZeus-CCC.sh in ~/Library/Scripts
, specified it in Carbon Copy Cloner’s Advanced Settings dialog, and then saved the task, setting Carbon Copy Cloner to execute the entire task whenever the disk appears.
It works perfectly, with Keyboard Maestro mounting the disk, Carbon Copy Cloner noticing and performing the clone action, and the post-flight script unmounting the disk.
Actually, there is one additional problem to solve. Whenever I reboot, the DoppleZeus volume mounts automatically. Luckily, this is solved easily with another Keyboard Maestro macro that executes at login; it simply unmounts DoppleZeus right away. (If you were doing this, you’d want to replace “DoppleZeus” with the name of your disk.)
Although I don’t currently use SuperDuper in my backup strategy, it’s an extremely popular, useful program for making duplicates as well. But it requires a slightly different approach. Nothing needs to be done to mount the disk before backup, because SuperDuper does that automatically as needed. And if you’re using an external hard disk or a removable disk, you can make it unmount after SuperDuper runs by choosing Eject from the On Successful Completion pop-up menu in the General view of
SuperDuper’s options for the backup. So if you’re not using an internal disk, you’re done.
However, I am using a partition on an internal hard disk in my Mac Pro as the backup destination. Because the Finder doesn’t give an internal disk an Eject button in the sidebar, the current version of SuperDuper doesn’t see it as ejectable. The next minor revision will do this, according to SuperDuper author Dave Nanian (I suspect he named the program SuperDuper just so it would sound like an adjective when used in front of his name—clearly I need to write a book called “BestSelling”).
In the meantime, Dave gave me a script that does what’s necessary, forking off an independent process that unmounts the disk that was just backed up to.
#!/bin/sh
nohup /bin/bash -c "sleep 30; diskutil unmount \"$4\"" &
To use this script, I simply put the above two lines in a text file called UnmountDoppleZeus-SD.sh in ~/Library/Scripts
and marked it as executable with the Terminal command below.
chmod u+x /Users/adam/Library/Scripts/UnmountDoppleZeus-SD.sh
Then I specified it in SuperDuper’s Advanced options, scheduled SuperDuper to execute at the appropriate time, and all was golden.
Out of Sight — So, regardless of whether you use Carbon Copy Cloner or SuperDuper to create a bootable duplicate of your hard disk, you can ensure that the clone is mounted only when necessary, and is properly unmounted and out of sight the rest of the time. And hopefully, that will prevent you from making the kind of mistake I made with my iPhoto Library.
Since I've been spending part of my day today restoring my iPhoto library which iPhoto '11 so rudely destroyed, I've been thinking about backups.
I'm wondering if you might experience a problem if you needed to mount your DoppleZeus disk to recover a file which would then trigger CCC to start a backup - one that you may not want to occur at that precise moment. I ask because something similar, but not identical, happened to me today. I was in the middle of restoring some files from my cloned backup when I stepped away from the computer for a moment. SuperDuper started (it's set to start at a specified time) and copied my corrupted drive onto my backup. Fortunately for me I had a separate Time Machine backup so no harm.
Just wondering
Charlie
Yes, this is a slight issue, but it's easy to stop CCC when it kicks in on disk mount. (I've been doing a lot of that while testing.) If it were a real issue, I could schedule Keyboard Maestro to mount the disk at 11:16 PM and schedule CCC to back up at 11:18 PM, or something like that.
I had the same experience with iTunes, requiring me to re-rip a multiple-CD album. Thanks for the unmount at login script; it was the missing piece.
Another approach: protection via *hand-carried* offsite backup.
To protect against easy mistakes (deletion of a file I should have kept, etc.), I have TimeMachine automatically backing up to a Drobo box -- but my absolute last-ditch defense is a pair of 2 TByte external drives. One's always in a desk drawer at work, the other one's in transit.
When one's at home, I connect it at bedtime and start up "Super Duper" to clone my iMac's drive. When I get up in the morning, I disconnect it and stuff it in my briefcase. This way, I can't get confused about what photo library I'm using, and I've got protection against home catastrophes.
Worth considering...
Adam...what about iPhone backups?
You might want to try to go into iTunes, and restore your iPhone from a previous backup (I think located somewhere in ~/library/iTunes or something). You might be surprised to find that your photos are still there in THOSE serial backups, even though you erased the pics from the iPhone every time you imported into iPhoto.
This should work unless 1) you cancel or prevent backups of your iPhone whenever it is connected or 2) you also have THAT library somehow on your "backup" drive.
Sorry to hear about your loss. After a very similar experience years ago, my Mac Pro has four 1TB drives: the start-up disk (always mounted) and three complete clones (unmounted unless SuperDuper! mounts them). The three other disks are Daily, Weekly and Monthly clones made by SD! This in addition to Time Machine and AASync that runs a nightly SFTP backup to Dreamhost of documents only.
Why not exclude DoppleZeus from Spotlight? Wouldn't you be constantly plagued by duplicate found items if you didn't? (Just a thought, haven't tested)
I had thought of that, but my initial testing was inconclusive. I know more now, and here's the deal. If I had put DoppleZeus in Spotlight's Privacy list initially, or if Spotlight hadn't lost it (I can't say for sure I did this, but I don't trust Spotlight not to forget settings), then I'd never have been presented with the DoppleZeus version of the iPhoto Library.
However, I was presented with it at some point, and I selected it, and once I did that, iPhoto remembered it in the com.apple.iPhoto.plist file, so regardless of whether DoppleZeus was in the Privacy list or not (and even after I deleted the Spotlight index on the volume manually just to be sure), iPhoto remembered that I had once selected it and made it available thereafter.
All of these capabilities make sense, but they can combine to undesirable behavior.
Just a side note on something that burned me: TimeMachine does _not_ back up your iPhoto library if iPhoto is open! I tend to leave it open, because I use it so much. Then my drive died, and I lost months of photos. Luckily for me, like Adam, I had an off-site backup (backblaze) which does back up everything, so I was able to recover from a drive crash (with a bunch of fiddling to get iPhoto to recognize the restored images). It is totally amazing to me that Apple doesn't at least pop up a warning to tell you that TimeMachine did not back up your photos when it last ran...
I prefer to use iPhoto Library Manager to open my various iPhoto libraries. Since you have to add each library to the list yourself there's no danger that you'll open the library on your clone.
Yes, absolutely, which is why I mentioned that in the article too. What worries me more is that there may be other situations where a confusion with a clone disk could cause problems, potentially even more significant ones.
I never understood why people risked their data by using hacks like CCC or SuperDuper. Apart from the fact that the latter actually costs money, both have caused a lot of trouble for a lot of people.
Instead, people could opt to use Apple's built-in and rock stable cloning tool that comes with every OS X installation. It does the same (minus incremental, which you don't want on a clone anyway), but it does it reliably and for free. Thanks to block copies it's also much faster than rip-off hackware like CCC or SD.
/Applications/Utilities/DiskUtility > Restore
(select 'erase destination' to make the clone bootable)
Disk Utility certainly does have this capability, but I disagree strongly that CCC and SuperDuper are hacks or rip-offs in any way. Both offer significant features, such as scheduling, pre- and post-flight scripts, copying only of changed files, and so on, that go way beyond what Disk Utility does.
You can script DU (or use asr on the CLI -> shell scripts) for pre/post-flight tasks.
Incremental backups are indeed something DU doesn't do, but CCC and SD offer. But why would you want to do that? Think about it. A clone is an exact bit-by-bit copy. By its very definition you do not want to modify it through an incremental procedure.
Backups can be incremental (like Time Machine). But cloning should never be incremental. The extra 'functionality' offered by CCC and SD are the very cause of much of the trouble these tools have given people.
I guess I simply haven't heard from people using CCC and SD and suffering notable problems.
I haven't heard about these problems, either. Can you point to where people are having difficulties? I've used smart clones from Super Duper to boot and restore drives several times.
You'll read about a ton of problems if you frequent Mac-related forums. Just google for issues with clones created with CCC and SD. Supposedly "bootable clones" not booting users' Macs is the most common but by far not the most serious.
In the end you get what ask for. If you want a reliable clone, don't fiddle (i.e, increment, etc.) with it. If you want to fiddle, make sure you use another backup method and another backup volume in addition.
We'll look into it. It hasn't been our experience among many users here at TidBITS and our colleagues, but that doesn't mean that our experience invalidates everyone else's! Thanks for the information.
I worry when I hear people using the terms 'clone' and 'backup' synonymously. Each has their place, but they aren't the same thing. In my view, a far better approach to cloning is to use a SoftRAID RAID 1 mirror. This way your clone is always up to date without you having to do anything; if the clone is on an external drive, it can be instantly transferred to another Mac.
A clone or a duplicate is a backup, but it is only a part of a real backup strategy (along with versioned backups and off-site backups).
The problem with a RAID 1 mirror, in comparison with a duplicate made on a schedule, is that any mistakes or logical corruption are immediately and automatically transferred to the mirror, rendering it useless as a backup. RAID is great for protecting against physical damage to the source drive, so it too can be part of a backup strategy, but isn't a backup on its own because it's so easily corrupted with bad information from the source.
Regarding the interim method of getting SuperDuper! to unmount a volume after backup (a GREAT idea), the command in the article
nohup /bin/bash -c "sleep 30; diskutil unmount "$4"" &
seems a bit cryptic. Does this command indirectly always refer to the volume that SuperDuper has just backed up, or does it have to be changed in some manner for other peoples' backups?
The $4 refers to the just-backed-up disk, as I understand it.
I knew $4 was a variable and assumed it represented the disk you want unmounted but I wasn't clear where the data inside that variable was coming from. Then I looked at the screenshot again, understood SuperDuper would run it and guessed $4 was the 4th parameter passed to this shell script. Sure enough, on pg. 43 of the SuperDuper manual, there's a list of parameters passed to pre- and post- scripts and the full path to the destination disk (e.g. /Volumes/DoppleZeus) was the 4th parameter.
Dear Adam, I know exactly how you felt (that horrible sinking feeling..) from times when I succeeded in deleting important files from my harddrive with no recovery possible. The core of your problem is, I think, the non-transparent way in which iPhoto selects its photo library.
I can totally support the modus of using a disconnectable external harddrive. I keep one Carbon Copy clone on a Freecom drive and a Time Machine backup on another drive in our weekend home. CCC + Time Machine is a very sensible double backup strategy.
Yes, it's really too bad that iPhoto presents a list, rather than a standard Open dialog, which would have made clear where I was in the disk hierarchy. It's a case of trying to make something easier and making it worse in the process.
Jolin Warren just gave me a good tip - to prevent a partition from mounting at login, the Unix way would be to add a line to the /etc/fstab file. See this writeup for more info:
http://hints.macworld.com/article.php?story=20060930150059172