How to Lose and Recover iCloud Data

Although I migrated most of my systems to iCloud on the same day, one laptop I use only occasionally for certain work projects lagged behind. When I finally had the time to update the system, I made a critical mistake and nearly lost all my calendars, including my essential work calendar, forever. But thanks to a little trial and error, I managed to pull back from the brink of disaster, and in the process discovered a useful technique for every iCloud user’s recovery kit.

Trial and Error… and Error — My first mistake was completely avoidable. After updating the laptop to Mac OS X 10.7 Lion, I opened System Preferences and started the iCloud migration. Since that computer had previously been linked to MobileMe, I chose the option to merge my data. In the old MobileMe days you could choose to replace your local data with the data stored up in MobileMe, but that’s no longer available for iCloud. Since I had followed the same process on all my other computers, I figured it would be smart enough to avoid duplicates, and I proceeded.

I left the Mac running in the background as I went back to work, and a little later checked in on it only to discover duplicates of every single calendar entry. Looking at the calendar list in iCal I saw calendars for both MobileMe and iCloud. It seemed as if both were running side by side.

I falsely assumed that, unlike my other systems, this Mac kept the old MobileMe data while also connecting to iCloud. That’s when I made my critical mistake… I deleted all the calendars listed under MobileMe.

Immediately iCal looked happier, with all the duplicates removed and matching my other device’s calendars. But within minutes I realized the enormity of my error as all my calendars, on all devices, simultaneously disappeared. Lacking a corporate calendar server, this meant years of old appointments, and months of upcoming appointments, were all gone. As a coworker posted on Twitter, “@rmogull doesn’t exist. iCloud has spoken.”

Since I’m good about backups, I figured I could restore from Time Machine. In a few minutes my calendars were back to normal… and a few seconds later they were all gone again. “This,” I thought to myself, “is bad.”

iCloud Is Not MobileMe — Back in the days of MobileMe, this problem wouldn’t have been that big of a deal. With MobileMe (and .Mac before that) every device was its own authoritative source. Data was synchronized across all devices, but as anyone who experienced a sync conflict could tell you (which was pretty much everyone) each device maintained its own data and made its own decisions.

Thus, if you accidentally deleted a calendar, you could just re-sync from any device that still had the data and propagate it out to all your other devices. Even if you deleted everything from all devices, simply restoring the data on one device could then send it to the others.

But iCloud uses an entirely different architecture. iCloud is the authoritative source for all data on all devices. Local copies always reflect what’s in the cloud. This approach dramatically reduces sync errors and increases consistency and reliability. But it comes at a cost — should you lose data in iCloud, it’s gone forever. And somewhat to my surprise, there’s no backup within iCloud, and thus no way to restore prior states. This is unlike cloud services such as Dropbox that back up everything stored on the cloud servers and allow you to restore selectively using a Web interface.

(I assume Apple backs up or replicates iCloud data somehow in case of server hardware failure; there just isn’t a way for users to access that backed-up data.)

If you attempt to restore data as I did, iCloud sees it as out-of-sync with the authoritative version in the cloud and removes it every time you load it back in. That’s because when you restore data with a tool like Time Machine, you also restore all the file metadata we don’t normally deal with, and that metadata likely tells iCloud that it’s older than the cloud version, which results in the local data being continuously deleted.

How I Saved My Day — After even more trial and error, I next attempted a more-complex process that, to be honest, made me a little nervous. Restoring data directly wasn’t working, but I most definitely still had my old calendars. Even disconnecting from iCloud, restoring my data, and reconnecting didn’t hold, since iCloud still saw the restored data as stale and removed it.

The trick was to disconnect from iCloud, restore the calendars, stay disconnected from iCloud, export the calendars, reconnect to iCloud, and then import the just-exported calendars. Here’s the process in more detail:

1. Go into System Preferences > iCloud and sign out of iCloud, which deletes all iCloud data from the device (including documents, contacts, and email). But don’t worry, it’s all still safe at Apple’s data center.

2. Using Time Machine (or your backup program of choice) restore the missing data. In my case this was the ~/Library/Calendars directory. (In Lion the Library folder is hidden by default; in the Finder, hold Option and choose Go > Library before activating Time Machine. Or, you can reveal it using a third-party tool or the command line; see “Dealing with Lion’s Hidden Library,” 20 July 2011).

3. Open the application that uses the data (iCal, in my case). Then export the data. For calendars, you can export an entire calendar as an .ics file by using iCal File > Export > Export; I saved my calendars to the Desktop. (iCal also supports exporting an iCal Archive, but I didn’t test that.) Other applications — such as exporting contacts from Address Book — will have different processes.

4. Go back into System Preferences and sign into iCloud again.

5. Watch as your data disappears again. It’s mesmerizing. In a bad way.

6. Create new iCloud calendars with the same names as your old ones (I had one name that iCloud kept changing on me, so I picked a new one that was almost the same. I suspect this was due to how quickly I was making these changes). For the rest of these steps, I’m going to focus on iCal, but a similar process should work for other applications.

7. Import the calendar files on your Desktop into the new, empty iCloud calendars. If you try to import the calendars without creating iCloud homes for them, you will be able to import them only locally, and not to iCloud.

8. Re-share any shared calendars and send out invitations. I share my work calendar with my coworkers and my home calendar with my wife, and when I initially deleted my calendars I disappeared from their systems (prompting my coworker’s tweet).

9. Ask everyone to send you sharing invitations again so you can see their calendars. Yes, I deleted my shared calendars, which fortunately (even though I had write access) deleted only my access and not my coworkers’ futures.

The entire process didn’t take long, but it was nerve-wracking considering how much important information I keep in those calendars. After I posted about my travails, fellow TidBITS staffer Michael Cohen wrote:

What I always do when making major changes to my calendar setup is to first export my iCal data. That makes it much easier to repopulate iCal (and, thus, iCloud) with my data if I have a brain freeze or other calamity.

What a good idea! I was hoping I could use AppleScript to automate this process and make non-iCloud backups of my calendars, but unfortunately the iCal export feature isn’t AppleScript-accessible. I’ll just learn my lesson and make sure I manually export backup copies of important data before mucking around with anything major in iCloud in the future.

iCloud data isn’t necessarily at greater risk than it was in MobileMe, but when you delete it from iCloud, it’s gone from the source, and recovery is definitely more difficult than it used to be. And than it should be.