All men have fears.
Many fear those physical threats wired into our souls through millions of years of surviving this harsh world. Fears of heights, confinement, venomous creatures, darkness, or even the ultimate fear of becoming prey can paralyze the strongest and bravest of our civilization.
These are not my fears.
I climb, crawl, jump, battle, and explore this world; secure in my own skills. My fears are not earthly fears. My fears are not those of the natural world. This is a story of confronting my greatest terror, living to tell the tale, and wondering if the threat is really over.
The tale starts, as they always do, on a dark and stormy night.
It was the beginning of August and I had just arrived in Sydney, Australia to begin a two-week tour speaking on security issues throughout Australia and Asia. I was staying in a hotel overlooking Darling Harbor, one of the main tourist spots in the city. My room was on the top floor, with a large wall-sized window filled with an expansive view of the harbor and downtown Sydney. Australia is in the midst of a drought, but that evening the sky was filled with dark clouds glowing with the lights of the metropolis. I remember, in my innocence, thinking the view was beautiful, especially the strange glow where the city met the night. At least, that’s what I thought caused the glow; now, I’m not so sure.
I was half-drugged from the combination of jet lag and the pill I took to help me sleep, but I remember noticing some oddness with my mail. Apple Mail was showing a large amount of spam in my Inbox; strange, since SpamSieve normally catches nearly all of it. I didn’t think much of it, or much of anything as the drugs and fatigue brought me down, and I drifted off to the restless sleep of the traveler.
The next morning I woke up, washed the sleep from my eyes, and settled in front of my MacBook Pro to catch up on email before heading off to meetings for the next few hours.
Something was wrong. Very wrong.
My Inbox was flooded with the evil detritus of the seductive false dreams offered by spam. I restarted SpamSieve, which returned an error telling me its corpus was corrupt.
“Reboot” the former Windows sysadmin in me commanded, and I restarted my Mac. I should have resisted that instinct, because with that one reboot I left the world as we know it, and entered a dark dimension of shadows that changed me forever.
The system rebooted normally and I logged in, but that’s where sanity ended. I first noticed my Dock; no longer the small, customized list of my favorite applications, but a gargantuan beast taking up inches of my screen with the default applications of a neophyte. On the menu bar only half of my usual icons appeared, and the mighty Quicksilver was no longer among them.
I launched Mail, and was prompted to set up my first account. I launched Safari, and was greeted not by Penny Arcade (my home page of choice), but the default Apple News page. I frantically started clicking, but application after application had reset to the default settings of a newborn Mac. Worst of all, Parallels had reset itself and could no longer see the Windows virtual machine I use to access everything at work.
I was on day two of a two week trip on the other side of the world. This was bad. Really bad.
I rebooted again, and again, and every time my system did something different. Sometimes settings stayed, sometimes they vanished, or some settings would stick while others disappeared. My menu bar rotated applications; I never knew what would work and what wouldn’t after logging in. “New” large files seemed corrupt – any utilities I downloaded failed to open, and file attachments were all corrupt, yet most large files on the system before… the event… were still fine. I first thought “virus,” but as a security expert I take rigorous precautions and the chances of an infection were very low. Only one thing could explain such unnatural behavior.
“Ghosts,” I thought to myself, “why did it have to be ghosts?”
As a chill settled into my spine I shut the lid of the MacBook Pro and left for my meetings. Walking the streets of Sydney I could feel the evil emanating from my bag, casting a pall wherever I walked.
My meetings finished and I raced back to my hotel room. Lacking the chickens, goats, salt, or newts for a proper exorcism I had nothing to rely upon but my technical skills. I did try chanting while holding a chicken sandwich and plastic knife, but hotel security convinced me that course of action wouldn’t bring the results I desired.
I cracked my knuckles, hunched over the keyboard, and began battling the evil head on. After more reboots and some experimentation I realized that there was something wrong with my FileVault. Built into Mac OS X, FileVault – if you enable it – encrypts your home directory as a sparseimage file, protecting your data if someone steals your laptop. I’ve been a security professional for a long time, and last year decided it was time to follow my own advice and protect my laptop (you can read more about how I use FileVault on my blog).
Whatever this supernatural creature was, it was restricted to that encrypted prison. When I logged into my separate maintenance/backup account (you do have a secondary admin-level account for troubleshooting, don’t you?), everything behaved normally. It was only when logged in under my primary account, the encrypted one, that my computer was possessed. Normally when FileVault fails, it fails hard, corrupting the encrypted data and destroying your home directory. My case was something… different… and supported my theories of the supernatural. I could log in fine, and run most of my applications, but settings and data randomly corrupted and behavior varied from login to login. Facing two weeks with nearly no ability to get my normal work done, I was starting to get desperate. It was time to call in a warrior to fight the demons – the mighty DiskWarrior.
DiskWarrior is a popular Mac utility known for nearly miraculous saves of corrupt hard drives (see “Shootout at the Disk Repair Corral,” 2007-09-07). Earlier, walking to my appointments, I saw a Mac store on the street. I packed up and sprinted over, grabbed a copy of DiskWarrior, and rushed back to my hotel. (And yes, I did pay for it first; troubleshooting is way harder when you’re rotting in an Australian prison.)
All for naught. I booted off the DiskWarrior CD and scanned my drive, but whatever haunted my system was more powerful than even this epic hero of saved drives.
I tried a few other desperate actions. At this point I realized there was some strange corruption to my FileVault image; not enough to keep me out or ruin my data, but enough to cause all this strange behavior. I tried to disable FileVault, but there wasn’t enough space on my hard drive to decrypt all that data. I tried deleting or moving files to an external drive, but FileVault wouldn’t recognize or recover the free space. I sacrificed file after file, including default applications, all in vain.
In the end I realized that the only way to defeat this demon would be to travel back in time and warn myself of the impending doom. Or restore from a backup. (You do have a recent and functional backup, don’t you?) Since I lacked the proper tools in Australia to build a functioning time machine, I’d just have to wait until I returned home and could restore from the backup I made using SuperDuper before leaving.
For the rest of my trip I’m amazed that airport security never picked up on the danger hiding in my bag (okay, as a security expert it’s not all that surprising). I traveled from Sydney to Perth, to Singapore, to Malaysia with nary a suspicious glance from any of the innocents surrounding me. Since none of my settings would save, I logged in using my backup (unencrypted) account and carried on as best I could (thank goodness for webmail).
Two weeks later I returned home and restored from my last backup, banishing the poltergeist from my Mac. Everything was back to normal, and I pulled what few files I changed on the trip from my external drive. Thinking back, I vaguely remember forcing a shutdown on the plane when logging off seemed to hang. I’m now certain that it was this forced shutoff (by holding the Power key for ten seconds) that created a dimensional hole between our world and the next, allowing the malevolent spirit to inhabit my drive. Either that or it corrupted the encryption, but not enough to lock me out. It was only my backups, extra maintenance account, and some basic investigative skills that kept me from being completely crippled.
I also realized that I’d taken an incredible risk. A complete backup of my hard drive is important, and externally bootable (which is convenient), but if that corruption was also in my backup my files may have been banished to the nether-world forever. I thought I was safe, but I was taking far more risk than I realized. Coming home I realized I also need file-level backups of my data within the FileVault, so I rushed to the Take Control site to consult Joe Kissell’s “Take Control of Mac OS X Backups” on backup strategies. Soon we’ll all have a Time Machine built into our Macs (and hopefully it will work well), but until then we’ll have to take the extra steps ourselves to protect our data.
For now, life is normal. I’m back in my home town of Phoenix, Arizona without a cloud in the sky. I’m working again, but I have yet to banish the lingering fear that my beloved laptop is now a portal to a darker world.
[Rich Mogull been working in the security world for 17 or so years, and breaking computers (usually by accident) even longer. After about 10 years in physical security (mostly running large events/concerts), he made the mistake of getting drunk in Silicon Valley and telling someone he “worked in security.” Next morning he woke up with a job as an IT security consultant. That’s not totally true, but it’s far more amusing than his full biography. He currently works as an independent security consultant and writer through Securosis.com and previously spent seven years as an analyst with Gartner. Rich has also worked as a paramedic, done stints as a firefighter and with Rocky Mountain Rescue, and recently retired from ski patrol when he moved to sunny Arizona. He still dabbles in disaster medicine, when nature cooperates.]