The Case of the Top Secret iPod
It was a gray day in late 2005. I was sitting at my desk, writing code for the next year’s iPod. Without knocking, the director of iPod Software—my boss’s boss—abruptly entered and closed the door behind him. He cut to the chase. “I have a special assignment for you. Your boss doesn’t know about it. You’ll help two engineers from the US Department of Energy build a special iPod. Report only to me.”
The next day, the receptionist called to tell me that two men were waiting in the lobby. I went downstairs to meet Paul and Matthew, the engineers who would actually build this custom iPod. I’d love to say they wore dark glasses and trench coats and were glancing in window reflections to make sure they hadn’t been tailed, but they were perfectly normal thirty-something engineers. I signed them in, and we went to a conference room to talk.
They didn’t actually work for the Department of Energy; they worked for a division of Bechtel, a large US defense contractor to the Department of Energy. They wanted to add some custom hardware to an iPod and record data from this custom hardware to the iPod’s disk in a way that couldn’t be easily detected. But it still had to look and work like a normal iPod.
They’d do all the work. My job was to provide any help they needed from Apple.
I learned that an official at the Department of Energy had contacted Apple’s senior vice president of Hardware, requesting the company’s help in making custom modified iPods. The senior VP passed the request down to the vice president of the iPod Division, who delegated it to the director of iPod Software, who came to see me. My boss was told I was working on a special project and not to ask questions.
I was the second software engineer hired for the iPod project when it started in 2001. Apple Marketing hadn’t yet come up with the name iPod; the product was known by the code name P68. The first software engineer later became the director of iPod Software, the guy who gave me this special assignment. I wrote the iPod’s file system and later the SQLite database that tracked all the songs. Over time, I worked on almost every part of the iPod software, except the audio codecs that converted MP3 and AAC files into audio.
(Those audio codecs were written by two engineers with advanced degrees from Berkeley and Stanford. When they weren’t teasing each other about which school was better, they were writing mathematical audio code that I was scared to touch. You would no more let a regular engineer mess with code like that than you’d let a bike mechanic rebuild the transmission in a Porsche. They had an occasional poker game I played in. The only reason I didn’t lose all my money was that one of them enjoyed his vodka.)
Compiling the iPod operating system from source code, loading it onto an iPod, and testing and debugging it was a fairly complex process. When a new engineer started, we typically gave them a week to learn all this before we assigned them any actual tasks.
The iPod operating system wasn’t based on another Apple operating system like Classic Mac OS or Darwin, the underlying Unix core of macOS, iOS, iPadOS, watchOS, and tvOS. The original iPod hardware was based on a reference platform Apple bought from a company called Portal Player. Portal Player had also provided the lower levels of the iPod OS, including power management, disk drivers, and the realtime kernel (which Portal Player had licensed from another company called Quadros). Apple bought the higher levels of the iPod OS from Pixo, a company started a few years earlier by ex-Apple engineers trying to write a general-purpose cell phone operating system to sell to mobile phone companies like Nokia and Ericsson. Pixo code handled the user interface, Unicode text handling (important for localization), memory management, and event processing. Of course, Apple engineers modified all this code, and over time, rewrote much of it.
iPod OS was written in C++. Since it didn’t support third-party apps, there was no external documentation on how it worked.
Finally, the iPod team developed on Windows computers. Apple didn’t have working ARM developer tools yet, because this was before the iPhone shipped. The iPod team used ARM developer tools from ARM Ltd., which ran only on Windows and Linux.
My job was to get Paul and Matthew up and running on a new operating system they’d never seen before, much less developed for.
I requisitioned an empty office for Paul and Matthew in our building. I had IS&T (Apple’s IT department) reroute the Ethernet drops in that office so they connected only to the public Internet, outside Apple’s firewall, preventing them from accessing Apple’s internal network. Apple’s Wi-Fi network always connects outside the firewall. Even inside Apple buildings, if you’re using Wi-Fi, you need a VPN to get past Apple’s firewall. This wasn’t a collaboration with Bechtel with a contract and payment; it was Apple doing a favor under the table for the Department of Energy. But access for that favor went only so far.
Needless to say, Paul and Matthew weren’t allowed to access our source code server directly. Instead, I gave them a copy of the current source code on a DVD and explained it couldn’t leave the building. Ultimately, they were allowed to keep the modified copy of the iPod OS they built, but not the source code for it.
Apple didn’t provide them any hardware or software tools. I gave them the specs for the Windows computers they needed, along with the ARM compiler and JTAG debugger. They bought retail iPods to work on, several dozen at least, possibly many more.
As with all Apple buildings, everyone had to present an Apple badge to the badge reader to unlock the door and enter the iPod building. Only employees cleared for our building were allowed in. On each floor, there was another locked door and badge reader, and only people cleared for that floor were allowed in.
So every day, Paul and Matthew called me from the lobby since they didn’t have Apple badges. I signed them in as guests and escorted them to their office. Eventually, I arranged to get them vendor badges, as if they were selling Apple coffee or memory chips, so I didn’t have to sign them in daily. I was a programmer, not a babysitter.
Paul and Matthew were smart—top men, even—and with a little help, they were up and running pretty quickly. I showed them how to set up the development tools, build a copy of the operating system from source, and load it into the iPod. We made some temporary changes to the user interface, so we could see that their build was actually running. I showed them how to use the JTAG hardware debugger, which was rather finicky. They dove into their work.
As they learned their way around the system, they explained what they wanted to do, at least in broad strokes. They had added special hardware to the iPod, which generated data they wanted to record secretly. They were careful to make sure I never saw the hardware, and I never did.
We discussed the best way to hide the data they recorded. As a disk engineer, I suggested they make another partition on the disk to store their data. That way, even if someone plugged the modified iPod into a Mac or PC, iTunes would treat it as a normal iPod, and it would look like a normal iPod in the Mac Finder or Windows Explorer. They liked that, and a hidden partition it was.
Next, they wanted a simple way to start and stop recording. We picked the deepest preferences menu path and added an innocuous-sounding menu to the end. I helped them hook this up inside the code, which was rather non-obvious. In all other respects, the device functioned as a normal iPod.
At the time, the latest iPod was the fifth-generation iPod, better known as the “iPod with video.” It was relatively easy to pop open the case and close it again without leaving obvious marks, unlike the iPod nano models that became popular shortly after. Plus, the fifth-generation iPod had a 60 GB disk, so there was plenty of room to have lots of songs and still record extra data. And it was the last iPod for which Apple didn’t digitally sign the operating system.
That was important because it made the fifth-generation iPod somewhat hackable. Hobbyists enjoyed getting Linux to run on iPods, which was hard to do without the special knowledge and tools Apple possessed. We on the iPod engineering team were impressed. But Apple corporate didn’t like it. Starting with the iPod nano, the operating system was signed with a digital signature to block the Linux hackers (and others). The boot ROM checked the digital signature before loading the operating system; if it didn’t match, it wouldn’t boot.
I don’t think Paul and Matthew ever asked Apple about signing their custom operating system build so it would run on the iPod nano. I’m pretty sure Apple would have refused. The larger fifth-generation iPod was better suited to their purposes anyway.
After a few months of on-again, off-again work in their requisitioned office, Paul and Matthew finished integrating their custom hardware into the iPod and wrapped up the project. They moved their computers and debugging hardware back to Bechtel’s office in Santa Barbara. They returned the latest DVD with Apple source code to me, along with their Apple vendor badges. They said goodbye, and I never saw them again. The DVD sat on a shelf in my office for years, until I finally tossed it while cleaning up.
What Were They Doing?
The Department of Energy is huge. Its 2005 budget was $24.3 billion. It’s responsible for the US nuclear weapons and nuclear power programs, including the Los Alamos National Laboratory, which was part of the Manhattan Project. As the DOE’s budget request says:
The FY 2005 budget proposes $9.0 billion to meet defense-related objectives. The budget request maintains commitments to the nuclear deterrence requirements of the Administration’s Nuclear Posture Review and continues to fund an aggressive strategy to mitigate the threat of weapons of mass destruction.
My guess is that Paul and Matthew were building something like a stealth Geiger counter. Something that DOE agents could use without furtively hiding it. Something that looked innocuous, that played music, and functioned exactly like a normal iPod. You could walk around a city, casually listening to your tunes, while recording evidence of radioactivity—scanning for smuggled or stolen uranium, for instance, or evidence of a dirty bomb development program—with no chance that the press or public would get wind of what was happening. Like all other electronic gadgets, Geiger counters have gotten smaller and cheaper, and I was amused to run across the Radiation Alert Monitor 200, which looks an awful lot like a classic iPod.
Whenever I asked Paul and Matthew what they were building, they changed the subject and started arguing about where to go for lunch. Standard geeks.
The Custom iPod That Never Existed
Only four people at Apple knew about this secret project. Me, the director of iPod Software, the vice president of the iPod Division, and the senior vice president of Hardware. None of us still work at Apple. There was no paper trail. All communication was in person.
If you asked Apple about the custom iPod project and got past the stock “No comment,” the PR people would tell you honestly that Apple has no record of any such project.
But now you know.
A great article! Though I’ll bet that Apple is under very strict non disclosure agreements that would make an Apple TV+ movie or series impossible, it could even be morphed into a James Bond-ish spy story. There has been years of coverage about Apple and Netflix dukeing it out over acquiring the rights to James Bond:
And they do have some espionage titles in the works:
I don’t think the secret project would have been a dosimeter or Geiger counter. It is possible, of course. But film dosimeters are highly effective and very small. 15 years ago they were almost exclusively used for this kind of “integral dose” measurement. Additionally, the case of the iPod would block a lot of radiation. Finally, the most important thing to know is if there is radiation at all - you don’t need a complex recording system and you wouldn’t really need to keep it a secret if you are measuring it. The level of detail described here in terms of capturing data to a secret partition would really be unnecessary for radiation dose, since it’s mostly just cpm at a given time. A simple graph or even audio file could record this data, and these iPods had microphone input already as a standard feature.
I think it’s more likely the device was recording some kind of electromagnetic emissions, maybe intercepting WiFi traffic or radio traffic of some kind. There’s also a lot more situations where you would want to stealthily capture traffic or information that way. They could even have another hidden device doing the broadcasting, or just be capturing traffic for analysis later.
That was tremendously entertaining. Crazy to say it but… simpler times.
I find it quite amusing that you helped spys build a tool to exfiltrate data, but expected them to follow the rule not to exfiltrate your own data
While Betchel does do a lot of work for the Department of Energy, I think possible that the custom iPod was actually for the DoD or a three letter agency such as the CIA. This would open up a lot more possibilities as to what the custom hardware was designed to do. It’s also possible that Betchel was making several different types of modified iPods, each with different capabilities. The information that was provided to them by Apple was not specific to the custom hardware, so they could have easily configured different devices with specific custom hardware for various end users. If it was actually just the DoE, the metal back of the iPod would likely make it a non-ideal candidate for a Geiger counter, especially as there where other similarly sized electronics on the market at the time which had plastic bodies. Thanks for sharing, this was an amazing story.
Thanks, great story. Looking forward to more behind the curtain Apple stories as they’ve all been fascinating so far, like Don Melton and friends. Wonder if Phil will ever spill any beans.
Your article really takes me back to old times. I think I was at some of those poker games you talk about, with Steve Bollinger, Nitin Ganatra, Jeff Robbin, etc… I was the long haired kid who drew an inside straight flush to the ace, and then barely bet anything on the hand. Good times.
Very interesting story, I enjoyed reading it.
Fun story, thank you. It’s also interesting to see how this has been picked up – and misunderstood – by other outlets. They’re all coming from the same source (right here) yet get different aspects of it wrong, or at least spun sideways.
The part about no ARM compiler for Mac was funny because Apple definitely had one on Mac back when I was involved in the Newton project! It was hosted by MPW (Macintosh Programmer’s Workshop, a semi-friendly command line driven system).
So…Rubenstein, Fadell, Rogers?
I’m very familiar with MPW. I used MPW from 1.0 beta 1 through the last version. According to wikipedia the last version of MPW came out in 2001. The events in this article take place in 2005. MPW ran in classic Mac OS, but by 2005 everyone was using Mac OS X. Also, iPod used newer ARM cores than those supported by the Newton ARM compiler.
I loved listening to Don Melton tell Apple stories on the old Debug podcast. Worth looking up.
Perhaps work on the Stuxnet delivery vehicle began at this time.
The Debug podcast was an important document of the wider Apple ecosystem, definitely worth checking out.
Did they have you under any kind of NDA?
I found this excellent news about “The Case Of The Top Secret iPod” via Apple News. Though CNN Business characterizes the article as a “blog post,” a click on the link leads directly to the clearly marked TidBITS page.
Congratulations to David Shayer and Adam Engst. It’s always great to see good journalism, especially with a mysterious twist, recognized.
It’s a few hours later and the story is rolling like thunder, getting lots of coverage in national and local media all over the globe. From a quick search, it looks like there are a wealth of pick ups:
And across the globe:
This is barely scratching the surface of all the coverage, and it’s just about 24/48 hours since publication. I hope it results in more TidBITS readers and Talkers.
Thanks for the links, @MMTalker. We’ve been watching them too, and it’s fascinating to see how they put different (often unwarranted) spins on the story).
Yup. I was part of the team the turned out the lights on MPW in 2001. With Greg Branche, Cheryl Ewy, Colin McMaster, Chaz Spillar, (I hope I’m not forgetting anyone).
MPW was fully carbonized, running natively on Mac OS X, but an edict from on-high said that ProjectBuilder was the future, and MPW was the past, and the team was disbanded and everyone was assigned to other projects.
Thank you for an interesting story! These days it is much easier to hide radiation detection in plain sight (if that is what they were doing). We (Image Insight Inc.) develop video analytic software (GammaPix™) that enables digital cameras to measure ionizing radiation that interacts with the image sensor (typically a CMOS). Comparable in sensitivity to the smallest dedicated detectors, it has the advantage of running on completely unmodified hardware.
Thank you again for an interesting Mac/engineering story!
Join the discussion in the TidBITS Discourse forum