Exactly what is it about Mac OS X that is responsible for the security vulnerability currently being discussed? The situation is a little confusing, and I may be muddling some of the details, but here’s my current understanding of the situation.
As you know, when you double-click a document in the Finder, the application that "owns" that document starts up and opens the document. For example, if you double-click a Word document, Word starts up. This requires that your computer have a notion of "types of document," and that it draw an ownership association between a particular document type and a particular application. Before Mac OS X, this association was performed by means of four-letter creator codes hidden in the meta-information for a file (the "desktop database"). But on Mac OS X, it’s done in a new way, by a part of the system called Launch Services. Apple documents the entire situation on this page of the developer documentation about Launch Services.
Here’s what to notice about what that Web page is saying. In addition to the old system of four-letter codes, Apple, in an attempt to improve cross-platform compatibility, added to the mix the notion of the file extension, a several-character code at the end of every filename that provides the clue as to what sort of document the file is. At the same time, Apple also introduced yet a third way of signifying a "document" type, namely by means of a URL scheme. An application can specify that it responds to certain schemes; it is then eligible to be messaged by the system when such a scheme is encountered.
I don’t quite know why Apple did this. Part of the reason may be that Apple seems to have a great deal of trouble making up its mind how to specify a file in general. In Cocoa, for example, a file may be specified either as a pathname or as a URL; on the whole, Mac OS X seems to be trying to break down the distinction between a file and a URL-in-general. This works nicely from one point of view, because it means that for the programmer the command to open a remote file via http is exactly the same as the command to open a text file on the same machine. In any case, though, the thing to understand is that under Mac OS X, a URL becomes a file specifier possessing the same status as a pathname.
There are various ways in which the system can become aware of an application, and of the fact that an application can respond to a scheme. As noted in the first paragraph of that document, this does not require that you launch the application; merely copying the file to your hard disk is sufficient. This seems like a sensible design decision, because after all, when you first receive a new application (Microsoft Word, for instance), you would not want there to be an arcane rule that says you must first open Word itself for the operating system to know what to do with the various .doc files included with the installation. You would prefer the user to be able to double-click a .doc file in the Finder and have Word open the file, even though Word has never run before on this machine.
So far, so good. But now, keeping all of that in mind, consider what happens when you’re browsing the Web in Safari and you click on a link. For example, when you click a mailto URL in a Web page, something needs to happen. Therefore, something needs to associate the mailto URL scheme with your default email client. Way back in the bad old days, Apple had no solution to this problem. But eventually a third-party solution called Internet Config appeared, and Apple later adopted it and built it into the Mac OS.
But a moment ago, you may recall, I said that any application can now simply declare that it "owns" a certain URL scheme, just as it can claim to "own" a certain document type. That point is reinforced by this page from Apple’s developer documentation.
In other words, Apple has folded together in Mac OS X’s Launch Services two very different mechanisms from the classic Mac OS: the Desktop Manager (which pairs documents with applications) and Internet Config (which pairs URL schemes with applications). All of that sounds reasonable enough, since in both cases we are using something (a document or a URL scheme) to launch an application, but the two notions are arguably not parallel. One does not expect the set of URL schemes to be infinitely extensible. It’s one thing to click on an email address that’s a mailto link and have Mail open; it’s another to click on a link and have Help Viewer open, or to have Script Editor open, or to have a hidden application you’ve never heard of download and mount a disk image.
The upshot, if I’m an evildoer, is that if I can get you to download my evil application and make Mac OS X aware of it, then I may be able to get you, through a mere Web page, to send that application a message that causes it to launch and do its evil deed. My malicious application declares to the system that it responds to the "evil://" URL scheme, so if I can get you to click on an "evil://" link, my application will launch. Furthermore, as Adam explains in his article elsewhere in this issue, the real trickery comes about if I can manage to deliver a one-two punch without your realizing what has happened. Using a page containing a redirect or even frames, I can effectively make your browser visit two URLs one after the other: the first uses a technique like the disk URL scheme to download the application to your hard disk and to make the operating system aware of it, the second uses my "evil://" protocol to cause Mac OS X to launch it.
If the overall thrust of the above discussion is right, then the problem we’re facing is rather deep-seated. Part of the trouble, of course, is that URL schemes (such as disk) can cause an application to appear on your machine before you know what is happening; but another part of the trouble is that a mere URL in a Web browser can be sufficient to launch that application. Therefore it is impossible to defend against this mode of attack merely by defeating certain individual schemes, because the number of possible schemes is infinite. That’s why Unsanity’s Paranoid Android is an effective patch: it warns the user about all unknown schemes that might have been registered by a malicious application, rather than focusing on any known schemes. But if I’m right in suggesting that the root of the trouble is the folding of Internet Config’s functionality into Launch Services, then plugging this hole completely might require Apple to adjust the architecture of Mac OS X at a level so fundamental that doing so could break some capability to which we’ve become accustomed.