Apple did more than merely patch a few (okay, 11) vulnerabilities with the recent release of. According to a  this update also included a series of improvements, for both Mac OS X and Windows Vista, designed to improve QuickTime's fundamental security by making vulnerabilities harder for attackers to exploit. To understand why these are so significant we need to take a moment to review a little bit about how bad guys attack computers, and why QuickTime is particularly difficult to secure.
As I discussed in my preview of Leopard security (see, 2007-10-22), a group of software bugs called "buffer overflows" are the favorite target of attackers. Buffer overflows are a vulnerability where an attack enters more data into an input than expected; if the programmer who wrote the software forgot to limit that input field, the data can flow past the expected limit and overwrite other parts of memory. Since memory on most of our computers is just a big stack of commands mixed with data, if you know exactly how much extra data to put in, you can trick the computer into running an arbitrary command by overwriting a spot where it expects a legitimate instruction with your malicious instruction. Done correctly, it gives an attacker complete control over your computer.
QuickTime, like many media players, is plagued by these vulnerabilities due to how it's built, and the unique demands of software that's expected to deal with real-time audio and video files in dozens of formats.
When you install QuickTime it includes extensions programmed in Java, which is a high-level language. A low-level language, like C, requires programmers to manipulate memory and the CPU almost directly. C is the foundation for nearly all of our software, even other programming languages, but is notoriously difficult to program and debug. It's like building a home with only hand tools, where more often than not you need to carve your own two-by-fours. High-level languages like Java make life easier for programmers by removing many tedious tasks, like managing memory. Java is also interesting because it's designed to let programmers write software once, then run it on any operating system that supports Java. In reality it's seldom that simple, but for the most part if you write a Java program for Mac OS X, it doesn't take much to enable it to run on Windows or Linux.
Because programmers don't directly manipulate your computer's memory in Java, it's nearly immune to buffer overflow vulnerabilities. In theory, this makes those Java extensions extremely secure. The problem is that Java isn't very good at things like audio or video that require really high performance, and we find ourselves going back to C. In the case of QuickTime the main program and all the plug-ins that play those media files on your computer are written in C code, making them potential targets for buffer overflows. Even worse, the very complexity of handling large media files increases the odds of a buffer overflow, since it isn't like the programmer looking at a video of unknown size can just impose a 140-character limit on an input field.
The way QuickTime works is that the main code processes the audio and video, then sends it to a plug-in (often called a codec, for "compressor-decompressor") that understands that file format and turns all those zeros and ones into high definition video with surround sound. It's hard for QuickTime to validate all this data, and the processes of handing things off between parts of the program itself are extremely complex. On top of that, the Java extensions add an additional layer of complexity, are nearly always available to an attacker, and expose parts of the QuickTime program that aren't normally accessible through a Web browser. The result is that attackers take advantage of this extra exposure and these complex handoffs to all the other bits and plug-ins programmed in C.
It turns out these Java extensions are one of the top three sources of security vulnerabilities in QuickTime. The next major source is the processing of all those different types of media files with all those different codecs; attackers craft malicious audio or video files that crash the player and create buffer overflows. The last problem also relates to supporting all those media types; attackers exploit vulnerabilities in the part of QuickTime that's responsible for first figuring out just what kind of file it is and what codec to use. They modify the file headers and exploit QuickTime before the file's contents ever get sent to the codec. QuickTime is thus difficult to secure because it has to be all things to all media files, and is written using multiple programming languages. QuickTime's main body and Java extensions are never really sure of what kind of data we're sending along, and must shovel data down to the C parts for processing. These interactions increase the odds of a buffer overflow somewhere along the way, or even in the process of the different bits talking to each other. QuickTime also supports random third-party plug-ins for new media types, which it has no way of protecting.
One way to reduce the chances of a successful buffer overflow attack is to protect the way the software interacts with memory. That's why we call this "anti-exploitation" - even if the software is vulnerable to a buffer overflow, such technologies make it harder for the attacker to turn that into a successful exploit of your system.
One example of this is the Address Space Layout Randomization (ASLR) technique used in Windows Vista. For the attacker to do anything specific on your system after they overflow the buffer, they need to "point" to operating system commands in memory with the instruction they insert. Vista moves all these commands around randomly every time it runs, so the attacker never knows where to point. With QuickTime 7.4.5, Apple added ASLR support to QuickTime, also moving around many of the QuickTime commands every time it runs. This keeps the attacker from being able to point back to any QuickTime commands and using that to take over the computer. Apple didn't enable this for all of QuickTime's commands, so QuickTime is still a little vulnerable, but it's a major step forward.
Apple included their own version of ASLR in Mac OS X 10.5 Leopard, but it doesn't work quite as well as Vista's. Called Library Randomization, it doesn't rearrange all the core system commands, and in particular it leaves fixed in place something called a dynamic linker that an attacker can still use to exploit the Mac. We're hoping Apple will fix this in a future update.
Apple also enhanced QuickTime for Mac OS X with two other ways of preventing a successful buffer overflow by protecting memory: stack protection and data execution protection.
In particular, stack protection could completely eliminate the possibility of an exploitable buffer overflow to the stack part of memory. Even if the attacker overflows the stack (where most user input and programming commands are held), stack protection should detect this and stop any of the attacker's commands from working. There's still another entire category of memory, called the heap, that's vulnerable, but exploiting heap vulnerabilities is considered more difficult than attacking stack vulnerabilities.
Data execution protection takes advantage of special hardware settings on Intel CPUs (and thus only works on Intel-based Macs). It enables programmers to set memory locations that will hold only data, and never execute commands, thus providing another way to foil a buffer overflow attack.
One area that will always present a potential problem for QuickTime is third-party plug-ins for new media types. Since Apple doesn't write or distribute these, there are no guarantees the programmers of said plug-ins will take the same precautions as Apple's engineers. It's one reason you want to be careful about installing third-party plug-ins.
The addition of ASLR, stack protection, and data execution protection doesn't make QuickTime immune to attack - buffer overflows are just one kind of vulnerability and Apple hasn't blocked every avenue of attack. In fact, just before publishing this article, on QuickTime running in Windows Vista.
That said, the combination of all these changes is an excellent start with practical security benefits for any QuickTime user, which equates to nearly every Mac user (and an ever-increasing number of Windows users). Now that Apple has taken these steps with QuickTime, they need to work harder to extend similar technologies to Mac OS X in general.