Series: DNS Flaws Could Have Led to Disaster
Flaws in the way in which the domain name system (DNS) has been implemented for years by dozens of different companies and entities - including the organization that maintains the DNS software used by Mac OS X client and server software - could lead to an easy exploit by those who want to redirect users to malicious sites for identity theft and malware installation. Fortunately, the problem was caught and fixed, although Apple's response was much delayed and poorly communicated.
Article 1 of 3 in series
Apple has made its biggest security stumble ever by not releasing a necessary patch for a serious DNS exploit that allows any domain name to be redirected to any IP address. Show full article
On 08-Jul-08, a massive security patch was released by dozens of vendors for a major vulnerability in DNS (Domain Name Service), discovered by security researcher Dan Kaminsky. DNS is one of the fundamental underpinnings of the Internet, translating domain names (like tidbits.com) into IP addresses (like 184.108.40.206). Because DNS is so core to the functioning of the Internet, this vulnerability is perhaps the most significant security problem to face the Internet in the last decade.
All users who connect to Mac OS X-based servers for DNS lookups are at risk: Apple has not yet provided a patch, unlike dozens of other companies that make or distribute operating systems or DNS server software.
Apple was clearly distracted by the largest set of launches in its history: the iPhone 3G, the iPhone 2.0 software, the .Mac-to-MobileMe transition, and the App Store. Nonetheless, their customers are now in danger and Apple needs to respond immediately.
All companies that provide DNS service to their customers should have already updated their DNS servers. Many have not. You can determine whether your ISP is at risk by visiting Kaminsky's site and clicking the Check My DNS button. If the site says your DNS is at risk of being poisoned, contact your ISP or your company's IT department immediately.
Poisoning the DNS Well -- Kaminsky accidentally discovered a new technique attackers could use to compromise DNS servers, allowing ne'er-do-wells to convince servers to accept an incorrect IP address for a given domain name from a source other than the one that properly controls information for that particular domain. (This is called cache poisoning.) The attack doesn't affect the DNS server software - it doesn't compromise the software itself - but rather the attack changes the information the server stores, or caches, to provide answers about domain names that the server has retrieved from elsewhere.
Thus, when you type www.tidbits.com into a Web browser's URL field, rather than your computer receiving back the correct IP address from its built-in DNS resolver - 220.127.116.11, in this case - an attacker would indirectly convince that resolver to believe the address was something else, like 172.31.0.16.
Your browser would obligingly use that IP address to make a connection to a Web server while displaying www.tidbits.com in the address bar. That site could be - certainly would be - loaded with malware. This is a particular problem for Windows users, whose systems could be infected simply by visiting a site. With no active exploits for Mac OS X that currently result from visiting a Web page, Mac users are more likely to fall victim to social engineering after visiting a site and being told to re-enter a password or provide details that a trusted site doesn't normally ask for.
DNS is distributed and can be recursive, meaning a server keeps working through a set of linked responses it gets from other DNS servers until it gets an authoritative answer. Your computer has a "stub" resolver, which knows to ask a full-blown DNS server for the name-to-number conversion. The full-blown DNS server is typically run by your ISP or the company you work for. That DNS server, in turn, asks root nameservers - run by a variety of organizations - where to find details about, say, .com.
The root nameservers direct your ISP or company's DNS server to the server that has the lookups for that domain. This can go on and on for every dot-separated part in a domain name, but it typically follows this path: root server, top-level (such as .com) domain server, and corporate domain server.
Weakening SSL/TLS, But Not Killing It -- This attack does not directly disable secure Web connections, although it weakens the signals you rely on for trust, and requires that you be more alert. Secure Web connections use SSL/TLS (Secure Sockets Layer/Transport Layer Security), a mechanism that encrypts a connection and also relies on trust outside that connection to validate the connection. The digital certificates used as the basis of these connections require that a domain name match a particular IP address; but if your DNS has been poisoned, a bogus certificate becomes a much more serious risk.
However, the outside trust element should save you. Certificates must be signed by third parties, known as certificate authorities, like Thawte or Comodo. These authorities are supposed to verify the identity of a party requesting a certificate before the authority signs their request. Authorities charge fees from tens to thousands of dollars depending on how much background checking and control over the certificate is asked for. Details about these authorities are pre-installed in browsers and operating systems completing the circle of trust: Your browser knows an authority's signature, which enables your system to validate the authority's approval of a Web's site certificate.
If an attacker's fake site tries to present you with a certificate that alleges it's www.amazon.com, your browser will alert you that the certificate hasn't been signed or at least wasn't signed by a known certificate authority. That's always a reason to refuse a connection, unless you're connecting to a Web site run by a trusted party that's given you explicit information about the certificate they've chosen to use.
A Coordinated Fix, Except for Apple -- While cache poisoning has always been a problem for DNS, the technique Kaminsky discovered is faster and more effective than any previous known exploits. Kaminsky's flaw allows an attacker to overwrite existing DNS entries that a server has already cached - something never before possible. This vulnerability is a flaw with the protocol itself, and thus affects nearly every DNS implementation in use.
After determining this flaw was legitimate and widespread, Kaminsky immediately contacted major vendors - operating system makers and DNS software developers - and other DNS experts who met secretly at a meeting hosted on the Microsoft campus in March 2008. In an unprecedented move, the vendors all agreed on a simultaneous release of fixes for their products, coordinated with the help of the United States Computer Emergency Response Team (US-CERT).
To obfuscate the nature of the vulnerability, the companies all agreed to use a fix - port randomization - that didn't necessarily reveal the details of the flaw, thus slowing down the ability of bad guys to reverse engineer it and attack servers before organizations could patch. This lasted for 13 days, until the vulnerability was disclosed by a security researcher who accidentally published a draft blog post with all the details. By 24-Jul-08 exploit modules appeared in the popular Metasploit penetration testing tool, empowering any attacker capable of downloading the tool and using a web browser.
(The brief explanation of the flaw is that by forcing a DNS server to look up certain domains by sending it requests, an attacker can take advantage of a predictable sequence of port numbers to send a massive number of fake answers to the DNS server. If just one of the fake answers gets through, the attacker "wins"; it's essentially a race in which the bad guy can have a million marathon runners and the good guy thinks they're off for a solo jog in the park. This can be accomplished in a couple of minutes with Metasploit. Randomizing the sequence of ports used in requests vastly increases the complexity of a bad guy winning. The general vulnerability of predictably used ports was understood in 2001 and built into the DNS server djbdns. The real answer to this problem is DNSSEC, which combines public-key cryptography with DNS, allowing only the legitimate domain owner to provide answers to DNS queries about its domain. DNSSEC has been bogged down for years, but a logjam broke in March 2008, and we're likely to see real use due to this basic DNS flaw being revealed.)
Apple Punts, Doesn't Patch Yet -- Apple has yet to patch this vulnerability, which affects both the desktop version of Mac OS X and Mac OS X Server. While individual computers that look up DNS are vulnerable, servers are far more at risk due to the nature and scope of the attack.
Apple uses the popular Internet Systems Consortium BIND DNS server which was one of the first tools patched, but Apple has yet to include the fixed version in Mac OS X Server, despite being notified of vulnerability details early in the process and being informed of the coordinated patch release date.
All users of Mac OS X Server who use it for recursive DNS must immediately switch to an alternative or risk being compromised and traffic being redirected. Installing the above-mentioned BIND should be relatively trivial for anyone who can compile software at the command line. The Mac community could take this up if someone created a compiled version of BIND 9.0.5-P1 and distributed it for simpler installation.
With active exploit code available in a common attack tool, it is imperative that Apple fix this vulnerability. Due to their involvement in the process and the ability of other vendors to fix their products in a timely fashion, it's hard to imagine any possible justification for Apple's tardy behavior.
If you are unable to patch a server system with new code, you could reconfigure those servers to forward DNS requests to alternative platforms, such as BIND on Linux or Unix, or Microsoft servers, until Apple issues a patch. Ask your ISP or network provider for assistance.
Although the desktop version of Mac OS X is also technically vulnerable, current attacks are directed at servers, so there's no need to panic.
This is an extremely serious security issue and we hope Apple will act responsibly and address it immediately, despite their initial tardiness.
[Author's note from Rich Mogull: I assisted Dan Kaminsky with the initial communications regarding the vulnerability and the coordinated release. Check out the initial executive summary.]
Article 2 of 3 in series
The SANS Institute finds that Apple's patch for a flaw in the DNS protocol doesn't fix client resolver software, leaving Macs vulnerable to a far-less-likely outcome.Show full article
The SANS Institute installed and tested out Apple's fix for the underlying flaw in the domain name system (DNS) protocol, and found that a patched copy of Mac OS X 10.5 Leopard (the desktop version, not Leopard Server) still suffers from the risky technique that makes DNS vulnerable to exploitation.
This exploitation, so far, seems extremely unlikely, but we won't know how unlikely until security researcher Dan Kaminsky, the discoverer of this flaw, provides full disclosure on 06-Aug-08 in his Black Hat conference talk, "Black Ops 2008: Its (sic) the End of the Cache as We Know It."
As Rich Mogull and I noted in "Apple Fails to Patch Critical Exploited DNS Flaw" (2008-07-24), servers are at a high risk from this DNS vulnerability. The flaw allows an attacker to send tens of thousands of fake responses for a DNS query to a server, which then poisons the server's DNS entries if the attacker matches the right pattern with their forged information before the legitimate response arrives from the DNS server for the domain that's being queried.
However, computers used by individuals without DNS server software in operation are also vulnerable to this flaw in DNS; we just don't know yet quite how vulnerable. With servers rapidly being patched worldwide, it's likely that the low-hanging fruit has largely disappeared, and attacks would then turn to clients - if clients are readily exploitable, too. Clients use stub resolvers, which forward requests for DNS answers to a full-blown, or recursive, DNS server run by their company, ISP, network provider, or co-location facility.
These clients pass their requests along, and it seems unlikely that they could be attacked directly unless an attacker had a computer on the same local network segment as the exposed system. In that case, the attacker would have a panoply of other network information poison available, and could disrupt DNS in a more efficient manner.
The DNS flaw relies on predictability in how ports are assigned to outbound requests for domain name lookups in a DNS query. An attacker forces a DNS server to look up a domain using a DNS server the attacker controls, and from that obtains the current port number being used for requests. If the ports are sequential - each query increments by one the port number used for each subsequent request - then the attacker starts sending forged requests using ports numbered just above the one it sniffed.
This is part of the question about client vulnerability: it's very hard to force a client to look up an evil domain to prime the pump because clients don't answer DNS queries to begin with, and typically aren't running mail servers which can be gamed when an attacker sends incoming email with an evil domain in the return address.
By increasing entropy - choosing a random port for each request - a patched DNS server prevents attackers from producing enough packets quickly enough to win the race with the legitimate DNS server, such that they cannot - statistically speaking - poison the DNS cache. (This is a patch, not a fix, actually; DNS itself must be overhauled to remove the fundamental weakness.)
I checked out my updated Leopard desktop system, and, sure enough, I saw precisely what SANS reported: sequential UDP ports returned in response to outbound requests, regardless of what this entails.
If you'd like to duplicate the SANS experiment, follow these steps:
- Launch Applications > Utilities > Terminal.
- Type the following, entering your administrative password when prompted.
- Open another Terminal window, and in it, type the following, press Return, and then press up arrow and Return a few more times to enter the command repeatedly:
- In the window with tcpdump running, you should see a series of lines that look like the following.
- Press Control-C (not Command-C) to stop tcpdump from running.
sudo tcpdump | grep domain
15:06:53.900835 IP 192.168.1.16.49229 > yourDNSserver.com.domain: 5228+ PTR? 18.104.22.168.in-addr.arpa. (43)
15:06:53.947838 IP 192.168.1.16.49230 > yourDNSserver.com.domain: 48400+ PTR? 22.214.171.124.in-addr.arpa. (43)
15:06:55.003628 IP 192.168.1.16.49231 > yourDNSserver.com.domain: 15730+ PTR? 126.96.36.199.in-addr.arpa. (43)
(If you don't see any results in step 4, you need to specify the network adapter with the tcpdump command. You can try en1, en2, en3, and so forth as in the following command.)
sudo tcpdump -i en1 | grep domain
We're not back where we started, because clients are enormously harder to attack. But it's still a hole that needs to be filled. We just won't know how deep a hole until next week.
Article 3 of 3 in series
Install Security Update 2008-005 now! Apple has finally released a security fix for a serious DNS flaw that's being exploited in the wild. The update also includes fixes for other serious vulnerabilities.Show full article
Twenty-four days after the rest of the industry mobilized to patch a serious flaw in the domain name system (DNS) protocol that's core to the functioning of the Internet, Apple has at long last released Security Update 2008-005, which includes its fix for the regular and server flavors of Mac OS X 10.4 Tiger and 10.5 Leopard. If 24 days doesn't sound like a long time, note that Apple was notified privately on 05-May-08, nearly 3 months ago, and this is for a vulnerability with significant exposure that had the potential to be disastrous for Apple's business and hosting customers, as amply described in an opinion piece for Macworld by Mac system administrator John Welch.
This update also repairs the ARDAgent flaw first reported 18-Jun-08 that enables someone either with access to a computer as a regular user, or who could convince someone to download and run software containing a Trojan horse, to gain root privileges on the system.
(For details on the DNS flaw and Apple's delayed response, see "Apple Fails to Patch Critical Exploited DNS Flaw," 2008-07-24. For more about how the ARDAgent vulnerability could be exploited, see "How to Protect Yourself from the New Mac OS X Trojans," 2008-06-25.)
You can download Security Update 2008-005 via Software Update (the easiest approach), or as standalone downloads for all versions of Mac OS X 10.5 Leopard (65 MB), for the desktop versions of Mac OS X 10.4.11 Tiger for PowerPC (88 MB) and Intel (143 MB), and for Mac OS X 10.4.11 Tiger Server for PowerPC (135 MB) and Intel (180 MB). While the Leopard update doesn't explicitly state it works with Leopard Server, we checked Software Update on TidBITS's Xserve running 10.5.4 Leopard Server and were prompted to install the same-sized and -named update as on a MacBook that uses Leopard's 10.5.4 desktop release.
DNS Flaw Fixed -- Those of you operating DNS servers via any version of Tiger or Leopard should immediately back up your current systems, make sure they have a good point to revert to in the case of failure, and install this security update. The same goes (with fewer potential repercussions) for all other Tiger and Leopard users.
Although we haven't tested this update in a production situation where we're answering DNS queries from servers all over the Internet, the update seems to have worked just fine on all the systems we've updated, including Leopard Server and a regular Leopard installation. Apple's security updates have a generally good track record in performing as expected and not introducing new complications.
Tiger users will see Internet Security Consortium BIND (the DNS software Apple relies on) updated to 9.3.5-P1, and Leopard systems will move to 9.4.2-P1. The latest version of BIND software is 9.5.0-P1, but Apple hasn't incorporated this update into Leopard.
Owners of systems running Mac OS X 10.3 Panther or earlier releases are still vulnerable, whether the systems are acting as recursive DNS servers that handle lookups from queries on the same computer or others, or merely as clients. The flaw is likely to be exploited on servers, but clients are still vulnerable. Servers can, at least, turn off recursion and forward requests to patched DNS servers, dramatically reducing the current risk profile. We'll write more about this as we understand the scope of the concern for ordinary users of Panther and earlier systems. While there may not be many such people - The Omni Group's operating system statistics show 57 percent of their users on Tiger, 42 percent on Leopard, and a vanishingly small 0.3 percent using other versions of Mac OS X - the last thing the Mac community needs is a small group of older systems being used as a springboard for new types of malware.
ARDAgent and Other Flaws Fixed -- Security Update 2008-005 repairs a number of other serious-sounding flaws in Tiger and Leopard that don't appear to have been exploited yet. As noted earlier, the update closes a hole that allowed the Apple Remote Desktop (ARD) daemon software, even when not running, to be used as a conduit to run a script that would allow a local user or malicious software installed by a local user to gain root access to a system.
The fix for ARDAgent (and similar programs) involves a change in the Open Scripting Architecture that prevents programs with system-level privileges from loading scripting additions, thus stopping attackers from using such software as a wedge for gaining system control.
The update also fixes a Disk Utility error that happens when you use Repair Permissions in 10.4.11. The terminal-based text editor emacs would be granted root privileges after permissions were repaired. The fix restores the correct controls within Disk Utility, but Apple doesn't state whether you should re-run the repair operation. We imagine you should, if you have other local users on a system that's running 10.4.11.
Also noteworthy is that Security Update 2008-005 installs PHP version 5.2.6 to address security flaws in the 5.2.5 release that was previously available in Leopard. PHP is widely used to power Web sites. Other potentially concerning but less-known problems were also fixed.
Serious Reputation Hit -- As usual, we'll never quibble with Apple releasing a security update, particularly one that fixes such serious vulnerabilities. But put bluntly, Apple blew it on this one - this update should have been released on 08-Jul-08 when the rest of the industry released their patches. Yes, Apple was busy with the iPhone 3G, iPhone software 2.0, and App Store launches, along with the .Mac-to-MobileMe transition (which itself turned into a debacle). It doesn't matter - Apple had plenty of time and all they had to do was package up and perform normal stress testing of new versions of BIND. The BIND installation shows a creation date of 25-Jul-08, meaning that Apple didn't finalize its update for testing until just a week ago.
Trust takes time to acquire, but it can be lost quickly. Apple has made much of Mac OS X's security and, after a slightly rocky initial start with the earliest versions of Mac OS X, has been doing a generally good job of responding in a reasonably timely fashion to security threats. But to delay the release of the fix for such an important vulnerability was simply negligent, and it both infuriated Macintosh system administrators and damaged Apple's reputation in the enterprise market.