Hide full article
The Internet's spam volume has increased exponentially over the past four months. How? Spammers have found a new way to send their spam, in far greater volumes than previously thought possible. Unfortunately, and perhaps for the first time, Macs are a small part of the problem.
When it comes to worms, viruses, and other forms of network abuse, including spam, the Macintosh community frequently sees itself as an island of immunity in a Windows-dominated world of insecurity. Mac OS X has a pretty good track record so far, and the previous versions of the Classic Mac OS were seemingly near perfect with regard to network security, though many experts, including myself, would tell you that the Classic Mac OS's invulnerability was due more to pure luck than intentional design.
That luck has now run out. The Mac OS Internet server community, once thought to be immune from exploit, has indeed become part of the spam and network abuse problem. How could such a thing happen? The same way every other operating system used as an Internet server has been exploited by evildoers: a fateful combination of software shipped "open by default" and system administrators failing to take the time to understand and configure their servers properly in order to prevent abuse.
What's the specific culprit in this situation? It used to be that spammers relied primarily on open mail relays, which are mail servers that accept mail from anyone on the Internet without restriction and relay it on to the final destination. As system administrators and mail server developers have become alert to the idiocy of a mail server set to relay mail without requiring authentication of some sort, spammers have changed their tactics and started relying on a new tool: the open proxy server.
What Is a Proxy Server? A proxy server is a piece of software that facilitates Web surfing by users on an internal network, usually one that's protected from the outside Internet by a firewall. In essence, the proxy server sits between the Web and all the users on the internal network, sending out all the requests for Web pages from its users, receiving the pages back, and passing them along to the appropriate users. Institutions use proxy servers to increase performance (because the proxy server can store a copy of retrieved Web pages for other users on the internal network to access without going out to the Internet) and for content filtering purposes (since the proxy server can refuse to return requested Web pages that contain sufficiently naughty words; schools often used proxy servers as content filters).
You would think that proxy servers are handy for enforcing security, and in fact, they can be, if configured and deployed properly by a competent network administrator. Unfortunately, those conditions are rarely met. Well-meaning software vendors, such as (but not limited to) Microsoft in the Windows market, and StarNine (now owned by 4D Inc.) in the Macintosh market, shipped proxy servers as part of their "Web Server Suites" starting in the late 1990s. It was a logical move because customers were clamoring for these features, but in the interest of simplifying setup and making everything work out of the box, these suites were usually configured to install and start the proxy server by default, and worse, to allow access by anyone, not just users on the internal network. Those decisions, now easily seen as mistakes, are what brings us up to today. Now, open-by-default proxy servers exist all over the Internet. A portion of those are Macs.
How many of these Macintosh Internet servers exist on the Internet? Google, the all-seeing eye of the Internet, can give you a glimpse with the link below, which searches for the default page installed by 4D's WebSTAR 4. Most users delete or overwrite this file, so the list on Google should show only a small fraction of the actual number of WebSTAR 4 servers that may or may not have the included proxy server turned on by default. Don't forget to click the "repeat the search with the omitted results included" link!
<http://www.google.com/search?q=Server+Suite+4+ Test+Page>
How Are Open Proxies a Security Risk? The problem with open proxies is that anyone on the Internet can use them as go-betweens to perform just about any action related to Internet access. (To see how you'd configure proxy servers for a number of types of Internet traffic in Mac OS X, check out the Proxies tab in the Network preference pane.) The most frequent exploit of an open proxy is to bypass local content filtering - ironically, this exploit basically uses one proxy filter to bypass another. In the many open proxy logs I have examined, 95 percent of the hits fall into this category.
Spammers seem to have discovered open proxies sometime in the last year, probably as the number of mail servers allowing open relaying started to drop dramatically. Some of the recent Windows/Outlook virus outbreaks were really just Trojan horses with hidden open proxy code as the true payload. Noisy, high-profile worms like Blaster kept everyone, including the media, distracted while the other worms managed to create, within the space of about two weeks, hundreds of thousands (or perhaps more) of open proxy servers that could be trivially exploited later on. Next, the spammers had to find all the open proxies their worms had created, so scanning programs searched out the available proxies.
It was at this point that the Macs were found, since those scanning programs, while looking for their own captive open proxies, also ran across old Macs running WebSTAR 3 and WebSTAR 4 with latent, unused, and unknown proxy software. And since the Macs were equally as useful, the spammers cataloged and starting exploiting them to send spam.
Once a spammer has access to an open proxy, he can do any or all of the following with complete anonymity, while using somebody else's bandwidth:
Send mail from unsecured form-to-mail scripts on that, or any other server
Send mail via local SMTP servers since the source will be a trusted, local IP address
Craft mail with forged Received headers
Connect to thousands of throwaway "freemail" (Hotmail, Yahoo, etc.) accounts per minute and send untold millions of spam messages
Create traffic on pay-per-click systems
Create traffic to generate high page ranks/search engine results
Generate distributed denial of service (DDoS) attack traffic
Run brute force password cracks on Web sites or email servers
Run buffer-overrun cracks aimed at any URL-accessible service
In the last week, I've spoken with several people on the development team for the version of WebSTAR that first shipped with a proxy server, including the former product manager, and the developer who wrote the proxy server code. I asked them why they'd decided to bundle a proxy server into WebSTAR.
In answering, they gave the example of a school, where a teacher would ask a class to visit a URL, and everyone would download the same pages at the same time, resulting in slow performance. A local proxy server would access the remote Web site once and distribute the content to everyone locally, preventing the class from overwhelming the school's bandwidth, which back in those days was frequently limited to a 56 Kbps frame relay or 144 Kbps ISDN line, or even dedicated modem connections in many places. They also cited bandwidth-constrained places such as Australia or New Zealand as containing customers who needed proxy servers to reduce bandwidth consumption and costs. These are very real situations: when I was working in Europe in the mid-1990s it was common for ISPs to run proxies (often called caching servers back then) to save on cross-Atlantic bandwidth costs.
When talking to the WebSTAR folks, I noted that we never installed WebSTAR's proxy component on any of digital.forest's servers, but I was finding it on some of our client-owned co-located servers, so I asked how it could have been installed without somebody knowing it. The former product manager explained how a new install or an upgrade could have installed the proxy component by default. Also, under certain conditions that I have yet to determine, the proxy was open by default, leading us to where we are today, with old Macintosh Web servers being exploited by spammers.
My story of how these servers were being exploited was met by with a mixture of wonder and regret: wonder that anyone would dream of doing stuff like this, and regret for not anticipating it. I've shared their reaction, since I don't think many people, if anyone, could have seen this coming. Seven years ago, when these products were being developed, spam was mostly an annoyance on Usenet, not the email scourge it has become today.
How Did I Discover These Exploited Macs? Earlier this year, I started hearing my peers in the network operations community talking about open proxy abuse. Intrigued, I read some excellent papers presented at conferences by researchers investigating the issue.
<http://www.uoregon.edu/~joe/proxies/open-proxy- problem.pdf>
<http://spamlinks.port5.com/proxy.htm>
<http://www.westdam.com/spamlinks/proxy.htm>
So I've known about the problem for a few months, but I didn't realize how close to home it was. At digital.forest, we sell Internet colocation services, and we bill clients who exceed certain bandwidth thresholds as measured at the Ethernet switch layer (which records all the traffic to and from the computer, rather than looking at just one service, like HTTP). But since most clients who use lots of bandwidth are running high-volume Web servers, they usually compare their HTTP access logs to their usage bills. Last month, one of digital.forest's clients noticed a large enough difference between our network usage bill and the amount of bandwidth usage reported in his Web server logs to request an audit. I expected the additional protocols of FTP and SMTP mail to explain the discrepancy, but instead I discovered that their WebSTAR server's proxy was the source of the extra bandwidth usage. My curiously piqued, I started to investigate further, and a post on a network abuse newsgroup alerted me to a few more open proxies in our network (though none running on the TidBITS servers, I'm happy to say).
<http://groups.google.com/groups? selm=59c3aad4.0310192058.6683a403%40posting.google.com>
<http://chuck.forest.net/images/tidbits/ port8000.txt>
In searching this published list, I noted over 100 that included WebSTAR's default proxy port of 8000, and a few with obvious Mac-related DNS names, so I began contacting their webmasters to let them know about their vulnerability. I've talked with quite a few webmasters, but there's no way I can track down and call all the people whose Macs are on this list. Worse, this list contains only a small fraction of the potential open proxies on Macs out there, and worse yet, because these Macs were so easy to set up and have been so reliable, many of the people who did the initial work have long since moved on, leaving others with less technical experience in their place.
Are You Part of the Problem? Luckily, it's easy to tell if you're running an open proxy in WebSTAR, unlike the worm-created Windows open proxies, which are invisible and which don't log their activities. In WebSTAR 3 or 4, check to see if the WebSTAR Proxy Plug-in is installed in the WebSTAR folder, inside the Plug-ins folder. Also be sure to check any folders that may be inside the Plug-ins folder. To disable the WebSTAR Proxy Plug-in, just remove it from the WebSTAR folder hierarchy and restart WebSTAR. Before you do that, however, switch to the WebSTAR application and choose WebSTAR Proxy Log from the Plug-ins menu (the screenshot linked below shows what it looks like).
<http://chuck.forest.net/images/tidbits/ ProxyLogMenu.gif>
WebSTAR then opens a window showing proxy server activity, which you can use to check what's currently happening (see the screenshot linked below). The top of the window shows current active connections, the total number of connections, a total number of bytes sent, what the cache efficiency percentage is (this last one is useless information when the proxy is being exploited), and the maximum connection limit. The window's bottom portion lists a scrolling log of current activity. In the example screenshot linked below, I've altered IP numbers, domains, and URLs, but you can see what's going on. There are two logins to two different Yahoo Mail accounts, one search engine hit, and three hits on adult Web sites, all in under two seconds:
<http://chuck.forest.net/images/tidbits/ proxylogwindow.gif>
If you don't want to disable your proxy server because it's serving a useful purpose for your organization, you can secure it to prevent spammers and others from using it. The WebSTAR Admin application provides a graphical interface for restricting both the "to" and "from" sides of the proxy to fit your needs. Consult the WebSTAR manual for details.
Please note too, that you risk being rejected, blocked, or blacklisted if your network is a source for spam. As system administrators on the Internet starts getting tough with proxies, as they did with open relays, your risk of hurting your legitimate traffic by being blacklisted will only increase.
If you think this issue is only a concern when spammers start misusing your network, you should also consider the penalty of not taking action quickly. You could find your network addresses added to blackhole lists, which are compiled by a number of well-meaning individuals around the world who constantly scan and test for open proxies, even before they're exploited. These blackhole lists, in turn, are used by Internet service providers, academic institutions, and companies to block email, sometimes with undesired effects. TidBITS Contributing Editor Glenn Fleishman's mail server was once blacklisted because of the problem I note in this article, and it took him weeks to have his mail server removed from all the blackhole lists. He even had to appeal to the chairman of the board of one large ISP after their published procedures left him still blacklisted. So an open proxy isn't just a problem for you or your bandwidth bill: it can be a messy cleanup that restricts the ability of everyone on your network to send email.
What about Mac OS X? WebSTAR V, which is the current Mac OS X-compatible version of WebSTAR developed and sold by 4D, does not include a proxy server, so it's not vulnerable to open proxy exploits. Nonetheless, the folks at 4D are doing the right thing and have already started alerting customers to this vulnerability in the older versions of WebSTAR.
<http://www.4d.com/products/webstar.html>
Apple's Mac OS X Server has never had a proxy server included by default either. I spoke with the product manager, and he said adding one has been considered, but I suspect after our conversation that Apple will think twice before doing so, or take careful steps to secure it prior to shipping.
How do we resolve this situation? What remains now is hard work, and this article is just the beginning. I've spent every waking hour over the last two weeks investigating this problem on our network, reporting the problem to the abuse departments of the largest ISPs, and contacting many webmasters who are running open proxies without realizing. My work is having an effect already. Some of the data I shared with AOL helped them complete their investigation of a "known criminal spammer," and Yahoo is shutting down thousands of email accounts based on the information I shared with them from exploited open proxy logs.
But I can't do this alone. We must all work to spread the word, farther than even TidBITS can reach, to other Macintosh news sites, and to individuals who may be running open proxies. My hope is that open proxies on all platforms can eventually be shut down everywhere, and that the Macintosh community can lead the way. Fortunately, and true to form, performing these tasks on a Mac is far easier than on other platforms.
If you are a webmaster or system administrator, take a look at your servers and secure them if necessary. If you are a network administrator I strongly suggest you read Joe St. Sauver's "Open Proxy Problem" PDF (linked previously) for a complete, well-written analysis of the issue. Then make use of the suggested tools to search out open proxies on your network. If you find one that has been keeping a log (WebSTAR's proxy server does by default), you can greatly assist other network operators and abuse desks in shutting down their open proxies, and even more importantly, track and shut down spammers and other network abusers.
I'm sorry to be the bearer of the bad news that spammers could be exploiting our older Macs, but now that we're aware of the problem, working to resolve it will also provide the satisfaction of stemming the flood of spam.
PayBITS: Chuck deserves a medal for identifying this problem,
so let's all reward him with a few bucks via PayBITS!
<http://www.paypal.com/xclick/ business=goolsbee%40forest.net>
Read more about PayBITS: <http://www.tidbits.com/paybits/>
