An improved firewall was one of the 300-plus features Apple touted before the release of Leopard, but a mix of design choices and functionality changes reduces its effectiveness compared to the firewall in Tiger, something I had heard only rumblings about when I wrote "How Leopard Will Improve Your Security," 2007-10-22. While it's not concerning enough that you shouldn't upgrade, it is something Apple will need to address fairly quickly with an update.
What's a Firewall? For you non-security-geeks out there, a firewall is a tool that blocks traffic to a system or network based on rules (for a more-detailed description, see Chris Pepper's "What's a Firewall, and Why Should You Care?," 1999-02-22). Firewalls have existed since the late 1980s and were developed in response to the first Internet worms, particularly the Morris Worm, as a way of protecting systems and networks by blocking any unwanted traffic. Before firewalls, if you placed a computer on a network (including the Internet), anyone else on that network could remotely probe your system for open connections and send you traffic directly. Since all computers tend to have some vulnerabilities, and some of those vulnerabilities are remotely exploitable over a network, this gives attackers an easy way to play on your network and potentially exploit your systems. Some of these attacks are self propagating - where malicious code takes over a system and then uses that system to take over other systems. This is what distinguishes a worm from a virus - a virus needs user interaction, while a worm "worms" its way through the network from system to system. Some of you might remember the Code Red worm from 2001 that took down major portions of the commercial Internet by hopping from computer to computer.
In the information security field we use many different kinds of firewalls. The most basic is a network firewall, typically a stateful packet inspection firewall, installed in a router. That's a fancy way of saying we use network firewalls that are a little smarter and can track inbound and outbound connections. The way Internet protocols work is that when you make a connection to a remote computer, you do it over a port. These ports are standardized, such as FTP on port 21, HTTP (the Web) on port 80, and SSH on port 22. The remote system needs to communicate back to you, so when you set up the initial connection your computer gives the remote computer an arbitrarily high port number for the return traffic. Otherwise, you would be limited to talking to only one Web site or FTP server at a time. A stateful packet inspection firewall keeps track of all these connections so it can allow traffic back to your system only if you have an open session, on those seemingly random ports that would normally be blocked.
Another kind of firewall, the one on our Macs, is a host-based firewall. Since our computers aren't always behind big network firewalls, it makes sense to build a firewall into our computers to protect us from attack as we wander between different networks, something that's increasingly common thanks to laptops. If you connect a laptop to any public network, such as at a wireless hotspot or a hotel, some person or automated program will almost certainly be scanning you.
The Tiger Firewall -- In Mac OS X 10.4 Tiger, Apple used a good open-source firewall called ipfw. ipfw is software that sits deep inside Mac OS X and filters network traffic before it makes it to the rest of the operating system, providing the same protection on the road as we have at home. When you opened the Firewall view of the Sharing preference pane in Tiger, that was just a graphical front end to ipfw. Tiger didn't let you adjust the really granular settings without writing your own configuration files, but the available controls were reasonably effective. When you enabled the firewall you could select which network services you wanted to let run. For example, if you had enabled file sharing, the Firewall view would show that file sharing was enabled and that you had to disable it in the Services view. The firewall functioned in a "deny all" mode that blocked everything except ports you specifically enabled, and it offered some advanced options to block all UDP traffic and ignore requests to filtered ports (what's called "stealth mode").
This approach wasn't perfect, but was good enough for the average user. It lacked any outbound filtering - a nice feature that lets you lock down your system to ensure that unapproved services on your Mac can't connect to the outside world, and a good technique to help limit attackers or talkative applications. It also lacked application control, a useful feature common in most host firewalls that lets only approved applications talk to the outside world, no matter what port they use.
Firewalls in Leopard -- Leopard still includes ipfw, but it's no longer the default firewall. Instead, Apple has replaced it with a black box - a firewall program that is unknown to security researchers - that behaves a little oddly. From what we can tell, Apple developed the new firewall themselves to add application control. The firewall now lives in the Security pane of System Preferences and now has three options for the firewall: Allow All Incoming Connections, Block All Incoming Connections, and Set Access for Specific Services and Applications. Apple made the decision to move the firewall in an entirely new direction, which isn't necessarily bad, but makes it more difficult to understand what's being filtered, and seems to leave some potential holes open.
The first problem with the Leopard firewall is that it's difficult to tell what the Set Access option does. It starts the new application-level firewall and lists in the Sharing pane any services you've opened, but it doesn't indicate if they are allowed or blocked. There's also no option for you to add your own open services or ports anymore. Instead, you can add or remove individual applications, but not network services. Stealth mode is still available in the Advanced settings, but the UDP blocking, useful to stop port scanning and some other attacks, is gone.
Worse yet, when you install Leopard, the firewall is turned off, even if you're upgrading and the firewall was previously enabled. Say what you want about Windows, but the firewall is enabled by default. Finally, the firewall can actually break your applications, which I'll explain more about shortly.
Further investigation revealed some really strange (for a firewall) behavior. Some applications ask for permission to access the network the first time you use them, like Safari, Firefox, and Cyberduck, while others are ignored, like Colloquy and Twitterrific. If you have a service enabled in the Sharing pane, but select Block All, it still appears open to the outside world when you scan the ports, but you can't connect to it. Some services seem to be open all the time, no matter what you do. If you ever connect to another computer for file sharing, TCP port 88 (for Kerberos authentication) is opened and stays that way until you reboot, no matter what you set on the firewall, even if you enable stealth mode. Bonjour (mDNS) is hidden in stealth mode but available even if you select Block All. Finally, the firewall is a black box - the only way I could learn what was opened or closed was to scan it from the outside using networking tools (such as Nmap, the same tool Trinity used in "The Matrix Reloaded"). Unlike in previous versions of Mac OS X, you can't check settings by looking in a configuration file.
There's one behavior that caught me completely by surprise and calls for an immediate fix. If you have the firewall set to control applications, those applications that don't already have their code signed are signed by Leopard when they access the network. (Code signing is the process of affixing a digital signature to an application, such that the operating system can tell if the application has been modified by malware, because the application's checksum would no longer match the checksum in the signature.) If the application changes itself while running, as Skype does (and as some other applications do too), it won't match the signature the next time you go to run it and your application won't launch. There are no warnings or errors, and the average user might assume something is seriously wrong with their system. I experienced this myself when I was recording a podcast with Glenn Fleishman: Skype failed to launch; I reinstalled, and it launched. The next time I tried to launch it, Skype failed again, and a reinstall fixed it. I looked in my console and saw a weird error. A quick Google search provided the answer.
All of these behaviors are considered "bad" on the whole firewall good/bad scale. Leopard breaks a number of conventions. First, if you select Block All, no network services should be enabled, even if you've turned them on somewhere else. Apple either needs to relabel that setting to "Block All Except...", or change the behavior to block all traffic, especially Bonjour. Application control behavior also needs to be more consistent - having some active applications appear in the settings, but not others, is confusing and could lead to wrong assumptions. I may think I'm only allowing a few applications, when, in reality, all sorts of applications are accepting network connections without my permission. More seriously, Kerberos shouldn't linger on an open port just because you connected to another computer. Having a firewall arbitrarily break approved applications is also unacceptable. Finally, firewall rules need to be user-accessible to allow customized configurations or just to allow the more-advanced users to understand expected behavior.
I've listed some of the more technical details I've discovered on the firewall on my blog at Securosis.com
These are all problems Apple is perfectly capable of fixing and I'll be surprised if they don't address them sooner rather than later. Until then, I still recommend you activate the firewall in Block All Incoming Connections mode so you don't break applications. If you need to enable file sharing or other remote access, you'll need to either select the Set Access method, or turn your firewall off. One last option is to use ipfw and manually configure firewall rules, or use a GUI tool like the free WaterRoof, and skip the Leopard firewall completely. In WaterRoof, just click Rules Sets to pick your rules, and then go to Tools > Startup Script and install a startup script to run those rules when you reboot.
The good news is that I don't know of any active remote exploits for the Mac, and if you have to take the risk you should be OK for now even without your firewall running, especially if you avoid AFP for file sharing and use SMB instead (selectable with the options button in the Sharing preference pane). This isn't ideal, but it does give Apple a little time to fix up the firewall so it protects users without breaking applications.