I recently received an email message from Cornell Alumni Affairs, warning of deliverability problems related to alumni email forwarding and participation in Cornell mailing lists. Since I’ve recently spent quite a bit of time learning about SPF, DKIM, DMARC, and other email deliverability technologies for TidBITS and Take Control, this piqued my interest. Some additional investigation showed that the problem extends far beyond Cornell, and is currently affecting any organization that relies on email forwarding, plus any mailing list with Yahoo or AOL subscribers.
Unfortunately, apart from users switching away from Yahoo and AOL, there’s not much that can be done about it.
First, to explain all this, we need to untangle the alphabet soup of technologies I blithely splashed into the above paragraph, all of which are designed to reduce the ability of spammers to spoof email to make it look like it comes from a legitimate sender.
SPF stands for Sender Policy Framework and enables the owner of an Internet domain, like tidbits.com, to specify which computers are allowed to send email containing sender addresses in that domain. A mail server receiving email from a sender claiming to be firstname.lastname@example.org does a DNS lookup on tidbits.com, checking a special TXT record to see if the sender’s mail server is allowed to send email on behalf of tidbits.com. You can think of SPF working at the envelope level, and it’s an Internet Engineering Task Force standard (RFC 7208).
DKIM stands for DomainKeys Identified Mail, and is also an IETF standard (RFC 6376). Whereas SPF looks at the envelope to verify that a message is being sent from an approved source, DKIM goes one step deeper, associating a domain name with an actual email message, via public key cryptography. First, the sending mail server uses a private key to sign the contents of each outgoing message using a DKIM-Signature header. Second, the receiving mail server does a DNS lookup on the domain name specified in the DKIM-Signature header to find another special TXT record that contains the associated public key; the receiving mail server can then use the public key to verify that the message hasn’t changed since it was signed.
DMARC, or Domain-based Message Authentication, Reporting, and Conformance, is an email authentication method that builds on SPF and DKIM. Again implemented in DNS, a DMARC policy lets a sending organization specify that its messages use SPF and/or DKIM, and what should happen to messages for which the domain in the header’s From line fails to match (is not in “alignment” with) the domains specified by SPF and DKIM. There are three options: do nothing (“none”), flag the message as suspicious (“quarantine”), or bounce the message (“reject”). Although DMARC is used by numerous major email providers in various ways, it is not an IETF standard.
A key reason DMARC exists is that the ways SPF and DKIM enable the sender to specify what should be done with failed messages are generally unused. In SPF, each receiving mail server has to determine what to do with results such as “none,” “neutral,” “pass,” “fail,” and “softfail.” Although “fail” was intended to equate to rejection, few senders set that, due to the high number of false positives. The DKIM specification is even more explicit about how its results should (not) be used:
In general, modules that consume DKIM verification output SHOULD NOT determine message acceptability based solely on a lack of any signature or on an unverifiable signature; such rejection would cause severe interoperability problems.
(There is a little-used extension to DKIM called ADSP (Author Domain Signing Practices, RFC 5617), which was the predecessor to DMARC and offers advice from the sender when the From header doesn’t match the domain in the DKIM signature. It was quickly identified as having the same problems that now afflict DMARC; hence its minimal usage.)
So DMARC enables a sender to tell receiving mail servers what to do with messages whose SPF- and DKIM-advertised domains don’t match the domain in the From line. That’s ideal for companies like PayPal, which send vast quantities of transactional email, are constantly spoofed as part of phishing attacks, and don’t have (many) individual users. PayPal has published a DMARC policy of “reject” for over a year with no one noticing. However, until recently, large email providers have stuck with a DMARC policy of “none,” which lets messages be delivered normally. You can check the DMARC policy for any domain at the DMARC Inspector — look at the “p=” tag.
In April 2014, Yahoo dropped a bomb on the email world by quietly changing its DMARC policy to “reject,” and AOL followed suit shortly after, though at least with a press release acknowledging the change. This had the effect of causing receiving mail servers to bounce messages that failed SPF and DKIM, generating two classes of problems:
Email Forwarding: Let’s say you have a cornell.edu alumni address, which forwards to a Gmail address. Someone from Yahoo or AOL sends you a message, which uses both SPF and DKIM. Receiving it at cornell.edu works fine, because SPF, DKIM, and DMARC alignment all pass at that point, but when Cornell’s mail server forwards the message, it rewrites headers such that, when Gmail examines the new incoming message, it fails SPF, DKIM, and DMARC alignment. Because Yahoo and AOL both have DMARC policies saying that failed messages should be rejected, Gmail bounces the message.
Mailing Lists: Now let’s say you use Yahoo or AOL and are a member of a discussion-based mailing list. When you post to the list, the list server receives the message and packages it for distribution to the list, changing a variety of headers and possibly adding a signature to the body in the process. Those changes cause the repackaged message to fail SPF, DKIM, and, most important, DMARC alignment, so it won’t be received by any list recipients at Yahoo, AOL, Gmail, Outlook.com, or other any other email provider that honors DMARC policies, again because of how Yahoo and AOL set their DMARC policies to require that failed messages be rejected.
There’s even more fallout on mailing lists. Depending on how things are configured, if Gmail bounces a received mailing list message sent by a Yahoo user, that bounce will likely go back to the mailing list server and be recorded against the Gmail user, potentially causing that user to be removed from the list. So Yahoo and AOL users can get other list members bounced, purely by posting.
Needless to say, as email and mailing list administrators have figured out what is going on, this change has caused significant consternation. The IETF discussion list has been dominated by DMARC-related discussions for months.
Why did Yahoo and AOL make such a sweeping change? Speculation is that both had suffered significant security breaches that allowed bad guys to steal user information, including address books. They then used that information to create spam to these users’ contacts, forging the users’ return addresses. Since that spam was being sent by botnets, nothing short of this DMARC change could stop it. In short, Yahoo and AOL are cleaning up their mistakes by damaging every mail forwarding service and mailing list on the Internet.
What can be done about this situation? Although email boffin John Levine has compiled a list of things mailing list developers and administrators can do, none are trivial and all have side effects. At the user level, there’s simply nothing that can be done, other than those who use Yahoo and AOL voting with their feet and switching to less-draconian free email providers like iCloud, Gmail, and Outlook.com. Or, if you want to pay an email provider to ensure that they’re working for you, TidBITS staffers have used FastMail and easyDNS’s easyMail service (which comes with domain hosting) without experiencing these problems.
As an aside, although iCloud doesn’t appear to be involved with these DMARC-related problems, beware of iCloud’s spam filtering. In the most recent case, Doug Adams, who has contributed hugely to the Apple community through his Doug’s AppleScripts for iTunes site, found that for three weeks his messages weren’t being received by iCloud users because Apple was silently deleting all messages that contained the string “dougscripts”. Since his domain is dougscripts.com, that pretty much put the kibosh on all his iCloud-related email. After complaining to AppleCare and having the problem escalated to a Mac Advisor, and then to Engineering, and then to the Spam Team, the problem finally disappeared, without further explanation. It’s astonishing that, in 2014, Apple is still doing simple string-based filtering and silent deletion of offending messages.
What I’d like you to take away from this article, apart from just understanding the Internet a bit better, is that there’s a lot of effort that goes on behind the scenes to make sure even the most basic of Internet technologies work in the face of competing forces. Email may seem simple, but it’s also far more subtle than most people realize.