This article originally appeared in TidBITS on 2014-06-17 at 3:41 p.m.
The permanent URL for this article is: http://tidbits.com/article/14843
Include images: Off

Yahoo and AOL Damage Mailing Lists and Email Forwarding

by Adam C. Engst

I recently received an email message from Cornell Alumni Affairs, warning of deliverability problems [1] related to alumni email forwarding and participation in Cornell mailing lists. Since I’ve recently spent quite a bit of time learning about SPF, DKIM, DMARC, and other email deliverability technologies for TidBITS and Take Control, this piqued my interest. Some additional investigation showed that the problem extends far beyond Cornell, and is currently affecting any organization that relies on email forwarding, plus any mailing list with Yahoo or AOL subscribers.

[image link] [2]

Unfortunately, apart from users switching away from Yahoo and AOL, there’s not much that can be done about it.

First, to explain all this, we need to untangle the alphabet soup of technologies I blithely splashed into the above paragraph, all of which are designed to reduce the ability of spammers to spoof email to make it look like it comes from a legitimate sender.

A key reason DMARC exists is that the ways SPF and DKIM enable the sender to specify what should be done with failed messages are generally unused. In SPF, each receiving mail server has to determine what to do with results such as “none,” “neutral,” “pass,” “fail,” and “softfail.” Although “fail” was intended to equate to rejection, few senders set that, due to the high number of false positives. The DKIM specification is even more explicit about how its results should (not) be used:

In general, modules that consume DKIM verification output SHOULD NOT determine message acceptability based solely on a lack of any signature or on an unverifiable signature; such rejection would cause severe interoperability problems.

(There is a little-used extension to DKIM called ADSP [9] (Author Domain Signing Practices, RFC 5617 [10]), which was the predecessor to DMARC and offers advice from the sender when the From header doesn’t match the domain in the DKIM signature. It was quickly identified as having the same problems that now afflict DMARC; hence its minimal usage.)

So DMARC enables a sender to tell receiving mail servers what to do with messages whose SPF- and DKIM-advertised domains don’t match the domain in the From line. That’s ideal for companies like PayPal, which send vast quantities of transactional email, are constantly spoofed as part of phishing attacks, and don’t have (many) individual users. PayPal has published a DMARC policy of “reject” for over a year with no one noticing. However, until recently, large email providers have stuck with a DMARC policy of “none,” which lets messages be delivered normally. You can check the DMARC policy for any domain at the DMARC Inspector [11] — look at the “p=” tag.

In April 2014, Yahoo dropped a bomb on the email world [12] by quietly changing its DMARC policy to “reject,” and AOL followed suit [13] shortly after, though at least with a press release acknowledging the change [14]. This had the effect of causing receiving mail servers to bounce messages that failed SPF and DKIM, generating two classes of problems:

Needless to say, as email and mailing list administrators have figured out what is going on, this change has caused significant consternation. The IETF discussion list [15] has been dominated by DMARC-related discussions for months.

Why did Yahoo and AOL make such a sweeping change? Speculation is that both had suffered significant security breaches that allowed bad guys to steal user information, including address books. They then used that information to create spam to these users’ contacts, forging the users’ return addresses. Since that spam was being sent by botnets, nothing short of this DMARC change could stop it. In short, Yahoo and AOL are cleaning up their mistakes by damaging every mail forwarding service and mailing list on the Internet.

What can be done about this situation? Although email boffin John Levine has compiled a list of things mailing list developers and administrators can do [16], none are trivial and all have side effects. At the user level, there’s simply nothing that can be done, other than those who use Yahoo and AOL voting with their feet and switching to less-draconian free email providers like iCloud, Gmail, and Outlook.com. Or, if you want to pay an email provider to ensure that they’re working for you, TidBITS staffers have used FastMail [17] and easyDNS’s easyMail service (which comes with domain hosting) without experiencing these problems.

As an aside, although iCloud doesn’t appear to be involved with these DMARC-related problems, beware of iCloud’s spam filtering [18]. In the most recent case, Doug Adams, who has contributed hugely to the Apple community through his Doug’s AppleScripts for iTunes [19] site, found that for three weeks his messages weren’t being received by iCloud users because Apple was silently deleting all messages that contained the string “dougscripts” [20]. Since his domain is dougscripts.com, that pretty much put the kibosh on all his iCloud-related email. After complaining to AppleCare and having the problem escalated to a Mac Advisor, and then to Engineering, and then to the Spam Team, the problem finally disappeared, without further explanation. It’s astonishing that, in 2014, Apple is still doing simple string-based filtering and silent deletion of offending messages.

What I’d like you to take away from this article, apart from just understanding the Internet a bit better, is that there’s a lot of effort that goes on behind the scenes to make sure even the most basic of Internet technologies work in the face of competing forces. Email may seem simple, but it’s also far more subtle than most people realize.

[1]: http://www.alumni.cornell.edu/services/dmarcissue.cfm
[2]: http://tidbits.com/resources/2014-06/Cornell-DMARC-email.png
[3]: http://en.wikipedia.org/wiki/Sender_Policy_Framework
[4]: http://tools.ietf.org/html/rfc7208
[5]: http://en.wikipedia.org/wiki/DomainKeys_Identified_Mail
[6]: http://tools.ietf.org/html/rfc6376
[7]: http://en.wikipedia.org/wiki/DMARC
[8]: http://dmarc.org/
[9]: http://en.wikipedia.org/wiki/Author_Domain_Signing_Practices
[10]: http://tools.ietf.org/html/rfc5617
[11]: https://dmarcian.com/dmarc-inspector/yahoo.com
[12]: http://jl.ly/Email/yahoobomb.html
[13]: http://jl.ly/Email/aoldmarc.html
[14]: http://postmaster-blog.aol.com/2014/04/22/aol-mail-updates-dmarc-policy-to-reject/
[15]: http://www.ietf.org/mail-archive/web/ietf/current/threads.html
[16]: http://wiki.asrg.sp.am/wiki/Mitigating_DMARC_damage_to_third_party_mail
[17]: https://www.fastmail.fm/
[18]: http://www.macworld.com/article/2029570/silent-email-filtering-makes-icloud-an-unreliable-option.html
[19]: http://dougscripts.com/
[20]: http://dougscripts.com/itunes/2014/06/the-silent-filtering-treatment/