Yahoo and AOL Damage Mailing Lists and Email Forwarding
I recently received an email message from Cornell Alumni Affairs, warning of deliverability problems related to alumni email forwarding and participation in Cornell mailing lists. Since I’ve recently spent quite a bit of time learning about SPF, DKIM, DMARC, and other email deliverability technologies for TidBITS and Take Control, this piqued my interest. Some additional investigation showed that the problem extends far beyond Cornell, and is currently affecting any organization that relies on email forwarding, plus any mailing list with Yahoo or AOL subscribers.
Unfortunately, apart from users switching away from Yahoo and AOL, there’s not much that can be done about it.
First, to explain all this, we need to untangle the alphabet soup of technologies I blithely splashed into the above paragraph, all of which are designed to reduce the ability of spammers to spoof email to make it look like it comes from a legitimate sender.
- SPF stands for Sender Policy Framework and enables the owner of an Internet domain, like tidbits.com, to specify which computers are allowed to send email containing sender addresses in that domain. A mail server receiving email from a sender claiming to be [email protected] does a DNS lookup on tidbits.com, checking a special TXT record to see if the sender’s mail server is allowed to send email on behalf of tidbits.com. You can think of SPF working at the envelope level, and it’s an Internet Engineering Task Force standard (RFC 7208).
- DKIM stands for DomainKeys Identified Mail, and is also an IETF standard (RFC 6376). Whereas SPF looks at the envelope to verify that a message is being sent from an approved source, DKIM goes one step deeper, associating a domain name with an actual email message, via public key cryptography. First, the sending mail server uses a private key to sign the contents of each outgoing message using a DKIM-Signature header. Second, the receiving mail server does a DNS lookup on the domain name specified in the DKIM-Signature header to find another special TXT record that contains the associated public key; the receiving
mail server can then use the public key to verify that the message hasn’t changed since it was signed. -
DMARC, or Domain-based Message Authentication, Reporting, and Conformance, is an email authentication method that builds on SPF and DKIM. Again implemented in DNS, a DMARC policy lets a sending organization specify that its messages use SPF and/or DKIM, and what should happen to messages for which the domain in the header’s From line fails to match (is not in “alignment” with) the domains specified by SPF and DKIM. There are three options: do nothing (“none”), flag the message as suspicious (“quarantine”), or bounce the message (“reject”). Although DMARC is used by numerous major email providers in various ways, it is not an IETF
standard.
A key reason DMARC exists is that the ways SPF and DKIM enable the sender to specify what should be done with failed messages are generally unused. In SPF, each receiving mail server has to determine what to do with results such as “none,” “neutral,” “pass,” “fail,” and “softfail.” Although “fail” was intended to equate to rejection, few senders set that, due to the high number of false positives. The DKIM specification is even more explicit about how its results should (not) be used:
In general, modules that consume DKIM verification output SHOULD NOT determine message acceptability based solely on a lack of any signature or on an unverifiable signature; such rejection would cause severe interoperability problems.
(There is a little-used extension to DKIM called ADSP (Author Domain Signing Practices, RFC 5617), which was the predecessor to DMARC and offers advice from the sender when the From header doesn’t match the domain in the DKIM signature. It was quickly identified as having the same problems that now afflict DMARC; hence its minimal usage.)
So DMARC enables a sender to tell receiving mail servers what to do with messages whose SPF- and DKIM-advertised domains don’t match the domain in the From line. That’s ideal for companies like PayPal, which send vast quantities of transactional email, are constantly spoofed as part of phishing attacks, and don’t have (many) individual users. PayPal has published a DMARC policy of “reject” for over a year with no one noticing. However, until recently, large email providers have stuck with a DMARC policy of “none,” which lets messages be delivered normally. You can check the DMARC policy for any domain at the DMARC Inspector — look at the “p=” tag.
In April 2014, Yahoo dropped a bomb on the email world by quietly changing its DMARC policy to “reject,” and AOL followed suit shortly after, though at least with a press release acknowledging the change. This had the effect of causing receiving mail servers to bounce messages that failed SPF and DKIM, generating two classes of problems:
- Email Forwarding: Let’s say you have a cornell.edu alumni address, which forwards to a Gmail address. Someone from Yahoo or AOL sends you a message, which uses both SPF and DKIM. Receiving it at cornell.edu works fine, because SPF, DKIM, and DMARC alignment all pass at that point, but when Cornell’s mail server forwards the message, it rewrites headers such that, when Gmail examines the new incoming message, it fails SPF, DKIM, and DMARC alignment. Because Yahoo and AOL both have DMARC policies saying that failed messages should be rejected, Gmail bounces the message.
-
Mailing Lists: Now let’s say you use Yahoo or AOL and are a member of a discussion-based mailing list. When you post to the list, the list server receives the message and packages it for distribution to the list, changing a variety of headers and possibly adding a signature to the body in the process. Those changes cause the repackaged message to fail SPF, DKIM, and, most important, DMARC alignment, so it won’t be received by any list recipients at Yahoo, AOL, Gmail, Outlook.com, or other any other email provider that honors DMARC policies, again because of how Yahoo and AOL set their DMARC policies to require that failed messages be rejected.
There’s even more fallout on mailing lists. Depending on how things are configured, if Gmail bounces a received mailing list message sent by a Yahoo user, that bounce will likely go back to the mailing list server and be recorded against the Gmail user, potentially causing that user to be removed from the list. So Yahoo and AOL users can get other list members bounced, purely by posting.
Needless to say, as email and mailing list administrators have figured out what is going on, this change has caused significant consternation. The IETF discussion list has been dominated by DMARC-related discussions for months.
Why did Yahoo and AOL make such a sweeping change? Speculation is that both had suffered significant security breaches that allowed bad guys to steal user information, including address books. They then used that information to create spam to these users’ contacts, forging the users’ return addresses. Since that spam was being sent by botnets, nothing short of this DMARC change could stop it. In short, Yahoo and AOL are cleaning up their mistakes by damaging every mail forwarding service and mailing list on the Internet.
What can be done about this situation? Although email boffin John Levine has compiled a list of things mailing list developers and administrators can do, none are trivial and all have side effects. At the user level, there’s simply nothing that can be done, other than those who use Yahoo and AOL voting with their feet and switching to less-draconian free email providers like iCloud, Gmail, and Outlook.com. Or, if you want to pay an email provider to ensure that they’re working for you, TidBITS staffers have used FastMail and easyDNS’s easyMail service (which comes with domain hosting) without experiencing these problems.
As an aside, although iCloud doesn’t appear to be involved with these DMARC-related problems, beware of iCloud’s spam filtering. In the most recent case, Doug Adams, who has contributed hugely to the Apple community through his Doug’s AppleScripts for iTunes site, found that for three weeks his messages weren’t being received by iCloud users because Apple was silently deleting all messages that contained the string “dougscripts”. Since his domain is dougscripts.com, that pretty much put the
kibosh on all his iCloud-related email. After complaining to AppleCare and having the problem escalated to a Mac Advisor, and then to Engineering, and then to the Spam Team, the problem finally disappeared, without further explanation. It’s astonishing that, in 2014, Apple is still doing simple string-based filtering and silent deletion of offending messages.
What I’d like you to take away from this article, apart from just understanding the Internet a bit better, is that there’s a lot of effort that goes on behind the scenes to make sure even the most basic of Internet technologies work in the face of competing forces. Email may seem simple, but it’s also far more subtle than most people realize.
I'd like to point out that gmail as a free mail provider to use with mail lists is a bad suggestion. Especially when dealing with multiple mail lists that are often cross-posted.
Gmail in-appropriately tracks the msg-id of the messages it receives and usually ignores duplicates. As mail lists do not change the msg-id header of a message, gmail often ends up dropping what it thinks are duplicate messages coming from the additional mail lists when an email is cross-posted to multiple mail lists. This can make it difficult for a receiver to know which mail lists were included if the mail lists cross-posted to, were BCC'ed.
I once took this up with Google, and they called it a feature, so it is unlikely Google will change this behaviour.
Interesting. I'm only on a pair of lists that often receive cross-postings (two local running clubs), but because one of them has the policy of prefixing the Subject line, there's never any problem with Gmail dropping one of the messages.
But yes, if knowing where cross-posts were directed is important, Gmail isn't the best option. I'd be surprised if most people even know what this involves, much less care, but for those who do, another service is a better choice.
I hear that yahoo managed to hose their own email lists along with everyone else's...
I run several mailing lists for non-profits, so was bitten by this. I reminded all subscribers that it's their responsibility to be able to receive their mail, and that is was time they became genuine customers of fastmail or similar paid service. To my surprise, most people on yahoo would rather lose a bunch of their legit mail and keep complaining about it than pay $10 a year for a known good account. With all of the extra bounces to sort through, I'm tempted to ban yahoo and aol addresses outright.
Microsoft mail servers (hotmail, msn, etc) stopped accepting any mail from my server over a year ago for reasons I was never able to fathom, even with the help of a friend who works at msn. Considering that MS bans all UW mail once or twice a year, and it takes several days for the campus network people to sort it out, I gave up.
Yeah, I think that banning Yahoo and AOL users from posting to mailing lists may become commonplace, since the collateral damage of letting them participate is too high.
I have been an AOL user for over 25 years (in fact I'm an "AOL Charter Member"), so AOL is my PERMANENT email address. I have had about 10 OTHER providers, but due to the problem with email not having a way to forward your email for a set period of time when you have to change Internet service providers (like you can with regular postal mail), AOL has to stay my default email.
My current ISP contracted with GMail years ago to manage their email. Because of GMail's excellent SPAM filtering I have it set to retrieve my AOL mail for me and have my mac.com email forwarded to the GMail-managed address. Unfortunately, I ran into this same DMARC problem with email sent FROM this "GMail" address to a list I was subscribed to with the "GMail" address. Only after canceling that registration and re-registering with my Dot Mac address did it stop being a problem.
Happily, Mailman was recently updated to allow for a couple of workarounds. The "from_is_list" setting under General Options in the admin interface allows the admin to either mung the "from" line or wrap the message.
In all the years I've been involved with administering various Yahoo Groups, I've never known Yahoo to properly respect email. Every email message that passes through their servers is mangled in some way.
For example: Forget about a PGP/GPG signature surviving Yahoo's email servers. It doesn't happen. The resulting signature at the receiver's end can never be verified thanks to Yahoo manglement.
This stuff is mostly over my head. Even so, I'm glad to learn more about these problems. It seems to happen that when it learn about an issue like this, I will soon have to deal with it for a client. So I'll save this article for future reference. In particular, I expect I'll need to know where to go when the free e-mail services fail. So thanks for the info.
I concur with Mr Le Blanc, and I also will keep a copy of this article for my consulting clients. As a side note, a non-profit I work with recently ran into a major problem with "Roadrunner" mail recipients (a Time Warner Cable service). They quietly implemented bounce routines for lists that use multiple concurrent connections as well as ANY sender of individual messages to more than 25 rr.com addresses simultaneously. We've been able to get some help from TWC tech support to achieve a workaround for our group...though the fix has broken twice in as many weeks.
Yes, sorry! It's a subtle and complicated situation, and there's no way around that. But hopefully if you run into problems with forwarded email or mailing list messages from Yahoo and AOL users bouncing for no apparent reason, you'll have a sense of why. Even if there's not much that can be done about it.
I sent the article link to the NewtonTalk list and to the MiataMail list since both are having the same problems.
yahoo and aol are wankers anyway. they deserve to lose users over this gaffe on their part. Serves them right.
I'm looking into ProtonMail.
Yet another verification on the "beware of iCloud" point.
I've had iCloud drop messages on at least four occasions now that it THOUGHT was Spam based on message content but were not, and the only way I ever found out was when the sender asked why I hadn't replied to their message.
I tested by sending the same message to myself and watching it silently disappear.
The only solution in such cases is to keep kicking it higher in Apple tech support until the right iCloud Mail person gets it and modifies their SPAM filters to let it through.
In short, though email delivery is NEVER guaranteed, iCloud email in particular should not be relied upon for anything more than normal correspondence.
I understand the need, but it bothers me that some messages get filtered before I can ever see them. I liked the way one of my prior employers had things set up: messages flagged as spam would be available to see in text-only via a web interface so I could delete or "OK" them individually. After 30 days they'd expire. I'd like to see something like that from iCloud. I'm probably in the minority on this, though.
I run a small list for a nonprofit. Messages that I send using smtp.pobox.com to said list are getting swallowed when passed along to mac.com addresses. Where do I even start with Apple to get this fixed?