This article originally appeared in TidBITS on 2013-10-22 at 8:58 p.m.
The permanent URL for this article is: http://tidbits.com/article/14219
Include images: Off

Mail in Mavericks Changes the Gmail Equation

by Joe Kissell

[Note: Apple’s OS X 10.9.2 release fixes many (though not all) of the lingering problems with Mail; see "Mail Improvements in OS X 10.9.2 [1]," 25 February 2014. This follows a special, separate update to Mail, which addressed some issues (as discussed in “Apple Updates Mail to Address Mavericks Bugs [2],” 7 November 2013) and the 10.9.1 update, which fixed others.]

Apple Mail in OS X 10.9 Mavericks treats Gmail accounts differently than any previous version of Mail did. Although some of the changes are quite clever and useful, the implementation is not without flaws. Your mileage may vary, of course, but based on my own experiences and those of hundreds of people who have commented online, Mail’s behavior with Gmail accounts — especially at first — leaves a lot to be desired. Here’s what I’ve observed and what you can (and can’t) do about it.

[image link] [3]

(The Joe Hulk drawing above is a snippet [4] of an awesome Joy of Tech [5] comic published in response to this article.)

Pseudo-IMAP Changes -- Mail and Gmail were never a fantastic combination out of the box, because Gmail has a wacky, highly nonstandard way of using IMAP, and Mail always wanted to treat Gmail as though it were a conventional IMAP server. But, after much trial and error, I eventually found a combination of Mail settings and Gmail settings that, prior to Mavericks, resulted in a stable — and indeed largely pleasant — experience. As I documented in “Achieving Email Bliss with IMAP, Gmail, and Apple Mail [6],” 2 May 2009, you just do x, y, and z (well, 21 steps’ worth of x, y, and z), and it will all work smoothly.

Well, forget about that under Mavericks. In fact, following those old directions now will lead you far from bliss. And if you followed them before upgrading to Mavericks, you’ll need to take some steps to undo some of the problems.

Before Mavericks, the approach that worked best with Gmail accounts in Mail was to go into Gmail’s settings and prevent the All Mail label from being exposed to IMAP clients. All Mail is exactly what it sounds like — all your saved and sent messages, regardless of whether or how they’re labeled — and having All Mail enabled, prior to Mavericks, meant that Mail would download at least two copies of every message (one in All Mail and one each in a mailbox corresponding to any labels you applied in Gmail). That led to lots of duplicate messages, wasted disk space and bandwidth, and reduced performance. But hiding All Mail prevented the problem.

Mail in Mavericks tries to meet Gmail on its own terms, more or less. As part of this approach, it now treats archiving Gmail messages essentially the way Gmail itself does — moving a message from the Inbox to Archive removes the Inbox label, which means it shows up only in Gmail’s All Mail list (unless you apply another label in Gmail or move it to another mailbox in Mail). Note that deleting a message in Mavericks Mail won’t archive it in All Mail; it (logically enough) moves it to the Trash. I’ve always counseled against using the Delete key to mean “save this forever,” but if you’re in the habit of pressing Delete to archive, that won’t work anymore in Mavericks.

Unfortunately, if you had hidden the All Mail label from Mail in Mavericks, then any messages you move from your Inbox (by filing or deleting) magically reappear back in the Inbox later — after you switch to another mailbox and switch back, or close and reopen Mail. That will, of course, drive anyone to distraction. I reported this to Apple as a bug, and it was marked as a duplicate, which means only that I wasn’t the first person to report it, not necessarily that Apple is planning to fix it. (A post on Pocket-lint [7] suggests that the changes to Mail’s behavior with regard to All Mail were at Google’s behest, but I don’t know the details.) The only way to “solve” this problem is to re-enable All Mail (which, by the way, affects all IMAP clients, not just Mail). Which I’ll now tell you how to do, but don’t do it until you read about the consequences.

To re-enable All Mail, log in to your Gmail account on Google’s Web site. Click the gear icon and choose Settings from the pop-up menu. Click Labels. Find All Mail (under System Labels, near the top) and select its Show in IMAP checkbox.

Now, here’s what’s going to happen. Mail — despite the fact that it has already cached all your Gmail messages — will download all of them again. For me, with about 321,000 messages totaling over 4 GB, that took nearly two full days, even with a super-fast Internet connection. That’s an unreasonably long period of time, and a crazy waste of bandwidth since I already had copies of all those messages! Mail actually does this in stages, and I won’t bore you with the details, but I will say that at a certain point in the process, your ~/Library/Mail folder could be twice as large as it should be, or even larger. Unless you’re running critically low on disk space, don’t panic about that, because it’ll eventually settle down — but be aware that it could take Mail a very, very long time to purge all those duplicate messages and return your ~/Library/Mail folder to a reasonable size.

When that big download is done, the good news is that Mail will have only one copy of each message — the one in All Mail! Messages you had labeled in Gmail will still show up in mailboxes bearing the same name. (And, for the first time, Mail will have a copy of all your messages that didn’t have any label at all — ones you had archived without explicitly filing.) But those messages are not really in those mailboxes. What Mail does behind the scenes is to add a little invisible XML code to the end of each message telling it which other mailbox(es) it should be displayed in. And that’s an entirely reasonable strategy, as far as it goes. Arguably, Mail should have started doing this years ago.

However, this change has some unwelcome side effects, and it seems Apple didn’t entirely think through the implications. For one thing, Mail doesn’t alert you in any way to the fact that it now needs access to All Mail; it simply fails to operate correctly. It could pop up a little message saying, “Hey there! I noticed that you have a Gmail account and I’m not seeing All Mail. That’s going to be a problem from now on; here’s how you fix it and what you can expect afterward.” But this is something you just have to work out for yourself — a major architectural change that isn’t even mentioned when you choose Help > What’s New in Mail?, let alone addressed in a helpful error message.

Be that as it may, once you’ve figured out that you need to enable All Mail, you may encounter another hurdle. What I found, and what numerous other users reported, is that even after All Mail was enabled and enough time had gone by for Mail to re-download, re-cache, and re-index everything, Mail’s Inbox didn’t stay in sync with Gmail’s Inbox. That is, reading, moving, deleting, or otherwise operating on messages in the Inbox on the Gmail Web site, on an iPhone or iPad, or in another IMAP client, made no difference to what appeared in Mail’s Inbox, even after several hours. And, the usual fixes, such as quitting and restarting Mail, rebuilding the Inbox, and forcing a synchronization, had no effect.

For me, after experiencing this frustration for about four days (during which time I had to keep Gmail open in a Web browser to make sure I saw incoming messages when they arrived), the problem spontaneously disappeared and Mail’s Inbox resumed syncing immediately, just as it always had in the past. I don’t know what caused that change, but I was happy about it, as it suggested that this particular problem may go away for others, too, given sufficient patience. But then, about 12 hours later, my Inbox suddenly stopped syncing again — and as before, nothing I tried was able to bring it back to life. Since then, it’s been off and on — sometimes it works, sometimes it doesn’t. For what it’s worth, anecdotal evidence suggests that those of us with extremely large numbers of messages in Gmail are likely to experience longer Inbox-out-of-sync times; those with only a few thousand messages often notice no misbehavior at all.

What if you never disabled All Mail for IMAP clients in the first place? I wish I could say that makes the upgrade smooth sailing — and for some people, it is. But I’ve also heard from users who never touched their default Gmail settings and, after upgrading to Mavericks, still found it took a ridiculously long time before Mail began displaying the contents of their Inbox and other mailboxes correctly.

Another problem is that with All Mail enabled, AppleScript breaks badly with Gmail accounts. If you have any AppleScripts that operate on messages in Gmail mailboxes (I have some that are crucial to my workflow), the scripts will report the mailboxes (except Inbox and All Mail) as being completely empty. Of course, they are empty in reality, but they don’t look empty in Mail because Mail does the right thing when interpreting Gmail’s labels. Unfortunately, AppleScript doesn’t know anything about this. (I also reported this as a bug, and it, too, was marked as a duplicate.) In the meantime, my workaround is to run 10.8 Mountain Lion in a VMware Fusion virtual machine so I can still use my old Mail AppleScripts.

As I mentioned, turning on All Mail affects not just Mail but any other email client you may use on other devices, as well as older versions of Mail. So, “fixing” Gmail on my Mavericks system breaks it on other systems (including my Mountain Lion virtual machine!), forcing me to choose which set of problems I’m willing to cope with.

The whole “All Mail” thing (and the delays that result) may be the most obvious change in the way Mail works with Gmail, but there are other problems too:

Further Issues -- Lest you think that only Gmail users are having trouble with Mail in Mavericks, the list of problems that affect all providers is also long and growing. For example:

And, I haven’t bothered to mention the fact that most third-party Mail plug-ins broke under Mavericks. That’s normal when OS X is updated – and most major plug-ins have already been updated to support Mavericks — but fixes for others are still in the works.

If you’ve noticed other wonky behavior with Mail in Mavericks (whether pertaining to Gmail or not), please tell us in the comments.

I’d like to think that an OS X 10.9.1 update will magically fix all this stuff, but I won’t hold my breath. I’m sorely tempted to look for a different email provider (something I was pondering anyway, for unrelated reasons), but it irks me that I should have to do so now just because Apple broke Mail in the very process of trying to improve the way it works with Gmail.

The alternative, of course, would be to switch email clients, but although I’ve tried many of them, I have yet to find one that offers all the crucial behaviors I get from Mail and my carefully chosen set of plug-ins. What I really want to do is continue liking both Mail and Gmail, but Mavericks makes that impossible at the moment.

[1]: http://tidbits.com/article/14547
[2]: http://tidbits.com/article/14280
[3]: http://tidbits.com/resources/2013-10/JOE-SMASH-border.png
[4]: http://www.geekculture.com/joyoftech/joyarchives/1917.html
[5]: http://www.geekculture.com/joyoftech/
[6]: http://tidbits.com/article/10253
[7]: http://www.pocket-lint.com/news/124607-apple-mail-in-os-x-mavericks-features-troublesome-gmail-changes
[8]: http://blog.fastmail.fm/2013/10/26/mac-os-10-9-infinity-times-your-spam/