FlippedBITS: IMAP Misconceptions
In today’s installment of FlippedBITS, I want to examine a handful of common misconceptions about IMAP, a familiar protocol for retrieving email from a server. IMAP stands for… well, thereby hangs the first tale. IMAP’s inventor, Mark Crispin (who, sadly, died in December 2012), called the first version of his creation Interim Mail Access Protocol. Versions 2, 3, and 2bis were referred to as Interactive Mail Access Protocol, and version 4 — what’s in use today — is officially Internet Message Access Protocol. Although many Web sites claim that the acronym once stood for Internet Mail Access Protocol, I have found no credible references to back up that claim.
By whatever name, IMAP has always been a means by which email clients can talk to email servers. That puts it in the company of POP (Post Office Protocol) and Microsoft’s MAPI (Message Application Programming Interface). Almost every modern email client — including Apple Mail, Thunderbird, Microsoft Outlook, and dozens of others — supports IMAP as a means of retrieving email. You may very well have been using it for years without even knowing it — iCloud and its predecessors MobileMe, .Mac, and iTools have always defaulted to IMAP for email access.
New Kid on the Block — The first thing I want to clear up is the persistent notion that IMAP is some sort of newfangled email system, a regular Johnny-come-lately compared to the ancient and revered POP method. Yes, POP has been around quite a while — it was invented in 1984. IMAP came along in 1986. (For perspective, Apple’s Macintosh System Software 5 — the first one to include MultiFinder — was released in 1988.) Both protocols subsequently underwent numerous revisions, but in any case, it’s a bit silly to consider POP “traditional” and IMAP “new.”
Now, it’s true that in the early days, email clients and servers alike were more likely to support POP than IMAP (and even today, IMAP support isn’t universal). So, many of us who have been using the Internet for a long time became accustomed to POP — and a surprising number of people still use POP, often out of habit more than necessity. (I’ll return later to whether that’s a good idea.) But IMAP has been a viable option for decades.
Are You Being Served? — The usual way people explain the difference between POP and IMAP is to say that with POP, all messages are downloaded from the server to your email client, whereas with IMAP, messages are stored on the server. That’s sort of true-ish, but it’s unfairly misleading in both cases. With POP, you can leave messages on the server if you want to, and with IMAP, you can download all your messages and store them locally. The simplified “IMAP-means-stored-on-the-server” explanation has led countless people to assume that you can use IMAP only when you have an active Internet connection. But that isn’t the case. For as long as
I’ve been using IMAP, I’ve maintained local copies of every single message in my accounts, and have never had trouble reading, searching, filing, or otherwise managing my messages when offline.
The best way to think about IMAP is that the server holds the master copy of every message. Whenever an IMAP client connects to the server, it can synchronize changes bidirectionally — for example, new messages in the Inbox download to the client; changes made in the client while it was offline upload to the server, updating the master records. But the exact behavior is determined by the design of the client and settings chosen by each user. By default, Apple Mail (like most other modern email clients) keeps all your messages in sync between client and server. But if you prefer, you can configure your client not to cache messages for offline viewing, to cache only some messages, or to cache the text of messages but not any
attachments.
I should add that even though the server stores all your email messages, this in no way prevents you from deleting messages. Although, again, the exact behavior varies according to your client and your settings, when you delete a message locally, your client normally tells the server to delete its copy too.
I’ll File Away — Another prominent difference between POP and IMAP is that IMAP lets you define mailboxes (that is, folders for email messages) that are stored on the server and (in most cases) synced with your email client. In general, the effect is that no matter which IMAP client you use, on which platform, it will always reflect the same set of mailboxes with the same contents; you’ll never have to worry that you might have filed a certain message on the wrong computer.
With POP, there’s no such thing as server-based mailboxes, just an Inbox, so any filing you do must, by definition, be done in the client. However, with IMAP, even though server-side mailboxes are supported (and quite handy), if you prefer to store some or all of your messages in local mailboxes, nothing’s stopping you. In fact, if your IMAP provider imposes a storage quota, you may want to move messages from server-based mailboxes into local mailboxes from time to time in order to free up space on the server.
The Same Thing, Only Different — I’ve heard it said that if you configure your POP client to keep all messages on the server — that is, not to delete them after they’re downloaded — then POP becomes so similar to IMAP that you probably won’t be able to tell the difference. But that’s very far from the truth.
Apart from the lack of server-based mailboxes (which, of course, you’re not obligated to use in IMAP), leaving messages on a POP server is much different from leaving messages on an IMAP server. Crucially, IMAP servers keep track of which messages you’ve read, replied to, and forwarded. So, suppose I connect to a POP account that has 15 messages in the Inbox. I download and read them, but leave them on the server. Now I go to a different client or computer and connect to the same POP account. The same 15 messages will download again (along with any that have arrived in the meantime), with no indication of which ones I’ve already read. By contrast, if I do the same thing with a pair of IMAP clients, each one will show me
the same thing — these messages have been read, those haven’t; this one has been replied to; that one was forwarded; and so on. This makes it much easier to switch among clients — something that becomes increasingly important as more of us have not only multiple computers but also smartphones, tablets, and other Internet-connected gadgets.
Speaking of multiple clients, you should be aware that POP permits only one connection at a time per account, while IMAP has no such limit. So, although your three Macs, two iPads, Windows PC, and iPhone can all maintain live connections to an IMAP account, they’re forced to take turns with a POP account.
All of a Piece — But now, let me turn that around and address another misconception, that all IMAP servers are created (more or less) equal. Would that it were so, but no. IMAP servers are as frustratingly different from each other as clients are. It all comes down to three words: specification, implementation, and configuration.
The IMAP specification, as I mentioned earlier, has undergone a number of revisions. In addition, it supports the use of optional extensions to provide extra features. When it comes time to implement the specification, one developer might use an older version of the spec, or interpret part of it in an idiosyncratic way, or choose to include or omit various extensions for one reason or another — while the next developer might make entirely different choices. And some developers might decide that the standard IMAP approach doesn’t meet their needs, so they leave things out, slap extra things on, and rejigger other things so they work in surprising ways. (This happens more often than I’d like to admit, although Gmail’s flavor of
IMAP is arguably the least IMAP-like, which is not surprising since it was an afterthought rather than a part of the original Gmail design.) Moreover, IMAP servers have a variety of settings a system administrator can configure, just as IMAP clients have user-configurable preferences. All these variables can make any IMAP client/server pair behave much differently from any other.
I can’t tell you how many times I’ve had to say things like, “Yes, your IMAP server supports subscribing to specific mailboxes, and so does Outlook, but Apple Mail doesn’t,” or “Apple Mail supports IMAP IDLE (see “How Apple Mail May Be Anything but IDLE when Pushing Email,” 22 October 2012) but your IMAP server doesn’t,” or “Gmail’s idea of archiving bears only the remotest resemblance to Apple’s idea of archiving.” One especially troublesome area is the way various IMAP servers and clients handle deleting messages — a messy topic I address somewhat in my books about Apple Mail (“Take Control of Apple Mail in Mountain Lion” and “Take Control of Mail on the iPad, iPhone, and iPod touch”) but won’t delve into further here.
POP on over to IMAP — Notwithstanding the several quirks and annoyances of certain IMAP implementations, my fondness for IMAP is right up there with my fondness for chocolate. (That’s way up there, in case you were wondering.) Let me summarize the advantages of IMAP over POP:
- The server keeps a master copy of all your data (including mailboxes and message metadata such as read or replied). So you’ll see the same thing with any client on any platform.
- You can connect to an IMAP account from multiple clients at the same time.
-
If your client supports it, you can have it download only message headers, with full message bodies on demand.
-
You can ask your client to search for messages on the server, even if they haven’t been downloaded. (The iOS version of Mail supports this, but the Mac version doesn’t.)
The oft-heard objection to IMAP that it takes away one’s control is a myth. As long as you have your client configured to cache a local copy of all messages and to delete messages on the server when you delete them locally, you maintain just as much control over your email as you do with POP. Most of the old assumptions that led users to favor POP — such as the expectation that a person will use a single computer for email most of the time, and the belief that online storage is expensive — are no longer valid in today’s world.
Are there still legitimate reasons to use POP? Sure. For one thing, it’s less chatty than IMAP, so it tends to be better in low-bandwidth situations, especially when lots of users are connecting to an underpowered server. (Having said that, mobile IMAP clients typically manage to do a great job even over slow cellular connections, but that assumes optimization of both client and server for that purpose.) Also, most providers cap each user’s IMAP storage quota, so if you have vast amounts of stored email, you may be forced to offload some of it to local mailboxes; by contrast, POP normally holds onto messages only until the user picks them up, so its storage requirements tend to be lower. And, if you’re concerned that your email
provider can’t be trusted or is vulnerable to hacking, you might prefer not to keep unencrypted email on a server any longer than necessary. Finally, not all email providers support IMAP. But that leads me to my final point.
Stuck in the Past — I’ve heard from a number of people who tell me they’d like to use IMAP, but they can’t, because their ISP doesn’t support it — or charges extra for it. So, two things here.
First, even if your ISP doesn’t offer IMAP for accounts on its own email server, that in no way prevents you from using another IMAP provider. When an ISP says they charge extra for IMAP, that means they charge extra to use their IMAP server, not any IMAP server. You can go right ahead and use iCloud, Yahoo Mail, AOL, Gmail, or any of a hundred other services that offer IMAP access to email — many of which are free.
Second, if your main email address comes directly from your ISP, and that ISP doesn’t support IMAP, you can usually set up an IMAP account with a different provider and then forward mail from your ISP to the new IMAP account. (Exact directions to do this depend on the provider.) That way, anyone with your old address can still reach you, while you get to enjoy the advantages of IMAP.
If you’ve weighed the pros and cons and decided that a switch from POP to IMAP is for you, see if your existing email provider offers an IMAP option — sometimes it’s as simple as flipping a switch on the server side, although you’ll likely have to configure an entirely new account in your email client. For further guidance, I recommend Kirk McElhearn’s Macworld article “How to convert a POP email account to IMAP.”
"The simplified “IMAP-means-stored-on-the-server” explanation has led countless people to assume that you can use IMAP only when you have an active Internet connection."
This has also been perpetuated by mobile devices that only keep what you have already read on the device, only checking the inbox by default, and usually having a limit for how much of a message is automatically downloaded. This is however getting better with push support in IMAP servers, although may still be limited to just the inbox.
For years I've also run a local IMAP server on my Mac which I use to store my message archive. I use uw-imap because it can use unix mailbox format files and these are easily transportable between systems. I would never trust a program like Apple Mail or Thunderbird to store my mail spool.
Jose: can you tell us a bit more about how you do this?
I built a version of UW IMAP (now Panda IMAP) that has been configured to look in the user's home directory for all its mail folders. I created a launchd plist in /Library/LaunchDaemons that runs the IMAP daemon listening on locahost. I configured Apple mail with an IMAP account where the server is the localhost. I've made an installer which sets this all up. I may make public when I iron out some of the details.
I use IMAP on my mobile devices, but POP on my Mac at home. Why? Because, for security of its users, my email provider does not keep messages on the server for longer than a month. By using POP on my home computer I can keep all my messages as long as I like.
This is a great summary for the somewhat less tech-savvy power users out there! I have a few people I'll forward this on to.
The only thing it's missing are some IMAP best practices and limitations. For instance, I've had LOADS of trouble with clients that want to use more than about 100 folders, folders nested more than 3 levels deep, or more than 10,000 messages in a single folder. Most mail clients let you get away with that in the local mail store. But, when trying to sync with the server, odd behavior starts to creep in. On iOS, the mail client usually just chokes and stops downloading folders and mail after some point.
My advice to clients is always to start archiving locally after a certain amount of filing or message accumulation. Anyone else have thoughts on IMAP best practices?
> archive locally
A checklist would help.
Yes, do it. I use CEsmail/Spamcop. A few years ago all the email disappeared for a while, though they got back all but two items eventually. I hadn't realized that if the server erases all your email, your home computer will cheerfully and immediately do the same thing -- all gone.
If they hadn't been able to restore it, it'd have been gone, all gone.
I found Mailstore.com (free for non-commercial use) and use that service regularly now. When properly set up that, run manually from time to time, copies everything off the IMAP server to a backup file. Mailstore is Windows-only so I have a Windows license and a virtual machine running it under Virtualbox.
I don't use IMAP because I monitor several email addresses from several computers, and I don't want my mail client to delete email from the server, making it unavailable to the other computers I use. My POP configuration only deletes email from the server when it is read by the computer where I want the mail archived. I'm open to moving to IMAP if there's a way to get that behavior.
IMAP does what you want by default. The server deletes messages only when a client tells it to. Plus, as a bonus, all your computers can see which messages have been read, replied to, or forwarded. Explaining this was sort of the point of the article! I also monitor multiple email addresses on multiple computers, and I can't imagine having to do that with POP!
Interesting read, but I noticed a couple of minor errors:
1. Retrieving an e-mail from a POP3 server does not result in that e-mail being deleted automatically. This has to be done explicitly by the e-mail client by first issuing a command (DELE) and then closing the session (QUIT). Sure, most (all?) e-mail clients default to do this after retrieving an e-mail, but here you're comparing protocols.
2. POP3 is just as capable to download headers only (TOP # 0) as IMAP, or only the first 10 lines for that matter.
Of course, those two points don't really change anything as IMAP's ability to synchronize, create folders etc can't (easily) be replicated using POP3.
I didn't claim that retrieving a message from a POP server deletes it automatically; in fact I mentioned a few times that this need not happen, depending on your settings. But, as you say, that's usually the default.
Although I didn't want to go into this level of detail in the article, IMAP is MIME-aware, so it can download (or not) specific message part(s), such as everything but attachments. That's the real difference with POP in terms of downloading portions of messages.
Don't mind me; I'm just nit-picking here =P
Thanks for the clarification/information regarding IMAP also being MIME-aware; I wasn't aware of that; and thanks again for an interesting read.
I use Thunderbird to monitor a dozen email accounts across 3 domains. Everything arrives in a unified inbox. I use filters to send messages that have been read to any of several hundred folders. Mail from different accounts may be filtered to the same folder.
Whenever I have experimented with IMAP, I have not found it possible to replicate these behaviors. Should I try harder?
You can absolutely do all that with IMAP. And, if you store all those mailboxes on the server, you'll be able to see the same thing on multiple computers. However, note that it would be a nontrivial exercise to transfer hundreds of local mailboxes to an IMAP server.
I don't know exactly what you've tried or what problems you've encountered, but in principle there's no reason you shouldn't be able to do what you describe.
Good to know, thanks for the reply. Sounds like a multiple-rainy-day project... :)
Sorry, here's a dumb question: All the hard work I've put into building my Thunderbird filters will be preserved? And going forward, all the folders will be magically created/updated on the server? In other words, can I continue to use Thunderbird as my "headquarters" for managing messages? Thx.
Well, there's no magic involved. If you want to replicate your local mailbox structure on an IMAP server, you'll have to move/copy/recreate those mailboxes (and their contents) on the server. That won't happen automatically, although in theory it could be a simple matter of drag and drop (and wait).
If your filters are designed to store messages in specific local mailboxes, you'll have to change them to point at their new server-based equivalents. Thunderbird has no way to know what you want it to do unless you tell it.
I'm saying: Thunderbird + IMAP can do what you want to do. Not at all saying the transition process will be automatic or painless.
I've used IMAP for years, but it does have a downside I've not heard of any way to address.
I have no problem with the fact that deleting an email on my iMac deletes it on the server. That's fine. I don't want either cluttered with spam.
But I'd like to be able to check that same mail on my iPhone on the go, delete almost everything there, but not have it deleted on the server. That I don't know how to dy.
Deleting mail from _any_ IMAP client will tell the server to delete it. That's the nature of IMAP—to make sure all your devices reflect exactly the same thing.
When you read mail on your iPhone, it'll be marked as read on the server (and on your other devices), and you can also file it on your iPhone and that filing will be reflected everywhere else. But no, you can't delete it just on one device and not the rest. I've never heard of anyone else wanting to do that.
Sorry, but count me agreeing with M. Perry.
To my mind this is very logical since you have tons of space (relatively), and chances are your computer's eMail client has more sophisticated viewing and handling options (assuming your not using a browser based interface).
This is the reason I still far prefer POP mail.
I think the question is, what's the goal in deleting it on the iPhone? Storage space isn't an issue there because the Mail app will manage that (it won't store much) and if you don't want to delete the message from the server, doing so on the iPhone is an unnecessary extra step, when you could just move on to the next message (or file it, or archive it).
An apparent peculiarity of Google's gmail POP service is that it will allow a message to be retrieved by a POP client exactly once. If a POP client on machine 'A' (configured to leave messages on the server) reads a message, a POP client on machine 'B' will never see it. Despite that, that message will be visible through gmail's web interface, and appear as "unread". (The other POP servers I regularly use do not exhibit this behavior.)
I haven't tried POP with Gmail, but now I have another reason not to! I sort of see what they're trying to do there, but I think they're creating a bigger problem than they're solving.