Thoughtful, detailed coverage of the Mac, iPhone, and iPad, plus the TidBITS Content Network for Apple consultants.

How Apple Mail May Be Anything but IDLE when Pushing Email

So much happens behind the scenes on our computers that at times technology can seem magical. For example, when a new email message arrives, Notification Center alerts me simultaneously on both my Mac and iPad. As with most daily technological magic, I didn’t think much of it until I was editing Joe Kissell’s immensely helpful “Take Control of Apple Mail in Mountain Lion”, when Joe and I had an enlightening discussion that took place in the manuscript’s margin comments.

At issue was Notification Center in OS X 10.8 Mountain Lion and how mail notifications would work with it. I proffered the suggestion that Notification Center in Mountain Lion would hook up with the same push service that iOS devices used, and was surprised when the manuscript came back to me with a comment from Joe saying that Apple Mail on the Mac did not use push services in earlier versions of Mac OS X and, more importantly, would not be using them in Mountain Lion. I was doubly surprised because, just as I was reading his comment, I heard the new mail sound simultaneously from both my Mac (running 10.7 Lion) but also from my nearby iPad. I knew my iPad was using push services, but what was Apple Mail on my Mac using that could account for the simultaneous new mail sound? It may not have been push email on my Mac, but it sure seemed to act the same as push email on my iPad to me!

A little bit of research provided the answer: Apple Mail in Mac OS X uses an optional bit of the IMAP (Internet Message Access Protocol) specification — the IDLE command. The clue that quickly led me to the answer was an option in the Advanced pane of Mail’s Account preferences: “Use IDLE command if the server supports it.” For my iCloud email account, this option was checked by default.

A Little IMAP Background -- IMAP is an advance over POP (Post Office Protocol), which was the way email used to work in the deep dark past of the Internet, and which, to the dismay of some, still is used by many email services today.

With POP, incoming messages arrive at the mail server and remain there until the client (that is, your email program), fetches them from the server and, optionally, removes them from the server. The mail server only keeps track of the messages on it; it doesn’t record whether you have actually read or downloaded the messages. Those tasks are the responsibility of the mail client. Thus, you can fetch messages from a POP server with your mail client, and, if that client doesn’t delete the messages from the server, you can fetch them again using another mail client (on the same computer or a different one), and those messages appear in that client as brand-new unread messages.

With IMAP, the mail client displays the messages as they exist on the server. The client may or may not download the messages to the local device — that’s not important. What is important is that the IMAP server knows when the client reads a message. This means that when you read your IMAP mail with one mail client, and then connect to your account with a different client (on any device or computer), you can see which messages you have already read, regardless of the client with which you read them. If you use multiple devices, IMAP’s advantage is synchronization of which messages have been read or deleted. For example, you can read a message on your iPhone and later find it marked as read on your Mac. This discussion intentionally scratches only the surface of IMAP, but the takeaway is that IMAP mail clients and servers can have much more complex “conversations” than those allowed by the more basic POP protocol.

Back to IDLE -- What does all this have to do with arriving email notifications? After all, whether the protocol is POP or IMAP, the mail client still has to connect to a mail server and send a request to it asking for mail. Except, that is, for when an IMAP server supports the IDLE command.

When a mail server lets a mail client know that it can respond to the IDLE command (Apple’s iCloud servers do, though many others don’t), the client can send the IDLE command to the server with the following result: the server maintains an open connection between it and the client so that, when new mail arrives, the server can immediately let the client know about it. Then the client can inform the user that new mail is available. (This description of how clients and servers use the IDLE command, of course, is almost criminally over-simplified; you can read all the geeky details in RFC 2177.)

So, the reason I know that I have new mail available on my Mac is because Apple Mail and the iCloud server are in constant contact with one another. The iCloud server says, “You have mail,” and Apple Mail says, “Oh, goodie, let me see it.” Plus, in Mountain Lion, Mail lets Notification Center know about it, too.

What about Push? -- IDLE in IMAP is relatively old: it was first specified late in the last century, a decade before the iPhone saw the light of day. IMAP IDLE assumes that the mail client is running and can maintain an open network connection to the mail server. That’s great for devices plugged into a power source and an Ethernet network, but not so good for a pocket-sized device with a tiny battery and a potentially expensive connection to a cellular data network. That’s where push comes in.

In iOS, there isn’t a whole lot of memory to support simultaneously running apps, nor enough battery power to keep a bunch of them active at any one time. There is, however, enough of each to keep a small program semi-awake and aware in the background so it can listen to the mobile network. That’s how Apple’s push system works.

With push, an iOS app registers with that small background program in iOS, informing it that the app expects to receive network notification messages — even when the app isn’t running! That small background program in iOS listens to the network and fields incoming communications, sending the appropriate notifications to the apps that have requested them.

On the other end, remote applications, such as mail servers, can register with an Apple Push Notification (APN) server, passing that server a token that identifies which device out there on the Internet is interested in receiving notification messages. It’s Apple’s APN servers that send the notification messages to the iOS device from the remote application.

The path is something like this: the mail server receives new mail for you and sends a notification message and a device token to the APN server. The APN server then sends the appropriate data to the small background program running on your iOS device so you can hear the new mail sound, see the badge number change on the Mail app’s icon, and read a brief summary of the incoming mail’s content (the push system can send a notification payload of no more than 256 characters). When you open the Mail app on your iOS device, the Mail app then establishes a full IMAP connection with the remote mail server so you can read the entire message.

(There’s yet another push technology Apple devices can use: Microsoft’s ActiveSync; curious readers can find out more about how that works in the Microsoft article, “Understanding Direct Push.”)

Subtle Differences -- These two systems, IDLE and push, account for the multiple notifications I received simultaneously on my iPad and on my Mac when the new message arrived in my iCloud account. But there are subtle differences in behavior between the two systems.

On the Mac, you receive new mail notifications only when the Apple Mail application is running, because the conversation between Mail and the mail server is what enables notifications to be made. When Mail is not running, or your Mac is asleep (and isn’t using the Power Nap feature in Mountain Lion), your Mac has no way of knowing when new mail arrives, so you don’t get notifications.

On an iOS device, you always receive notifications as long as your device is turned on and has a network connection — even when the device is asleep, that little background program is awake enough to receive incoming push notifications. (By the way, that’s why you can eke out a little more battery life if you disable push on your iOS device: that little background program doesn’t use much power, but it uses some.)

However, the Apple push system provides for notifications that travel from the APN service to the iOS device; it doesn’t send messages back the other way. So, if you get a notification on your iOS device telling you that you have new mail, and you read that mail on a different device (like, say, your Mac), the iOS device won’t know you have read it (and won’t adjust the number in the badge on the Mail app icon) until you open the Mail app and establish the IMAP connection between your iOS device and the remote mail server.

If that seems slightly awkward, it is. As Joe Kissell has pointed out with regard to alerts in “An Alarming Abundance of Alerts” (13 May 2012) and as has become problematic with Messages, Apple needs to do something to reduce the duplication of events that currently happen simultaneously on multiple devices. In an ideal world, these notification systems would understand which device is in use and cascade appropriately to other devices as necessary.

So there you have it: Apple is using two completely different notification systems that produce very similar results. And all so you can get your singing cat emails as soon as possible, no matter what device you are using!

Check out the Take Control ebooks that expand on the topic in this article:

Join author Joe Kissell as he explains Mail's new Mountain Lion features—such as VIPs and Notifications—and walks you through the basics of sending and receiving email. Special topics include attachment handling, searching, rules and smart mailboxes, integration with Gmail and iCloud, digital signatures and encryption, and more.


PDFpen and PDFpenPro 9 add 100+ enhancements to improve your PDF
editing experience, with annotations, Tables of Contents, and more
export options. For PDF reviewing, editing, signing, redacting and
exporting, PDFpen has you covered. <>

Comments about How Apple Mail May Be Anything but IDLE when Pushing Email
(Comments are closed.)

Beau Giles  2012-10-22 19:56
When connected to a Mac OS X 10.7 or 10.8 server for Mail, Contacts and Calendars (IMAP/CalDAV/CardDAV), iOS devices will use APNS for new mail, while OSX will still use IMAP.

However, CardDAV/CalDAV is APNS on both iOS and OSX.
Matt Neuburg  An apple icon for a TidBITS Staffer 2012-10-22 23:18
Actually my big problem is that I want NOT to get emails as soon as possible. I want to fetch them manually only. I set my Mail (on the desktop) to fetch manually, but mail kept arriving automatically. Then I discovered the IDLE setting! For me, the takeaway message turned out to be: turn IDLE *off*!
Nicholas Barnard  2012-10-27 01:17
Perhaps this is a limitation of Apple's push email notification, but Apps can set their badge icon to any value per Table 3-1 of Local and Push Notification Programming Guide:

I would think email providers could push this information as well...