TidBITS: : FlippedBITS

| Comments (8)

Introducing Our New FlippedBITS Column

Joe’s new ongoing column, FlippedBITS, promises to set readers straight on a wide variety of confusing technical topics.Show full article

Today I’d like to introduce a new series of articles we’re calling FlippedBITS. Our premise is that technology breeds misconceptions, and far too often, due to a lack of information, we develop mental models of the way things work that are plausible — but wrong. Those mistaken ideas, in turn, make it more difficult to solve everyday problems and can lead us to waste time, effort, and money. In each installment of FlippedBITS, I’ll examine one or more of these misconceptions and do my best to set the record straight.

My 2-year-old son is still learning the basics of how the world works, and it’s fascinating to watch his understanding of technology evolve. Just a few months ago, he’d pick up a remote control and try to talk on it like a telephone. In his mind, any oblong plastic box with buttons on one side must be a telephone — it’s easy to see how he might make that mistake. (And, in all fairness, we had a telephone and a remote control that looked mighty similar to each other.) Now he understands that a remote control is different — it’s the thing we point at the TV to make it play his favorite shows. But then he became confused and frustrated when we handed him a toy remote control that only made noise when he pressed the buttons; he couldn’t understand why the TV wouldn’t respond.

We expect kids to make these kinds of errors, and we laugh knowingly as we watch how they try to put something into one logical category, notice that it doesn’t quite fit, and then try another. This is all part of growing up. But in fact, we never stop trying to make sense of the world. We encounter a new thing we don’t entirely understand, and we automatically — perhaps unconsciously — start trying to construct a mental model of what must be happening behind the scenes. These models not only help us explain what we’re seeing, they help us predict how things will work in the future. It’s just that sometimes, through no fault of our own, we guess wrong.

For example, I remember the first time I heard of this newfangled device called a “laser printer.” I was a freshman in college. I got that paper went in blank and that, due to something involving a laser, it came out with crisp black text. But the initial idea I had about how this worked was that the laser was somehow burning the letters directly on the paper, because after all, burning is what lasers do. Later, when I found out that laser printers used a black powder called toner, I had to revise my theory. Maybe the paper was covered with toner before the laser zapped it, and the heat from the laser caused the toner to melt in spots and stick to the paper. That turned out to be wildly wrong too, of course. I had no idea at the time that a laser beam could reverse an electrostatic charge that otherwise causes toner to stick to a drum, that when paper rolled along that drum, it picked up the remaining toner (again, due to electrostatic attraction), and that a combination of heat and pressure then fused the toner to the paper. My theories had seemed reasonable based on the available information; they even correctly predicted that the paper would come out of the printer warm. But my mental model didn’t happen to reflect reality.

Misconstruing how a laser printer worked had no negative consequences for me. But sometimes erroneous mental models lead to serious problems. If your mental model of how a car’s air bags work is that they offer complete protection in any sort of crash, that could lead you not to bother wearing a seatbelt, which might prove deadly if, for example, your car flipped over.

I get lots of technical questions from people who have read my books and articles or heard me speak somewhere. A fair percentage of the time, the questions are phrased in a way that shows they come from a mistaken mental model. For example, in the last several weeks at least three different people have asked approximately the same question: “Since FileVault encrypts all the files on my disk, doesn’t that mean when I copy a file to another disk, it’s still encrypted?” No! It absolutely does not mean that. (I’ll explain why in a future FlippedBITS article.) But I can easily see how someone might draw such a conclusion — and misunderstanding something like that could cause someone to make an unsafe decision about how to handle sensitive files.

I’ll admit it: When I hear questions like this, I sometimes have to fight the temptation to roll my eyes and say, “What an idiotic idea!” But I’ve had (and probably still have) plenty of idiotic misconceptions myself. Not understanding something, or having a faulty conception of how it works, doesn’t make you stupid. It only means you haven’t yet acquired enough information about something. I’ll do my best to supply those missing facts and put us all on the right path. I already have a healthy list of prospective FlippedBITS topics, but if there’s a topic you think might be an appropriate fit, please feel free to suggest it.

So, whence the name “FlippedBITS”? Apart from the fact that we at TidBITS like to append “BITS” to everything, we thought that flipped bits would be an apt description of the kind of error we’re trying to correct. Computers, as you know, store information as a series of ones and zeroes. Every “slot” that can hold either a one or a zero is a bit. If the bit’s value is zero and you flip it, it becomes a one. Flip it again, it’s back to zero. Sometimes bits get flipped inadvertently due to programming errors, mechanical failures, media degradation, cosmic rays — really! — or other random occurrences. And unfortunately, a single flipped bit — a one where a zero should be, or vice-versa — can mean the difference between a program succeeding and failing. After all, 01110111 is “w” in binary, but 01110011 — almost the same, but with one bit flipped — is “s.” Sometimes a change as small as a single flipped bit can spell the difference between a “win” and a “sin”!

With that, allow me to direct you to the first article in what I hope will be a long and helpful series: “FlippedBITS: Booting Your Mac from a Duplicate” (13 March 2013)

Article 2 of 5 in series

Problem Solving | 13 Mar 2013 | Listen

| Comments (17)

FlippedBITS: Booting Your Mac from a Duplicate

Bootable duplicates are an important part of a complete backup strategy. But when you have to boot a Mac from such a disk — even if only temporarily — things may not always behave as you expect. Joe Kissell sheds light on some of the common points of pain and confusion when starting a Mac from a duplicate.Show full article

In this first installment of FlippedBITS, I want to look at what happens when you boot your Mac from a duplicate (or “clone”) of your startup disk. In doing so, I hope to clear up several common points of confusion, particularly regarding ongoing backups and syncing other types of data.

For years I’ve recommended a three-pronged backup strategy consisting of versioned backups (such as those produced by Time Machine or CrashPlan), bootable duplicates (complete copies of everything on your startup disk, stored on an external drive), and offsite data storage (either in the cloud or by rotating physical media to other locations). Together, this combination can protect your data against almost any disaster, while making recovery as painless as possible. (For complete details about my suggested strategy, including the steps to create a bootable duplicate, see “Take Control of Backing Up Your Mac.”)

It’s simple enough to make a duplicate using a tool such as Carbon Copy Cloner or SuperDuper; having done that, you can use the duplicate to boot your Mac either by selecting it in the Startup Disk pane of System Preferences or by holding down Option while restarting and selecting the volume containing the duplicate. Doing so enables you to get back to work immediately if anything goes wrong with your startup disk; running from the duplicate makes your Mac behave as though nothing had happened. So far, so good.

But, based on numerous email exchanges I’ve had with people who have read my various books and articles about backups, what happens once you’ve booted from the duplicate is sometimes unclear. Some people expect a duplicate to behave entirely like the original in every situation, which turns out not to be quite true. Others worry that the duplicate will cause all sorts of problems because it’s not enough like the original, resulting in extra, unnecessary steps. To untangle things, let’s start with the least ambiguous situation.

Swapping Your Startup Drive -- Suppose your Mac’s internal hard drive dies completely, so you remove it from your Mac and replace it with the drive on which you’d previously stored your bootable duplicate. Your Mac doesn’t care about the fact that the new hard drive may have a different brand, capacity, or speed. All it knows is: here’s a disk with exactly the same data in exactly the same place. It is, for all practical purposes, the same disk. You can carry on as if nothing happened; everything, including your backups, should simply pick up where they left off, which is almost certainly what you want.

Now, there is one little catch. What if there was a time lag between when you made the duplicate and when you started using it? And what if, during that lag, you created or edited data on your startup disk — and backed it up to a destination other than where your bootable duplicate is stored? Now you have a startup disk that’s somewhat out of date, and to make it current, you’ll have to go to that other backup to locate and restore any important files that were changed after you last updated the duplicate. This, I’m sorry to say, is often an entirely manual procedure. In many backup apps (and again, I’m thinking especially of Time Machine and CrashPlan), there’s no simple way to say, “Show me all and only the files that changed and were backed up after time x.” You may have to dig through folders one by one in your backup archive to find these files. You could ask the software to restore everything backed up after a certain time, overwriting any existing files, but that would take quite a while. It’s a pity that many otherwise highly competent backup apps don’t account for this usage case.

Booting from an External Drive -- Although the situation I just described is the least ambiguous, it’s also relatively infrequent. The most likely scenario, which gets more confusing, is when your regular startup disk is still present and functional but you hook up your duplicate and boot from it temporarily. Perhaps you’re doing this to verify that the duplicate works (in which case you may be running from the duplicate for only a few minutes), or perhaps your startup disk is having problems and you want to run a disk repair utility while carrying on with your regular work.

Either way, let me start by saying what isn’t a problem in this scenario. For one thing, it doesn’t matter if your startup disk is external. Apart from speed differences, your Mac should behave identically whether the startup disk is connected via an internal SATA port, USB, FireWire, Thunderbolt, or whatever. So, don’t let that trouble you in the least.

For another thing, it doesn’t matter if data happens to sync with the cloud while you’re booted from the duplicate. For example, if you use iCloud, your calendars, contacts, bookmarks, and so on will sync in the background. You need not worry that the outdated data already on your duplicate will somehow overwrite what’s in the cloud; on the contrary, the cloud has the “master” copy (sometimes called the “truth”), so it will bring the data on your duplicate disk up to date. Similarly, if you use Dropbox or another cloud-based file storage service, it will bring your disk up to date with the latest truth from the cloud, and it’s unnecessary for you to fret over that in the slightest.

(You do need to fret if you use POP for email, or if you have any rules or filters that file incoming email from IMAP or Exchange servers into local mailboxes. That could get messy, with the duplicate being changed in ways that can’t easily be applied back to the original, so if in doubt, refrain from checking your email at all while booted from the duplicate.)

You need not even worry about aliases — usually. When you create an alias to an item that’s on your current boot drive (that includes items in the Finder’s sidebar and in your Login Items list), Mac OS X creates relative links. That means if you make a duplicate, boot from the duplicate, and open an alias, the item that opens is the one on the duplicate, not on the original disk. (And that’s probably what you want.) That’s not to say your Mac might not have a script, a symbolic link you created in Terminal, or some other pointer that references a file or folder by disk name, and if it does, you could accidentally open the wrong copy of a file or application, or save data to the wrong disk. (If you’re concerned and want to be absolutely sure which item you’re opening, navigate manually from the top level of your disk when booted from a duplicate.)

However, at least one significant thing is most likely different, even though you may not notice it. If your duplicate had the same name as your startup disk (presumably the most common case), something slightly weird can occasionally happen. Mac OS X won’t let two mounted volumes have exactly the same name. In the Finder, they may look like they have the same name, but if you already have a volume called “Macintosh HD” mounted and then you mount a second one, behind the scenes, the second one gets a different working name (in this case, “Macintosh HD 1”). That’s because many things that happen on your Mac depend on being able to locate a disk by name, and if there were any ambiguity, a file might get put in the wrong place.

Ordinarily, this on-the-fly renaming just works, but it’s not foolproof. What if, during the time you have both “Macintosh HD” and “Macintosh HD 1” mounted, another user on your network connects to your Mac and copies a file to what is now “Macintosh HD” — your duplicate? You might not notice it, and when you switch back to your usual startup disk, the file would be missing. Similar things can happen with file-synchronization apps, software downloads, and other operations. Furthermore, sometimes Mac OS X gets confused and doesn’t correctly update its behind-the-scenes list of volume names, so you could, for example, encounter a situation in which “Macintosh HD” is a volume you mounted after “Macintosh HD 1.”

On the other hand, renaming your duplicate doesn’t necessarily solve these problems. If your normal startup disk is named “Cindy” but you’ve booted temporarily from “Kate,” you may avoid mismatched name issues right now — but later, if you have to start using “Kate” permanently, apps and users that were still trying to save data to “Cindy” could get confused. All in all, I think you’ll get the best results if your duplicate has the same name as the original disk, but you should follow a few steps (just ahead) to avoid problems while running from the duplicate.

Meanwhile, you may have to think about another subtle background process: backups! After all, your backup software is probably configured to run automatically — perhaps once an hour (like Time Machine) or continuously (like CrashPlan). Backups usually don’t begin immediately when you boot your Mac, but they could easily kick in within 10 or 15 minutes. What if you’re planning to run your Mac from the duplicate for longer than that, but not permanently? Should you let your backups proceed — meaning they’ll be backing up the duplicate — or should you turn them off?

In general, there’s no harm, and considerable benefit, in letting backups run. Your backup software should act as though your duplicate is your regular startup disk and keep copying files to its normal destination as though you had restarted normally. That’s probably what you want, because if you create or modify a file while running from the duplicate, it can then be backed up. The problem is, in fact, with the opposite case: what if you modify a file but it isn’t backed up (perhaps because the time for the next periodic backup run hasn’t rolled around yet)? When you switch back to your regular startup disk, the file won’t be there (or won’t be current), and it won’t be in your backups either. It will still be on your duplicate — but only if you think to check there before you update your duplicate the next time; doing so will probably delete the new file because it’s not on your startup disk.

Taking all this into account, here are my recommendations for what to do when you must boot from a duplicate for a short period of time:

If you can avoid creating, modifying, or downloading files, do. If you can’t, make sure they’re synced to the cloud, copied back to your regular startup disk, backed up, or otherwise made available to yourself when you return to your usual disk later.
Let regular, versioned backups (such as Time Machine and CrashPlan) run normally. But if (per the last point) you can’t avoid creating files, make sure your backup software has in fact backed them up before switching back to your customary disk.
Turn off any scheduled updates to your bootable duplicates. The last thing you want is for your duplicate disk to clone itself back onto your main startup disk while you’re testing it, or for the software to freak out in trying to clone the original over the now-booted duplicate.
Avoid letting other users connect to your Mac, especially to upload files.

Booting a Different Mac -- There’s one more scenario to consider: booting one Mac with a duplicate of another Mac’s startup disk. For example, imagine that you created a duplicate of your iMac’s startup disk and then you had to take your iMac to the shop for repairs. In the meantime, you hook up your duplicate to your MacBook Pro so it can “pretend” to be the iMac. As long as the MacBook Pro supports the same version of Mac OS X your iMac was running, this arrangement should work fine — with, as you might have guessed, a couple of qualifications.

First, although this situation is ostensibly similar to the last one — booting a Mac temporarily from another drive — the time frame involved could be longer (days or weeks instead of hours). So, it’s impractical to avoid modifying files, checking your email, and the like. Therefore, I recommend that you use the duplicate disk on your MacBook Pro as you normally would use the internal drive on your iMac, and then — once your regular iMac is back in service — hook up the external duplicate and reverse the cloning process (copy everything from the external drive back to the iMac’s internal drive it started on).

Second, while your MacBook Pro is running from the duplicate, it will by almost every measure appear to be the iMac. It’ll even use the iMac’s name for file sharing, screen sharing, and the like. However, every Mac has several unique identifiers, including a serial number, a UUID (universally unique identifier), and a MAC (media access control) address for each network interface. Some pieces of software check one or more of these unique identifiers to verify that they’re still running on the same Mac on which they were authorized or licensed.

The most common example is iTunes. If you authorize your iMac to use your iTunes account and then start up your MacBook Pro with a duplicate of the iMac’s disk, that doesn’t mean the MacBook Pro is automatically authorized. You must authorize it manually, if you haven’t done so previously (in iTunes, choose Store > Authorize This Computer). But, if you’ve already run out of authorizations — Apple limits you to five — you may be out of luck unless you can deauthorize one of your other computers or reset all your authorizations, the latter being something that Apple allows you to do only once per year. Other software may make you jump through similar (or worse) hoops.

Don’t Sweat It -- As convoluted as this all may sound, booting your Mac from a duplicate is usually a simple and problem-free operation. But it never hurts to have a better grip on what’s going on behind the scenes, just in case. In particular, think about whether you’re going to be using the duplicate long enough to add or modify files on it manually. If not, there’s no harm in simply switching back to the original. But if you are going to be working from the duplicate for any significant period of time, you’ll need to clone it back to the original when you’re done.

A final note: keep those duplicates up to date. It would be overkill to update them every hour, but once or twice a day is not a bad idea. Remember, the longer the gap between the last time you updated your duplicate and when you discovered you needed to boot from it, the greater the chance of missing or outdated files that you may have to laboriously restore.

Article 3 of 5 in series

Safe Computing | 20 Mar 2013 | Listen

| Comments (23)

FlippedBITS: Four Password Myths

Are your passwords strong enough to resist an automated attack? If you believe any of several common password myths, they may not be. In this installment of FlippedBITS, Joe Kissell examines a few of the most dangerous myths about password security and explains smarter and safer practices.Show full article

In the course of writing “Take Control of Your Passwords,” I came across — and attempted to debunk — quite a few myths involving password security. Of course, I encourage you to buy the book to read about password problems and my recommended solutions in detail, but for this installment of FlippedBITS, I want to focus on four extremely common misconceptions about passwords, all of which can lead to dangerous behavior.

1: Nine Is Enough -- I want to begin with a myth I propagated myself in my now-obsolete 2006 book “Take Control of Passwords in Mac OS X.” Although what I said in that book was reasonable based on the available data at the time, I grossly underestimated the rate of technological progress. So, I hereby retract and apologize for a particular piece of advice I gave back then: I said that if you chose a random 9-character password consisting of upper- and lowercase letters, digits, and punctuation, you’d be effectively safe from any attack, because it would take centuries, on average, for even a supercomputer to crack such a password by brute force.

Well, it turns out that I was off by a few orders of magnitude. Today, with off-the-shelf hardware and freely available cracking software, a nine-character password can be broken in a maximum of five and a half hours (that’s maximum, not minimum!). If your password contains nine or fewer characters, regardless of how random it may be, it’s about as unsafe as a Wi-Fi connection protected with WEP (which is to say, safe against only the most casual snooping).

If nine characters are too few these days, how long should a password be? I wish I could give you a straight answer, but the truth is “it depends.” For example, I could claim, with some justification, that a random 14-character password is effectively safe from brute-force attacks given today’s technology. But I’d have to qualify that in a few different ways.

First, I have no idea what tomorrow’s technology will look like. Maybe a few years from now, someone will develop a quantum computer that can crack any 14-character password in the blink of an eye. I don’t expect that to happen so soon, but I’d be foolish to bet against it.

Second, not all encryption techniques are equally secure. A password that’s protected with a weak encryption algorithm might be crackable in seconds, whereas the same password, encrypted with a better method, could thwart a brute-force attack for years. Related to this is that some password security systems put additional barriers in place to slow down the rate at which passwords can be guessed. Although these aren’t foolproof (as I discuss in a moment), they can, in certain situations, give a simple password much higher effective strength.

Third, length isn’t the only factor that affects a password’s strength. As illustrated brilliantly in the xkcd comic Password Strength, even a password consisting entirely of lowercase English words (such as correct horse battery staple) can be just as strong as a shorter but more random password with a mixed character set. That’s because a password’s entropy (a mathematical approximation of how hard the password is to guess) can come from length, character set complexity, randomness, or any combination of these. Higher-entropy passwords are more resistant to automated attacks, but there’s more than one path to entropy. (If you’d like to test a given password’s entropy, there are many online tools that let you do so. I quite like the zxcvbn tool for this purpose.)

We can take some comfort in the fact that each additional character in a password increases its strength exponentially. So, if we were to restrict ourselves to just 26 lowercase letters, a 10-character password wouldn’t merely be 10 percent better than a 9-character password — it would be 26 times better! There are over 5 trillion possible passwords consisting of nine lowercase letters (26^9), but make it ten letters (26^10), and there are more than 141 trillion possibilities. That means a system that can crack a 9-character random password in 5.5 hours could take over 500 hours to crack a 10-character random password — a huge difference.

Even so, 500 hours is too little for my comfort. You could make that more than 500 years by choosing a 12-character password, which certainly seems safe enough for all practical purposes. But then, that’s what I thought about 9-character passwords seven years ago. So, when I suggest 14 as a safer number, I’m building in enough of a buffer to account for a few years of technological development, not in any way saying that such a password will in fact be uncrackable for the over 4,000 millennia it would take at today’s rate.

2: Old Tricks from Old Dogs -- I’ve encountered quite a few people — including some major names in the Mac world you’d recognize — who have developed mnemonic techniques for creating and remembering passwords that they imagine to be quite strong. Although specifics vary, there tends to be a consistent element or easily constructed pattern in each password, along with some site-specific portion. For example, maybe I use zombieGooCats for Google and zombieAppCats for Apple. (In reality, most people I know who do this sort of thing have far more sophisticated techniques, but you get the general idea.)

I myself once (cough) advocated such an approach, but I’ve since seen the light. The problem with all such tricks — and that also goes for “leet” or “1337” (replacing letters with similar-looking numbers), using patterns of keys on a keyboard, and so on — is that no matter how clever you think you are, hackers and their advanced cracking algorithms are smarter. These tools can test a vast number of subtle patterns that few humans would notice, which means even a fairly long, fairly random-looking password might in fact be quite easily guessable. Because remember, we’re not worried so much about humans guessing your password but about machines guessing it, and machines are likely to test lower-entropy passwords — especially those based on common mnemonic techniques — long before higher-entropy passwords. (And, if you use the same technique to construct all your passwords from patterns, an attacker who learns one or more of your passwords has an even bigger leg up in guessing the rest.)

More to the point, any technique that relies on your brain for creating and remembering all your passwords is, in my opinion, a waste of mental effort that could be put toward more useful pursuits, such as thinking up bad puns. We have computers and iPads and iPhones and other devices to do this sort of tedious work for us, and they’re much better at it than we are. Let a password manager such as 1Password or LastPass generate, remember, and enter passwords for you, and then you can make them as long and random as you like — it’s no more effort for an app to make a 32-character password than a 10-character one. Sure, you’ll still need to remember a few passwords, but if you’re doing it right, it’s only a few. (I have only 5 passwords memorized, out of more than 600.)

3: One Password to Rule Them All -- Speaking of password managers, these tools make it easy to create a unique random password for every single site and service that uses passwords, and I recommend doing so. I can’t emphasize strongly enough what a bad idea it is to use the same password in more than one place — even if it’s a great password. The fact that reusing passwords is entirely unnecessary if you rely on an automated tool makes it that much more egregious an offense.

Why is it so bad to reuse passwords? Well, it seems like every week or so, there’s another news report about some big company experiencing a security breach of some sort in which thousands or even millions of passwords are lost, stolen, leaked, or hacked. This happened recently to Evernote; before that, a long list of other companies had passwords compromised – Facebook, LinkedIn, Twitter, and more. You can bet this trend will continue.

Now, if someone hacks Amazon.com’s servers and gets your password, that’s bad news, no question about it. But if all your passwords are unique, at least the damage will be limited to that one account. On the other hand, if you use the same password for iCloud, PayPal, Twitter, Gmail, and so forth, you run the very real risk that the attacker may try your password at all those other sites, too, doing considerably more damage.

I’m saying: using unique passwords — even strong unique passwords — doesn’t guarantee security. But it does enable you to contain the damage if your password for any one site is compromised. The people most likely to be harmed by password breaches are those who are oblivious to the problem of password reuse. Don’t be one of them!

4: Online vs. Offline Attacks -- Earlier, I mentioned that some sites and services put barriers in place to slow down or derail automated attacks. For example, if you mistype your password once, you might get one or several additional chances to enter it — but with increasing time delays between guesses. And if you enter it incorrectly several times in a row, you might be locked out entirely for a period of time, or until you take some independent action to confirm your identity. The whole point of these barriers is to prevent an automated system from trying many passwords per second until it breaks into your account.

While it’s an excellent idea for developers to employ such barriers, they aren’t as strong as they might appear. That’s because most successful attacks don’t go through the front door, as it were. The real danger comes when, due to a leak or security breach of some kind, someone gets hold of an encrypted file or database that holds all the passwords for a site. With the file in hand, they can perform what’s known as an “offline” attack — they hammer on the raw file with automated tools that check billions of possible passwords per second. Because they’ve entirely circumvented the security measures that slow down guessing, they can potentially decrypt massive numbers of passwords in a short period of time. (I’m simplifying the story here. Smart developers can also use a combination of techniques — the key terms to look for are “salting” and “hashing” — to frustrate offline attacks, but all too often, a programming error or infelicitous security choice leaves gaping holes that hackers can exploit.)

So, don’t assume you can use a short, simple password because you can’t see any way an attacker could try billions of passwords a second. You’d be surprised what someone can do, particularly given physical access to the computer where the password is stored. Your best defense is to use high-entropy passwords (which take longer to guess) and make sure each one is unique.

Don’t Worry, Be Happy If I’ve increased your anxiety about passwords by telling you what’s wrong with techniques you depend on, I’m sorry. Well, only a little bit sorry, because I want you to have just enough discomfort that you take action to improve your password security and reduce the chance that bad things could happen to your digital life. For extensive details on passwords, including further threats and risks you might face — and my stress-free, three-point strategy for password security — please pick up a copy of “Take Control of Your Passwords.”

Article 4 of 5 in series

Networking | 11 Apr 2013 | Listen

| Comments (23)

FlippedBITS: IMAP Misconceptions

One of the most popular methods for receiving email is also the source of numerous misunderstandings. Joe Kissell explains why IMAP may be a more effective and useful protocol than POP, and addresses common sources of confusion.Show full article

In today’s installment of FlippedBITS, I want to examine a handful of common misconceptions about IMAP, a familiar protocol for retrieving email from a server. IMAP stands for… well, thereby hangs the first tale. IMAP’s inventor, Mark Crispin (who, sadly, died in December 2012), called the first version of his creation Interim Mail Access Protocol. Versions 2, 3, and 2bis were referred to as Interactive Mail Access Protocol, and version 4 — what’s in use today — is officially Internet Message Access Protocol. Although many Web sites claim that the acronym once stood for Internet Mail Access Protocol, I have found no credible references to back up that claim.

By whatever name, IMAP has always been a means by which email clients can talk to email servers. That puts it in the company of POP (Post Office Protocol) and Microsoft’s MAPI (Message Application Programming Interface). Almost every modern email client — including Apple Mail, Thunderbird, Microsoft Outlook, and dozens of others — supports IMAP as a means of retrieving email. You may very well have been using it for years without even knowing it — iCloud and its predecessors MobileMe, .Mac, and iTools have always defaulted to IMAP for email access.

New Kid on the Block -- The first thing I want to clear up is the persistent notion that IMAP is some sort of newfangled email system, a regular Johnny-come-lately compared to the ancient and revered POP method. Yes, POP has been around quite a while — it was invented in 1984. IMAP came along in 1986. (For perspective, Apple’s Macintosh System Software 5 — the first one to include MultiFinder — was released in 1988.) Both protocols subsequently underwent numerous revisions, but in any case, it’s a bit silly to consider POP “traditional” and IMAP “new.”

Now, it’s true that in the early days, email clients and servers alike were more likely to support POP than IMAP (and even today, IMAP support isn’t universal). So, many of us who have been using the Internet for a long time became accustomed to POP — and a surprising number of people still use POP, often out of habit more than necessity. (I’ll return later to whether that’s a good idea.) But IMAP has been a viable option for decades.

Are You Being Served? -- The usual way people explain the difference between POP and IMAP is to say that with POP, all messages are downloaded from the server to your email client, whereas with IMAP, messages are stored on the server. That’s sort of true-ish, but it’s unfairly misleading in both cases. With POP, you can leave messages on the server if you want to, and with IMAP, you can download all your messages and store them locally. The simplified “IMAP-means-stored-on-the-server” explanation has led countless people to assume that you can use IMAP only when you have an active Internet connection. But that isn’t the case. For as long as I’ve been using IMAP, I’ve maintained local copies of every single message in my accounts, and have never had trouble reading, searching, filing, or otherwise managing my messages when offline.

The best way to think about IMAP is that the server holds the master copy of every message. Whenever an IMAP client connects to the server, it can synchronize changes bidirectionally — for example, new messages in the Inbox download to the client; changes made in the client while it was offline upload to the server, updating the master records. But the exact behavior is determined by the design of the client and settings chosen by each user. By default, Apple Mail (like most other modern email clients) keeps all your messages in sync between client and server. But if you prefer, you can configure your client not to cache messages for offline viewing, to cache only some messages, or to cache the text of messages but not any attachments.

I should add that even though the server stores all your email messages, this in no way prevents you from deleting messages. Although, again, the exact behavior varies according to your client and your settings, when you delete a message locally, your client normally tells the server to delete its copy too.

I’ll File Away -- Another prominent difference between POP and IMAP is that IMAP lets you define mailboxes (that is, folders for email messages) that are stored on the server and (in most cases) synced with your email client. In general, the effect is that no matter which IMAP client you use, on which platform, it will always reflect the same set of mailboxes with the same contents; you’ll never have to worry that you might have filed a certain message on the wrong computer.

With POP, there’s no such thing as server-based mailboxes, just an Inbox, so any filing you do must, by definition, be done in the client. However, with IMAP, even though server-side mailboxes are supported (and quite handy), if you prefer to store some or all of your messages in local mailboxes, nothing’s stopping you. In fact, if your IMAP provider imposes a storage quota, you may want to move messages from server-based mailboxes into local mailboxes from time to time in order to free up space on the server.

The Same Thing, Only Different -- I’ve heard it said that if you configure your POP client to keep all messages on the server — that is, not to delete them after they’re downloaded — then POP becomes so similar to IMAP that you probably won’t be able to tell the difference. But that’s very far from the truth.

Apart from the lack of server-based mailboxes (which, of course, you’re not obligated to use in IMAP), leaving messages on a POP server is much different from leaving messages on an IMAP server. Crucially, IMAP servers keep track of which messages you’ve read, replied to, and forwarded. So, suppose I connect to a POP account that has 15 messages in the Inbox. I download and read them, but leave them on the server. Now I go to a different client or computer and connect to the same POP account. The same 15 messages will download again (along with any that have arrived in the meantime), with no indication of which ones I’ve already read. By contrast, if I do the same thing with a pair of IMAP clients, each one will show me the same thing — these messages have been read, those haven’t; this one has been replied to; that one was forwarded; and so on. This makes it much easier to switch among clients — something that becomes increasingly important as more of us have not only multiple computers but also smartphones, tablets, and other Internet-connected gadgets.

Speaking of multiple clients, you should be aware that POP permits only one connection at a time per account, while IMAP has no such limit. So, although your three Macs, two iPads, Windows PC, and iPhone can all maintain live connections to an IMAP account, they’re forced to take turns with a POP account.

All of a Piece -- But now, let me turn that around and address another misconception, that all IMAP servers are created (more or less) equal. Would that it were so, but no. IMAP servers are as frustratingly different from each other as clients are. It all comes down to three words: specification, implementation, and configuration.

The IMAP specification, as I mentioned earlier, has undergone a number of revisions. In addition, it supports the use of optional extensions to provide extra features. When it comes time to implement the specification, one developer might use an older version of the spec, or interpret part of it in an idiosyncratic way, or choose to include or omit various extensions for one reason or another — while the next developer might make entirely different choices. And some developers might decide that the standard IMAP approach doesn’t meet their needs, so they leave things out, slap extra things on, and rejigger other things so they work in surprising ways. (This happens more often than I’d like to admit, although Gmail’s flavor of IMAP is arguably the least IMAP-like, which is not surprising since it was an afterthought rather than a part of the original Gmail design.) Moreover, IMAP servers have a variety of settings a system administrator can configure, just as IMAP clients have user-configurable preferences. All these variables can make any IMAP client/server pair behave much differently from any other.

I can’t tell you how many times I’ve had to say things like, “Yes, your IMAP server supports subscribing to specific mailboxes, and so does Outlook, but Apple Mail doesn’t,” or “Apple Mail supports IMAP IDLE (see “How Apple Mail May Be Anything but IDLE when Pushing Email,” 22 October 2012) but your IMAP server doesn’t,” or “Gmail’s idea of archiving bears only the remotest resemblance to Apple’s idea of archiving.” One especially troublesome area is the way various IMAP servers and clients handle deleting messages — a messy topic I address somewhat in my books about Apple Mail (“Take Control of Apple Mail in Mountain Lion” and “Take Control of Mail on the iPad, iPhone, and iPod touch”) but won’t delve into further here.

POP on over to IMAP -- Notwithstanding the several quirks and annoyances of certain IMAP implementations, my fondness for IMAP is right up there with my fondness for chocolate. (That’s way up there, in case you were wondering.) Let me summarize the advantages of IMAP over POP:

The server keeps a master copy of all your data (including mailboxes and message metadata such as read or replied). So you’ll see the same thing with any client on any platform.
You can connect to an IMAP account from multiple clients at the same time.
If your client supports it, you can have it download only message headers, with full message bodies on demand.
You can ask your client to search for messages on the server, even if they haven’t been downloaded. (The iOS version of Mail supports this, but the Mac version doesn’t.)

The oft-heard objection to IMAP that it takes away one’s control is a myth. As long as you have your client configured to cache a local copy of all messages and to delete messages on the server when you delete them locally, you maintain just as much control over your email as you do with POP. Most of the old assumptions that led users to favor POP — such as the expectation that a person will use a single computer for email most of the time, and the belief that online storage is expensive — are no longer valid in today’s world.

Are there still legitimate reasons to use POP? Sure. For one thing, it’s less chatty than IMAP, so it tends to be better in low-bandwidth situations, especially when lots of users are connecting to an underpowered server. (Having said that, mobile IMAP clients typically manage to do a great job even over slow cellular connections, but that assumes optimization of both client and server for that purpose.) Also, most providers cap each user’s IMAP storage quota, so if you have vast amounts of stored email, you may be forced to offload some of it to local mailboxes; by contrast, POP normally holds onto messages only until the user picks them up, so its storage requirements tend to be lower. And, if you’re concerned that your email provider can’t be trusted or is vulnerable to hacking, you might prefer not to keep unencrypted email on a server any longer than necessary. Finally, not all email providers support IMAP. But that leads me to my final point.

Stuck in the Past -- I’ve heard from a number of people who tell me they’d like to use IMAP, but they can’t, because their ISP doesn’t support it — or charges extra for it. So, two things here.

First, even if your ISP doesn’t offer IMAP for accounts on its own email server, that in no way prevents you from using another IMAP provider. When an ISP says they charge extra for IMAP, that means they charge extra to use their IMAP server, not any IMAP server. You can go right ahead and use iCloud, Yahoo Mail, AOL, Gmail, or any of a hundred other services that offer IMAP access to email — many of which are free.

Second, if your main email address comes directly from your ISP, and that ISP doesn’t support IMAP, you can usually set up an IMAP account with a different provider and then forward mail from your ISP to the new IMAP account. (Exact directions to do this depend on the provider.) That way, anyone with your old address can still reach you, while you get to enjoy the advantages of IMAP.

If you’ve weighed the pros and cons and decided that a switch from POP to IMAP is for you, see if your existing email provider offers an IMAP option — sometimes it’s as simple as flipping a switch on the server side, although you’ll likely have to configure an entirely new account in your email client. For further guidance, I recommend Kirk McElhearn’s Macworld article “How to convert a POP email account to IMAP.”

Article 5 of 5 in series

Problem Solving | 02 May 2013 | Listen

| Comments (26)

FlippedBITS: Java, JavaScript, and You

Serious security threats involving Java have been a recurring theme for Mac users in recent months. But what exactly is Java, and what’s the problem? What’s this Java Runtime you hear about from time to time? And is JavaScript the same thing? Grab a cup of coffee and join Joe Kissell for the story of Java in this installment of FlippedBITS.Show full article