Spotlight Strikes Back: In Leopard, It Works Great
In earlier articles, we’ve talked about some of the great new features of Leopard that might make an upgrade worthwhile. I wrote an article about Spaces, Glenn Fleishman explained how File Sharing is light years better than it used to be, and Joe Kissell gave us the low-down on Time Machine. (The best way to reference that coverage is from our “Leopard Arrives” series.) In this article, I want to tell you about what I think is the last big piece of the Leopard improvement puzzle – the all-new, all-singing, all-dancing Spotlight.
In order to explain why Spotlight in Leopard is so good, I have to talk briefly about why Spotlight in Tiger was so bad. If you already know that, or if your teeth can’t handle any gnashing, you might want to skip this next section, where I recount a bit of regrettable history.
Tiger Spotlight: The Good, the Bad, and the Ugly — When Spotlight was introduced in Mac OS X 10.4 Tiger, it was touted as a major improvement for users, and it’s not hard to see why. Finding things on your hard disk(s) has always been hard – my mother can’t find a newly created Word document five seconds after she’s saved it – and now that your hard disk is really big and you’ve got lots of files, it’s getting harder. The old-style Finder Find involves searching through the hard disk, file by file and folder by folder, so it’s slow; and besides, it requires that you know, with a fair degree of correctness, the name of the item you’re looking for, which is often exactly what you do not know.
Back in the old System 7 days, on the other hand, a lot of us were crazy about a wonderful utility called ON Location, from ON Technology. It generated an index of the names of your files, so searching for a file by its name was very fast. What’s more, it used third-party translators to look inside your files (regardless of their format), read their content, and index that as well, so you could do a fast search for a file based on some words used inside the file. Well, Spotlight promised to bring that kind of technology to Mac OS X, only even better. ON Location had to build its index, and to keep the index up to date, it had to rebuild it periodically. Spotlight, on the other hand, once its initial index was built, would
always be up to date, because every time you made any change to the hard disk, Spotlight would be notified right then and would modify the index accordingly. Small wonder that Glenn’s article introducing Spotlight to our readers was so hopeful (“Spotlight on Spotlight“, 2005-05-02).
Right from the beginning, however, there was trouble. Some features didn’t work; for example, there was an option to search for invisible files, but no invisible files were ever found. Some areas of the hard drive were excluded from the index, so files in those places couldn’t be found, even by name; this exclusion was hard-coded into Spotlight (it wasn’t a preference the user could access), so there was no way even of learning what the problematic places were. Files of certain types were not found properly; I experienced this particularly with some font files, and Apple confirmed that this was a bug (perhaps caused by the distinction between a file’s visible name and its “display name,” which was sometimes a weird string to which the
user had no access). The indexing would mysteriously stop working, and would have to be restarted using the Terminal command line.
Worst of all, however, was the interface through which you actually performed a search and viewed your found results. There were three such interfaces: the Spotlight menu, the Spotlight window, and the Finder search window.
- The Spotlight menu didn’t act like a real menu, it often froze up as you were typing your search, and it displayed only a limited number of results. To see all the results, you had to open the Spotlight window.
- The Spotlight window was annoying in every conceivable way. It belonged to no application; it just hung there mysteriously on your computer, refusing to come to the front when you cycled through your windows or your applications. Its interface was unlike any other window; if anything, it seemed like something out of a Web browser, or a Windows machine. Results were clumped by default into annoying categories; getting information about found results (such as, “Where is this file?”) required a great deal of clicking; results could not be easily manipulated; and the search could not easily be refined (beyond the simple default refinements listed down the right side of the window).
- The Finder search window had one big advantage: a search could be refined though a Location Bar and multiple Criteria Bars that could be summoned to describe in detail what you wanted to look for. However, you were inconveniently forced to do this even for something as simple and common as searching for a file by name; you could use the Finder search window only to look for files (not, for example, iCal events); and things were still clumped into groups (mysteriously, not the same groups as in the Spotlight window), though you could ask for a flat list. When you did ask for a flat list, the Finder search window became almost downright good: it started acting quite like a normal Finder window, a familiar and effective interface
for working with your results.
The upshot was that none of Apple’s Spotlight search interfaces was very pleasant, and none of them gave you access to anything like the full power of Spotlight as implemented through the “mdfind” command-line syntax. For example, mdfind lets you specify wild cards, case sensitivity, and sophisticated Boolean criteria combinations. That’s why a host of third-party alternative Spotlight interfaces sprang up, including my own NotLight. But even these were restricted in what they could do by the underlying Spotlight indexing technology (for example, NotLight couldn’t find invisible files, because neither could Spotlight); and many users preferred to revive the
pre-Tiger search behavior with a free utility such as EasyFind.
A New Deal — In Leopard, Spotlight is faster, less biased, and far more compliant. Under the hood, the index is both constructed and consulted more quickly, so you spend less time listening to your hard disk thrash and more time looking at search results. Everything within the scope of your permissions is indexed and searchable (or if something isn’t, I’ve yet to hear about it). Searches that are supposed to work (like searching for invisible files, or searching for a file by the name the user believes it has) do work. And the search interface is so good that it might just put third-party interfaces out of business.
The Spotlight window is completely gone. If you want to move quickly and see the top results, you use the Spotlight menu; if you want to see all results, or get some interface assistance in constructing elaborate search criteria, you use the Finder window. Those are your only options. The Finder window is now really close to being a normal Finder window: it comes in all the normal Finder views except Column view (though, unfortunately, in List view you can’t ask for extra columns of information, such as Size), and you can do in it nearly anything you can do elsewhere in the Finder, so you’ll hardly know you’re in a special Spotlight-oriented world. And yet, you are in a special Spotlight-oriented world, as is proven by
the fact that you can search in the Finder search window for things that aren’t files or folders, such as iCal events and Safari history items. (The main difference I’ve noticed so far between what you can search for in the Spotlight menu versus the Finder search window is that only the former lets you look up a word in the built-in Dictionary.) Plus, the Finder search window’s criteria-construction interface lets you say nearly anything you’d be able to say using mdfind in the command line.
So, for the rest of this article I’m going to explain how to construct a search. There are actually two different “languages” for doing this: there’s the textual language of what you type in the search field, which works either in the Spotlight menu or in the search field of a Finder window, and there’s the more gestural, interface-based language of manipulating the Finder search window’s various options.
The Search Term — When you type “tonya” into the Spotlight menu’s search field, that’s a search term. Spotlight interprets this as a request to seek matches in a fairly broad way. Capitalization is ignored, so a document containing “Tonya” will match. Diacritical markings are ignored too, sort of; a document containing “Tönya” will match, but if your search term had been “Tönya” then the document containing “Tönya” would match but documents containing “Tonya” would not, as if your use of a diacritical in the search term had indicated a kind of diacritical wild card. You’re doing a word-based search, but what you’re searching for is the start of a word; so, you’ll also match a document containing
“tonyatastic”, though not a document containing “retonyafication”. (To specify that you want to match entire words, put “tonya” in quotes; now you won’t match “tonyatastic”. Quotes can also be used to search for exact multi-word phrases.) But the notion of a word includes camel-cased word components, so you’ll also match a document called “HelloTonya”. Oh, and the search is performed over every kind of metadata, so you’ll match documents with “tonya” in their names, in their contents, in their Spotlight comments, and so on.
Two kinds of modification permit to you restrict the search term’s application. First, you can specify the kind of metadata you’re interested in searching. This is done using a colon-based syntax. For example, to find files that have “tonya” in their Spotlight comments in the Finder, but not files with “tonya” in other types of metadata, you’d put “comment:tonya”. The Help documentation gives several other examples of this syntax, some of which are surprisingly powerful. For example, you can ask for files modified on or before a certain date by saying “modified:<=8/10/2007”, or files created in a certain range of dates with “created:8/10/2007-8/12/2007”. The trouble, though, is that as usual Apple spurns the notion of stooping
to provide you with any real documentation: there is no complete conspectus or systematic explanation of the syntax, or even a list of the metadata terms you can specify in this way. (The way I found out about “comment:” in the first example was by trial and error.)
Second, you can combine terms using the Boolean operators AND and OR (in capitals), and modify a term with NOT; a minus sign before a term, with no space, means “and not”. The default operator, supplied if you use multiple words without quotation marks or an intervening Boolean operator, is AND. Thus, on my machine, searching on “tonya tidbits” finds 103 items, those that contain both terms; “tonya OR tidbits” finds 530 items; “tonya -tidbits” finds just 15 items, because it’s so rare on my computer for Tonya to be mentioned without also mentioning TidBITS.
The Finder Search Window — To summon the Finder search window, click Show All in the Spotlight menu after a search, or press Command-Option-Space, or (in the Finder) choose File > Find (Command-F), or just start typing in a Finder window’s search field. You can use the search term syntax I described in the previous section, but you can also use the Location Bar and the Criteria Bars to restrict and specify your search in a more graphical fashion.
The first question to ask yourself is whether you want to restrict the search location to one particular folder. If you do, then you must start by being in that folder in the Finder before starting the search by pressing Command-F or typing in the search field. When the window changes to a Finder search window, the Location Bar will display the name of the folder you started in; click that name to restrict the search to that folder.
Another nice feature of the Location Bar is that it offers an option to restrict the search to the “File Name”, as opposed to the “Contents” – the latter being a misleading term which actually means the default of searching all the metadata at once. These two choices, search by name or search by all metadata, are the two most common forms of search, so it’s very sensible of Apple to provide some simple, up-front interface for choosing between them.
To tweak your search further, click the + button at the right end of the Location Bar. This reveals a Criteria Bar. Here you can choose a metadata type in the leftmost pop-up menu. By default, there are just six sorts of metadata listed here: Kind, Last Opened Date, Last Modified Date, Created Date, Name, and Contents. (Here, “contents” really does mean contents.) When you choose one, other operators, fields, and pop-up menus appropriate to your choice appear. So, with “Contents” the only operator is “contains” and you get a text field for typing some text. With “Name” you get a pop-up menu of five operators: “matches”, “contains”, “begins with”, “ends with”, and “is”. (The difference between “matches” and “is” is that “matches” is
word-based; thus, “tonya” matches a file named “Adam and Tonya” using “matches” but not using “is”.) With “Kind” you get a pop-up of subtypes, and some of those subtypes have subtypes of their own; thus, the “Kind” called “Music” can be “All”, “MP3”, “AAC”, or “Purchased”.
There is also a seventh item in the leftmost pop-up menu of a Criteria Bar: Other. This is where things really start to get good. When you choose Other, you get a dialog listing all the kinds of metadata the Spotlight index knows about. You can just pick one to use it; you can also select a checkbox to specify that that option should appear in the menu from now on, so you don’t have to pass through the Other dialog to access it. I recommend that you immediately check two items that I think you’ll be using quite a lot:
- System files. When set to Include, files are sought even in special locations such as /Library/Caches and ~/Library/Preferences. For example, if you search on “com.apple” you won’t find much, but if you include system files, you’ll find hundreds of preference files.
- Spotlight items. When set to Include, searches are expanded beyond files and folders to include other sorts of entities, such as iCal events, Safari history items, and preference panes.
A huge power user tip: When you summon the Finder search window with Command-Option-Space, or from the Spotlight menu, Spotlight items is set to Include. When you summon the Finder search window with Command-F or by typing in a Finder window’s search field, Spotlight items is not set to Include. This is actually quite brilliant. Spotlight is making a very reasonable distinction and assumption here: if you started in the Finder, you probably just want to look for files and folders, but if you summoned the search window in a more global way, you probably want to look at all kinds of entities. Of course you can always summon a Criteria Bar and change the setting if the initial default isn’t what you intended.
You specify additional criteria by showing and configuring additional Criteria Bars; to do so, just click the + button in any existing Criteria Bar. But here’s the real trick: if you click the + button while holding the Option key, you get a special Boolean Operator Criteria Bar. The pop-up menu here says Any, All, or None (the equivalents of the Boolean OR, AND, and NOT operators), and it applies to the Criteria Bars that are grouped just after the Operator Bar and indented to the right. Such groups can themselves include a Boolean Operator Criteria Bar, and so you can form Boolean expressions of any depth and complexity (the equivalent of using parentheses in a logical expression). The default operation, used if you simply set
multiple criteria without grouping them, is AND (that is, all the criteria must be true at once to get a match).
Conclusions — Spotlight in Leopard is what Spotlight in Tiger should have been but wasn’t. (Don’t get me started on a rant about why Apple has so much trouble getting these things right the first time out.) How good is it? Maybe not quite good enough to put NotLight completely out of business. NotLight will need modification in order to take advantage of some of the new features of Spotlight’s underlying technology, but it has three features that the built-in Spotlight interfaces do not:
- With NotLight, the search is not live, so things don’t keep flashing and bogging down while you’re typing a search term; you type until you’re ready, then do the search.
- The Finder Path Bar is great for determining where a found item is by selecting it, but with NotLight you know where every found item is, without having to select it.
- NotLight lets you choose between case-sensitive and case-insensitive term matching; sometimes that’s actually useful.
Nevertheless, the improvement in Leopard’s Spotlight is very, very dramatic – so dramatic that, whereas, in Tiger, once I’d written NotLight, I never used any of the built-in Spotlight interfaces, but used NotLight exclusively for all searching, in Leopard it is quite probable that I will very rarely turn to NotLight. Coming from me, that’s big praise. The fact is that the difference from Tiger to Leopard is like night and day: from being a pain and a trial to use, Spotlight is now a joy; from a wretched, ill-advised interface, we now have a model of how interface ought to be, a gorgeous, easy-to-use graphical expression of a powerful and complex underlying syntax. In short, Spotlight could be another major reason for
upgrading to Leopard.