TidBITS Issue #638

Previous: TidBITS 637 Next: TidBITS 639

TidBITS#638/15-Jul-02

Do you cringe when email arrives with garish colors and graphics? Wish you could use grep pattern matching when searching your email? Follow along with William Porter as he reviews Mailsmith, Bare Bones Software's powerful email client. Also on the email front, Adam continues to focus on the issue of overzealous server-side content filtering. We also note the releases of a Mac OS X security update and FileMaker Pro 6, and ask how much spam you receive.

Show the full text of all articles

(Published 9 years and 31 weeks ago)

15 Jul 2002 | Print

Security Update 7-12-02 Fixes Software Update

by Adam C. Engst

Security Update 7-12-02 Fixes Software Update -- Apple has released Security Update 7-12-02 to fix a recently reported problem with Mac OS X's Software Update utilityShow full article

15 Jul 2002 | Print

FileMaker Pro 6 Released

by Jeff Carlson

FileMaker Pro 6 Released -- FileMaker, Inc. has begun shipping FileMaker Pro 6, the latest version of the company's database software. New in this evolutionary release is the capability to import large batches of images easily, or under Mac OS X, to import images directly from digital camerasShow full article

15 Jul 2002 | Print

Content Filtering Exposed

by Adam C. Engst

Hide full article

Geoff's article "Email Filtering: Killing the Killer App" in TidBITS-637 struck some chords. Not surprisingly, the volume of messages to TidBITS Talk exploded, and I struggled to direct messages into appropriate threads. We received a number of reprint requests (including one to post to the Investigative Journalists and Editors mailing list) and were contacted by other publications (including the New York Times) about the subject. David Strom ran with the topic in his Web Informant column, and I was a guest on David Lawrence's Online Tonight radio show on Wednesday to talk about it (there's a streaming version available if you'd like to listen to the conversation).

<http://db.tidbits.com/article/06866>
<http://db.tidbits.com/getbits.acgi?tlkthrd=1679 +1680+1681+1682+1683>
<http://www.nytimes.com/2002/07/15/technology/ 15SPAM.html>
<http://strom.com/awards/293.html>
<http://www.online-tonight.com/archives/ 000655.html>

It's certainly gratifying to see that we can raise awareness of problems like this to such an extent, but the real story is that we're coming up on a critical tipping point for email. The mushrooming volume of spam has caused the value and utility of email to drop significantly for many people already, and the way overzealous server-side content filtering makes email unreliable stands only to worsen the very problem it's attempting to fix. Spam seeks to fill your mailbox, and poorly targeted content filtering, in attempting to prevent the spam from passing through your mail server, can block many of the messages you want to receive. Both hurt the utility of email. Making the problem even worse is that many people (such as Mac.com users - see the discussion Dan Frakes started on MacInTouch on the topic) don't even realize that such content filtering is taking place, so they may never realize that legitimate messages are going missing.

<http://www.macintouch.com/ applemailfiltering.html>

Collateral Spammage 2002 -- There may be no way to determine what percentage of mail servers have some sort of content filtering in place, but I think this is a good occasion to reprise our poll from two years ago asking how much spam you receive each week. When I went back to check that poll's results, I was shocked to see that the answer at the highest end of the range was "more than 71 spams per week." I'm receiving almost that many per day now! That's 25 percent of my mail! So please, visit our home page and tell us how many spams you're receiving per week these days so we can all see how much worse the problem has gotten over the last two years.

<http://www.tidbits.com/>
<http://db.tidbits.com/article/05929>
<http://db.tidbits.com/getbits.acgi?tbpoll=39>

Clarifications and Effects -- Although many people instantly understood what the effect of an overzealous server-side content filter could be, it wasn't entirely clear to all. First off, I want to make it clear that our concerns with content filtering in no way mean that we're in favor of spam. As far as we're concerned, spam is the scourge of the Internet, and we've devoted far more time, energy, and money in fighting spam than almost any small business. Similarly, the fact that we're opposed to erroneous content filtering doesn't mean we're opposed to spam filtering in general, whether it's performed at the server or in users' email programs. There are many ways of blocking spam and rampant PC worms at the server that don't rely on arbitrary content filtering. We employ server-side filters ourselves, but we take pains to minimize the likelihood that our filtering will cause problems for legitimate senders, and whenever we find that it has done so, we work to help address the problem.

Second, since Geoff focussed on the effect that server-side content filtering was having on our attempts to deliver TidBITS issues to our subscribers, many people didn't put it together that this sort of content filtering applies to all email messages, not just those coming from mailing lists or email publications like TidBITS. The size of our mailing list means we notice the problem sooner and suffer more than individuals will, but email messages sent from individual to individual cannot escape the effects of poorly written server-side content filters. The lost mail may not be a big deal, or it might be exceedingly important personal news or critical business communication. Neither the sender nor the recipient have any way of knowing. To paraphrase John Donne, never send to know for whom the content filters toll; they toll for thee.

Third and finally, a common theme among the messages we received was that losing some legitimate messages was worth the reduction in spam thanks to content filtering. Obviously, I can't argue with individual situations - it may be that your mail is sufficiently unimportant that you don't care if some never arrives. More generally, though, I feel that attitude is a tremendously slippery slope. Spammers are parasites who will kill their host, but treating the disease with content filtering is almost certain to have the same effect. Just as we don't automatically treat infections with amputation, neither should we automatically treat spam with server-side content filters.

Overall Practicalities -- Last week, we suggested a few ways you could get TidBITS even if your mail server refused to accept an issue; this week, let me suggest a few ways we can work together to address the problem of bad content filtering.

Contact your ISP or network administrator and ask them point blank if they are performing content filtering on your email, being clear to distinguish from general spam filtering. If so, see if they understand the consequences of those actions. Most likely will, but may consider the loss of some legitimate mail an acceptable trade-off. If they persist in that belief, ask if there's any way the content filtering can be turned off, at least for your account if not every account. I doubt most will respond to a single person complaining, but if you can make the case (often a business case) to other users affected by the content filters, the groundswell might be sufficient to get content filtering removed. As an alternative approach, you might suggest they modify the system so messages caught by content filters are merely quarantined, rather than being deleted, so users at least have the opportunity to recover important messages that ran afoul of the content filters. (The downside of the quarantine approach is that it makes checking email more difficult for the user, thus potentially increasing the cost of dealing with spam.)

If all else fails, I would encourage you to find another ISP or mail host that does not perform content filtering (or at least lets you control what happens to matched messages). Be sure to convey your reasons for switching ISPs to the customer service department at the old ISP so they understand how the lack of reasonable filtering policies negatively affects their business too. Obviously, if you're dealing with your company's network administrator, there's no way to switch, but it will probably be easier to make a business case to management about the effect of legitimate mail being deleted.

Once you've established that all the messages that should reach you are coming through, the next step is to manage them effectively on your machine. TidBITS Talk participants have contributed a number of suggestions for how they manage their spam, and for those of you who are Macworld subscribers, check the August issue (not yet on the Web) for my article on stopping spam.

<http://db.tidbits.com/getbits.acgi?tlkthrd=1684>

The core problem is, of course, spam itself, and the Internet community will have to come together to address spam at a fundamental level. There have been numerous proposals, ranging from legislation (probably necessary at some level, but flawed in its geographic scope and enforcement provisions) to modifications to the Simple Mail Transfer Protocol (SMTP) that delivers every message to its intended recipient. Other efforts focus on plugging the economic loophole that spam exploits; ensuring that spam doesn't pay would certainly take a bite out of the spam load. Most likely, we'll need a combination of approaches, and the urgency of developing them increases daily.

15 Jul 2002 | Print

Mailsmith 1.5: Lean, Mean Email Machine

by William Porter

It's safe to assume that nearly everyone with Internet access uses email, but you can't assume that everyone needs the same features in an email clientShow full article

Hide full article

It's safe to assume that nearly everyone with Internet access uses email, but you can't assume that everyone needs the same features in an email client. Mailsmith, from Bare Bones Software, eschews other programs' emphasis on making messages pretty (in fact, HTML email is not supported), and instead focuses on making it easy and powerful for you to create, store, and locate your messages.

Mailsmith 1.0, released in 1998, boasted a feature set that compared favorably to its established big-league competition, Microsoft's Outlook Express and Qualcomm's Eudora. And Mailsmith had certain strengths that the competition couldn't match, especially in searching and filtering. It was a great version 1.0. Unfortunately, some impatient users (including yours truly) found it unacceptably slow.

With the latest release of Mailsmith 1.5, my old gripe about Mailsmith's performance has been eliminated. And as of version 1.5, Mailsmith is now a full-fledged Mac OS X application (it can also run under Mac OS 9.) But Mailsmith 1.5 is more than an optimized and carbonized version of Mailsmith 1.0.

<http://www.barebones.com/products/ mailsmith.html>

Mailsmith and Text -- Mailsmith's older cousin and Bare Bones' flagship product is BBEdit, in my mind the world's greatest plain text editor. BBEdit is a perennial favorite with programmers, Web developers, practitioners of the arcane art of grep, and anyone else who needs to manipulate or process text in a serious way. In keeping with its bloodline, Mailsmith is an email client with, well, mind-bogglingly deep support for text processing in the broadest sense.

Mailsmith's writing tools - and I'm talking here just about tools that I personally use every day - include user-controlled text wrapping and rewrapping, quoting and unquoting, adding or removing line breaks, multiple clipboards, unlimited undos, support for draft messages, a sort feature, and a find/replace tool that includes full support for regular expressions (the aforementioned grep). The fact that I can access all of these commands without lifting my hands from the keyboard is a serious plus.

New in Mailsmith 1.5 is a glossary feature, very similar to BBEdit's. A Mailsmith glossary can contain as much or as little text as you like, and it can be inserted either from a floating palette or by means of a user-defined keystroke. The glossary feature is overkill if all you want to do is create shortcuts for frequently typed words; something like TypeIt4Me is a much better tool for that and works great with Mailsmith under Mac OS X. What makes glossaries truly powerful is that, in addition to your custom text, they can use placeholders to insert the current date or time, or to control where the insertion point is placed after the glossary item is entered, or what text is selected.

<http://www.typeit4me.com/>

You've Got Mail - Lots of It -- Mailsmith's greatest strength lies in the tools it gives you for managing your mail intelligently, a boon these days as spam continues to multiply.

First off, you want to file messages in logical locations. Mailsmith of course lets you manually drag messages from one mailbox to another or use the contextual menu's Move to Mailbox command, but there is a much better way: filters. Set up the right filters and you will seldom have to worry about moving messages around by hand. Messages from Mom can end up in the your Family mailbox, or messages from your top client can appear in his specific box and flagged as important. An example of a traditional filter - written out as text - might look like this:

IF "From" CONTAINS "editors@tidbits.com" THEN TRANSFER TO MAILBOX "TidBITS"

Filtering is to email what formatting by styles is to word processing, namely, the single most useful task that the program can perform for you. And no program gives you more or better ways to create and use filters than Mailsmith. Mailsmith actually provides three distinct ways to approach the problem of filtering: basic filters, traditional filters, and distributed filters.

You can use Mailsmith's filters in the standard way, to transfer incoming messages directly into the mailboxes where they belong. Considering Bare Bones's reputation for catering to code-it-yourselfers, it may come as a surprise that Mailsmith 1.5 offers an innovation aimed equally at technophobes and at those for whom convenience trumps all other considerations. In Mailsmith 1.5, when you create a new mailbox, you are immediately asked if you would like to create a basic filter to route messages into that mailbox, and if so, what criteria the message must meet. If you had just created a mailbox for your most important client, for example, you might tell Mailsmith to transfer to that mailbox messages whose From header line contains the email address myclient@example.com. (It's easy to tell where your new mail goes: mailboxes with unread messages are highlighted, and they display the number of unread messages.) Additionally, you can ask Mailsmith to notify you ("Mail from your top client has arrived!"), or play a sound, or perform one or two other simple actions. Setting up a filter this way is quick and couldn't be easier, in part because the user does not have to specify the an action transferring mail to the target mailbox, and in part because the number of actions available is intentionally limited to common filtering actions.

On the other hand, if you are up to the challenge, Mailsmith also includes options for its traditional filters that run circles around the competition. In Mailsmith, unlike in Eudora, filters can have an unlimited number of test criteria and actions; and you can give your filters custom names, making it easier to manage them intelligently. And unlike Entourage, Mailsmith filters can use tests with simple logical branching: if (a or b) and (x or y). You can test to see if "any address" matches a particular string, which allows you to use the same filter to catch incoming and outgoing correspondence, say, with a particular client. And finally, you can use regular expressions (grep) to test for text patterns in the subject, headers, or body of messages, which means the sky's the limit.

I would have stopped at running rings around the competition, but Bare Bones did not. There's a third approach: distributed filtering, which is not just a better way to make filters, it takes the whole idea of filtering to a new level. Distributed filters can be attached to any mailbox, and are then triggered as messages percolate through your hierarchy. A single distributed filter can be attached to several different mailboxes and have a slightly different effect in each location, due to changes in the context. I now use distributed filters to process my mail entirely, from the moment it comes in, to the moment it gets deleted; the only thing these distributed filters can't do is read the mail for me.

Distributed filters are much more powerful and flexible than traditional filters, yet at the same time, they are actually easier to set up and troubleshoot. The only catch is that distributed filters are so original that it's hard for new users to grasp the idea at first. I'll cover them in more detail in an upcoming issue.

My only gripe with Mailsmith's filters is that there's no link between filtering tests and the address book. In some other programs, you can test to see if an incoming message is from someone whose email address is in the address book, or who belongs to a mail group. Mailsmith does not provide filtering tests that let you do this.

If half of managing your email is getting messages filed in the right mailboxes, the other half is being able to find those messages later. Like several other programs, Mailsmith provides both simple and advanced query options. The Advanced Query definition dialog is identical to the one used to define filters and gives you access to the same extraordinary range of options. In just a second or two, you can find all messages to or from a particular person, or sent within a certain date range. And once again, you can use regular expressions in these advanced queries.

I use Mailsmith's advanced queries all the time and couldn't imagine life without them. I have only two complaints. One is that Mailsmith does not let you search in several different mailboxes at the same level of the hierarchy at the same time, as you can in Eudora. (You can search inside a parent mailbox and its children, just not in a mailbox and one of its siblings.) My other wish is that you could save queries, another feature that's available in Eudora.

But that's not all. New in Mailsmith 1.5.3 is support for Sherlock-style searching that ranks found messages by how relevant they are to your search terms. Where you'd use Mailsmith's standard searching tools for finding messages that come from a specific person or within a specific date range, the Find Messages About dialog lets you search your entire collection for messages related to your search terms, a much fuzzier approach that's a godsend when you can't remember an appropriate keyword in the messages you hope to find.

Tyrannosaurus Regex -- For years, I pooh-poohed the idea of searching based on patterns of text, generally called regular expressions, also known as regex or grep. When users of BBEdit and Nisus Writer tried to tell me how useful regular expressions could be, I laughed - but it was a hollow, nervous laugh. The truth is, I had taken a peek at grep and found it rather scary. Now I'm hooked, and my addiction has a lot to do with my desire to make the most of Mailsmith's strengths.

Say you want to search for my company's name in the body of messages in a given mailbox, but you can't remember whether it's spelled Polytrope, Pollytrope, Polutrope, or Poletrope. You could perform the search four times, changing the search string slightly each time, or, if you knew just a tiny bit about regex, you could search once for this regex string:

pol+.trope

This string matches any of the four possibilities I gave. Note that coming up with this search pattern does not require a computer science degree. The plus sign (+) and the period (.) are among the most common replacement symbols used in regex. The plug sign means "possibly more than one of the preceding character," and the period means "one of any character."

Regex is also useful in building filters. One of the simplest and most effective tests for spam is this regex :

IF "Subject" MATCHES REGEX PATTERN " +"

That's two spaces and a plus sign. It simply looks for strings of two or more blank spaces in the subjects of incoming messages. I just ran this particular test on my main mailbox and found 90 messages; only four of them were not marketing messages or downright spam. [Searching for more spaces would probably make this filter less likely to catch legitimate messages. -Adam]

Knowing even a tiny bit about regex is immediately useful. Knowing more than a tiny bit means you'll be using it everywhere. How do you learn? The Mailsmith documentation will take you through the basics and then some, although it's short on examples that make sense in the context of an email program. The Mailsmith Talk and BBEdit Talk mailing lists are both frequented by folks who use regex to edit their weekly grocery lists and are helpful to new users. Also useful is the Web site Steve Ramsay's Guide to Regular Expressions. And if the bug bites you, you'll want to get Jeffrey Friedl's book Mastering Regular Expressions, Second Edition, published by O'Reilly and Associates.

<http://etext.lib.virginia.edu/helpsheets/ regex.html>
<http://www.amazon.com/exec/obidos/ASIN/ 0596002890/tidbitselectro00/>

Have It Your Way -- In the best Macintosh tradition (and in striking contrast to Apple's Mac OS X Mail), Mailsmith is extraordinarily customizable. You can assign a keystroke to almost every command in the Mailsmith menu, something that power users will love.

Then there's AppleScript. AppleScript can control almost every aspect of Mailsmith and also interact with other applications. And Mailsmith is not just scriptable, it's recordable. This means that you can write a script - or at least get a draft version of a script - by turning recording on, then performing a series of actions in Mailsmith. You may have to do some editing afterwards, but recording is often a great way to great a simple script started, and it's a boon to AppleScript amateurs like myself. Mailsmith is also one of the few Mac OS X applications that are AppleScript-attachable, which means you can link a script to a Mailsmith menu item so the script runs when you select that command.

Miscellaneous Topics -- A few things seem worth mentioning but don't fit into the larger topics above.

Nothing in Mailsmith better deserves to be described as "bare bones" than the address book. Even Mailsmith adepts admit in private that it might be nice to be able to sort by last names or to have a category field for organizing address book entries. I won't mind if Bare Bones tinkers with the address book in a future release, but I don't want them to "improve" it too much. I keep my fuller contact information in an application of my own which interacts nicely with Mailsmith and OmniWeb under Mac OS X.

Mailsmith stores messages in individual mailboxes, not the single huge database file favored by the Microsoft email clients or PowerMail. Now it appears that Mailsmith examines every mailbox when it starts up, and this scan seems to change each mailbox file's modified date; so unfortunately, a backup program like Retrospect which looks at modified dates will want to back up all the mailboxes every time, regardless of whether or not they have actually received new messages. Even so, storing your messages in many database files does avoid putting all your eggs in one basket.

Mailsmith 1.5 provides built-in support for SpamCop, a spam reporting service. This feature doesn't help you deal with spam that you have received or are destined to receive - you use filters for that - but it does improve the ecosystem by making life more difficult for spammers. A SpamCop reporting account is free; the Mailsmith documentation explains how to configure Mailsmith for use with SpamCop.

<http://www.spamcop.net/>

So What's Not to Like? Mailsmith is as good as I claim, for both ordinary users and power users. But that doesn't mean it's for everybody. Mailsmith does not support IMAP, a type of email account that some people must use or may prefer over standard POP.

And then there's the shocking fact that Mailsmith does not handle HTML. Mailsmith has been designed by people who believe that email should be plain text, for users who believe the same thing. The person most likely to love Mailsmith is someone who likes his email neat: no ice, no soda, no cute little umbrellas. And for heaven's sake, no HTML!

Personally, Mailsmith 1.5 handles HTML as well as I like. If an incoming HTML message is properly formatted, Mailsmith generally does a good job of extracting the raw content and displaying it. If you insist on having the message dance at your table in all its painted and tattooed HTML glory, simply double-click on the HTML enclosure to view it in your browser.

I should mention in this context that Mailsmith 1.5 does use QuickTime translation and playback view images and movies without opening another application (though in a window separate from the incoming message that contains the movie or graphic). This is a service provided by the operating system itself, so it makes sense to take advantage of it.

Also, Mailsmith's server management options seem to have been designed more for safety than user convenience. You can tell Mailsmith to leave messages for one account on the server but delete messages for another account as soon as they have been downloaded. However, you cannot tell Mailsmith to delete mail on a server after a specified number of days. However, the easy-to-use POP Monitor does let view messages on the server and delete or download them individually.

Summing Up -- Should you adopt Mailsmith as your new email client?

The answer will be an easy no for users who aren't using either Mac OS 9 or Mac OS X 10.1 or later, who require IMAP, or who want to compose email in double-byte languages like Chinese or Arabic (which Mailsmith does not support). Some other users may find Mailsmith's user interface simply too austere.

Otherwise, you may want to do yourself a favor and take a close look at Mailsmith. Its combination of powerful editing features, innovative filtering tools, comprehensive support for AppleScript, straightforward interface, and the extraordinary degree to which you can customize it, makes it very hard to resist. It won't be right for everybody, but in my judgment, for most users, Mailsmith may be the best email client for the Mac.

Mailsmith 1.5 normally costs $100, with $80 copies available for academic users or those upgrading from another email program. Upgrades from previous versions are $40. From 15-Jul-02 through 19-Jul-02, you can also take advantage of Bare Bones Software's "Not-at-the-Show" special price of $80 plus a free BBEdit t-shirt. If you want to try before you buy, you can download a fully functional demo (8.4 MB) that works for 24 launches.

<https://store.barebones.com/MWNY2002.html>
<http://www.barebones.com/products/mailsmith/ mailsmith-demo.html>

Show the full text of all articles

TidBITS Watchlist

Stylin' a Word Comment Balloon

Other articles in the series Filtering Gone Bad

TidBITS#638/15-Jul-02

Security Update 7-12-02 Fixes Software Update

FileMaker Pro 6 Released

Content Filtering Exposed

Mailsmith 1.5: Lean, Mean Email Machine