TidBITS: Issue #420

Previous: TidBITS 419 Next: TidBITS 421

TidBITS#420/16-Mar-98

Ever wondered how we run one of the largest Macintosh-oriented mailing lists and do it exclusively on Macs? Technical Editor Geoff Duncan explains our system in detail. Also, Ron Risley offers advice for people in need of mobile computing solutions now that Apple's terminated Newton development, and announcements this week include the long-awaited Speed Doubler 8.1 and the next generation PalmPilot device, the Palm III.

Show the full text of all articles

(Published 15 years and 12 weeks ago)

16 Mar 1998 | Print

Connectix Releases Speed Doubler 8.1

by Geoff Duncan

Connectix Releases Speed Doubler 8.1 -- Connectix has released Speed Doubler 8.1, an update to the performance-enhancement and utility package that's compatible with Mac OS 8.1Show full article

16 Mar 1998 | Print

3Com Announces Palm III

by Jeff Carlson

3Com Announces Palm III -- Last week, 3Com announced plans to ship a new Pilot, the Palm III "connected organizer," in April. The $399 Palm III, an incremental update to the successful PalmPilot line of PDAs, will offer a more curved case design with an optional flip cover, 2 MB RAM, 2 MB flash RAM, Palm OS 3.0, and an infrared portShow full article

16 Mar 1998 | Print

What's Next After Newton?

by Ron Risley

After publication of my article "Reflections on Life Without Newton" in TidBITS-418, I received many email messages with a common theme: "Your article confirmed that Newton technology is what I've been looking for, but in light of Apple's decision to stop Newton development, what should I do now?" [See "Newton Falls from Apple Tree" in TidBITS-419Show full article

Hide full article

<http://db.tidbits.com/article/04735>
<http://db.tidbits.com/article/04746>

For those who already own Newton MessagePads (or eMates, which also use the Newton OS), the answer is simple. The Newton will continue as a viable platform for at least the next year. There is a wealth of products, and most vendors of Newton software have committed to supporting the platform in the near future. Inevitably, new software development will shift toward platforms with a more definite future and migration to a new system will become necessary.

<http://www.newton.apple.com/>

The Essentials -- Folks looking to make their first investment in handheld computer technology have a tougher choice. When I decided to try handheld computing, I identified two essential features.

The first was that the device must be operable while standing, carrying a book or briefcase, and without having to stare directly at it. Paper-based organizers meet this criterion, and it is one of the reasons that laptop computers failed to displace daily planners. As a resident physician, I generally work in clinics or hospitals where table space is at a premium, and I often write while standing in hallways carrying books or charts. In my years as a consulting engineer, I often found myself in similar situations. A device that isn't usable under these conditions might as well stay on my desk.

The second feature was reliability. For years I had used computers to help organize my personal life, but I realized that, even with scrupulous attention to backups, data could often be lost. In the medical field, losing data you gathered even minutes before (or losing a critical to-do item) could be devastating. In other fields the effect might be less dramatic but could be equally damaging in the long run. If you cannot absolutely trust your handheld computer, you will have a difficult time migrating away from paper.

In 1994 only the MessagePad met these requirements. In 1998, the situation has changed little. Handwriting recognition seems to be necessary to satisfy the first criterion. Physical keyboards require either a solid surface on which to set the device, or two-handed operation. Onscreen keyboards require constant visual attention so you can't enter data while looking elsewhere.

Any device based on DOS, the Mac OS, or Microsoft Windows cannot provide the bulletproof reliability needed to compete with paper. Windows CE might eventually approach Newton's reliability, as might the rumored handheld version of the Mac OS, but reports from early users of "WinCE" devices are discouraging: crashes with data loss seem to be fairly common. It is a difficult engineering task to add, retrospectively, the kind of reliability the Newton platform has had from its earliest releases.

The Future in Your Palm? Of the remaining players in the handheld computing field, 3Com's popular PalmPilot is best poised to fill the gap left by Apple's departure. Though some people object to the PalmPilot's gesture-based entry system, Graffiti (originally developed to rescue the error-prone handwriting recognition of early Newtons), I find Graffiti an easy-to-learn, rapid, and accurate input method. In some cases it is superior to natural handwriting recognition, especially when I'm not looking at the screen while writing. The PalmPilot's operating environment is both open (encouraging development of third-party software) and reasonably reliable.

<http://www.palmpilot.com/>

The PalmPilot suffers mainly from hardware limitations. Reliability, for example, is compromised because current devices lose their memory if left without batteries for over a minute. If your replacement batteries are defective or installed incorrectly, you could suffer irretrievable data loss. The next generation PalmPilot device, the just-announced Palm III, incorporates some flash memory and should be more stable.

<http://www.palmpilot.com/catalog/palmiii.html>

3Com is also addressing other hardware limitations. The new Palm III features an infrared port for data exchange. If it can eventually be made to support IrDA printers as well, that will go a long way toward increasing the flexibility of the device. The Palm III still lacks the capacity to add PC Cards or other storage devices, which limits its usefulness as a reference library or repository for significant amounts of data. Palm devices with more than 1 MB of memory can access Web and Telnet services, which is a step forward in this regard.

Newton or Not? If all you want is a device to replace your paper-based organizer, the Palm III looks reasonable. Unfortunately, its relatively small RAM configurations, limited expansion options, and lack of a direct printing capability restrict its usefulness as a complete replacement for a handheld computer in the MessagePad category.

If you seek a device that fulfills more of the promises of handheld computing, stores personal, business, professional, and reference data in your pocket, and accesses the limitless resources of the Internet, Newton is still the only viable option. For some, the falling prices of the Newton MessagePad 2100 will soften the blow of having to convert to a different platform as their Newtons become obsolete. Others will opt to continue with paper-based methods until 3Com/Palm Computing, or some other forward-looking organization, produces a device that can fill the Newton's shoes.

16 Mar 1998 | Print

Not Your Grampa's Mailing List

by Geoff Duncan

As a general rule, TidBITS likes to be self-documenting: if we set up a new system or service, make some changes, or stop doing something, we generally write about itShow full article

Hide full article

As a general rule, TidBITS likes to be self-documenting: if we set up a new system or service, make some changes, or stop doing something, we generally write about it. We've even written about a few unusual events like the poll to determine whether we should continue listing product reviews from other magazines and our contest to find a full-text search engine.

We do this not only so TidBITS readers know what we're up to, but also because we use Macs to solve real-world problems. Sure, industry news, product announcements, reviews, and analysis are necessary, but explaining how we do things can be as important as explaining how other people do them.

If you've read TidBITS for a while, you might remember that in August of 1996 we moved the TidBITS mailing list from a LISTSERV-based service at Rice University to a system we maintain and administer ourselves. We frequently receive questions about how we manage a mailing list the size of TidBITS. What hardware and software do we use? Did we do anything special to set up the mailing list? How does it all work, anyway?

<http://db.tidbits.com/article/00937>

Although there are now several mailing lists associated with TidBITS, the biggest and longest-standing is the TidBITS setext mailing list, which currently boasts about 50,000 subscribers. We also have separate mailing lists for some TidBITS translations; we host most of these (Dutch, French, and German, for example), while the Japanese translation team handles the sizable Japanese list separately.

<http://www.tidbits.com/about/list.html>
<http://www.tidbits.com/about/translations.html>

Important Notes -- Before going further, let me cover a few important administrative details. First, mailing lists hosted by TidBITS are absolutely confidential. We do not sell, share, lend, or disclose information about individual TidBITS subscribers to anyone for any reason or under any circumstances. Period.

Second, although subscription information is included in every TidBITS issue, it bears repeating. To subscribe to TidBITS, send email to <tidbits-on@tidbits.com>; to unsubscribe, send email to <tidbits-off@tidbits.com>. You can also use forms on many of our Web pages to subscribe. If you want to receive a TidBITS translation via email, check the translation's Web pages to see if a separate mailing list exists.

To move your subscription to a new address, unsubscribe from your current address, then re-subscribe from your new address once you're set up. If you no longer have access to your old address, send a message to <editors@tidbits.com> asking to update your subscription (be sure to include both your old and new address).

Geography & Server Tricks -- Each week, we distribute TidBITS using StarNine's ListSTAR/SMTP mailing list software, running on a mere Power Mac 6150/66 (a PowerPC 601 machine). The 6150 - along with another Power Mac that acts as our primary Web server - are kindly hosted in downtown Seattle, Washington, by Point of Presence Company (POPCO). When we distribute a TidBITS issue, the 6150 uses a fast Unix system at POPCO as a mail gateway because the Unix system pushes issues out to the Internet much faster, helping to ensure prompt delivery. In contrast, an SE/30 running Fog City's LetterRip at Adam and Tonya's house handles several of the smaller TidBITS translations lists.

<http://www.starnine.com/liststar/liststar.html>
<http://www.popco.com/>
<http://www.fogcity.com/>

I maintain the primary TidBITS mailing list using a set of FileMaker Pro databases. Those databases essentially live on my main desktop machine, a Power Mac 7600/120, and no one else uses them.

<http://www.filemaker.com/>

If this arrangement seems convoluted, you're right. Understanding this setup means explaining TidBITS's physical topography. Adam and Tonya work from their home well east of Seattle, and managed to get a 56K frame relay connection installed after battling US West for nine months; I also work from my home east of Seattle, where my Macs are connected to the Internet via a dedicated ISDN line; and TidBITS's primary servers live in Seattle, where they enjoy a T1 connection. The upshot is that the subscription databases are physically separate from our distribution server.

How do we tie ListSTAR and FileMaker together? Since they don't live on the same machine or the same AppleTalk network, we use a combination of ListSTAR rules, custom HyperCard stacks, FTP, and some slight-of-hand. When someone subscribes to TidBITS, ListSTAR appends the name and address to a list of people who want to subscribe and sends an acknowledgment. (The same sort of processing is done for unsubscribe requests.) Every night, HyperCard-based automation on one of my Macs uses an administrative capability in NetPresenz's FTP server to tell ListSTAR to quit. The automation then deftly downloads the lists of addresses to be added to or removed from the TidBITS mailing lists and replaces them with empty lists. Finally, the automation launches ListSTAR again, which resumes mail processing, unaware it's just been hoodwinked.

<http://hypercard.apple.com/>
<http://www.stairways.com/netpresenz/>

Now, the subscription information in the ListSTAR files must be integrated into the FileMaker database. Another HyperCard stack processes each of the ListSTAR files, taking the appropriate action for each address, then composing and sending an appropriate reply or subscription report via email. I chose HyperCard for this task because it's very stable, I have years of experience with it, and building an interface is easy with its tools. HyperCard's capabilities provide flexibility; for instance, since this system went live, email replies from the subscription database have been sent using a variety of tools, including Eudora, custom email XCMDs, Allegiant's Marionet, Atul Butte's TCP/IP Scripting Addition, Chuck Shotton's scriptable TCP utility NetEvents, and FileMaker Pro 4's built-in SMTP capabilities.

<http://www.eudora.com/>
<http://www.allegiant.com/marionet/>
<http://www.mangotree.com/scriptingaddition.html>
<http://www.biap.com/downloads/NetEvents.html>

Why All This Trouble? There are simpler ways to host a mailing list. We could handle the whole thing in ListSTAR, avoiding both FileMaker and these scripting shenanigans. Or, we could pay someone to host TidBITS and never have to manage a mailing list again.

We rejected those options for a variety of reasons. As one of the largest Macintosh mailing lists, we wanted to find a Mac-based solution for distributing TidBITS. Back in 1996, no mailing lists the size of TidBITS were hosted using Macs, and we felt we had an opportunity to change that. Second, fees for hosting a large mailing list were extraordinary, and there was no guarantee of quality service or - importantly - the security of the TidBITS mailing list.

It is possible to host a mailing list the size of TidBITS using only ListSTAR. However, ListSTAR's address list editor won't open a mailing list with more than about 15,000 entries, making it difficult to search for addresses or names manually. Also, since none of us are in the same physical location as the ListSTAR server, we'd have to perform administrative tasks for the mailing list via Timbuktu Pro, which isn't ideal.

<http://www.netopia.com/software/tb2/mac/>

There are other advantages to using a database to manage a mailing list, rather than a simple add-and-remove system. First, a relational database makes it possible to centralize subscription information for multiple mailing lists. If you subscribe to both TidBITS and NetBITS, you may have noticed that the subscription reports returned by the database list every subscription for your particular email address. So, if a subscriber writes to say that her email address has changed, I only have to update information in one place. Similarly, the ability to search and replace through a database makes it simple to correct or remove batches of email addresses when a domain name changes or disappears (remember eWorld?), eliminate duplicate addresses, or search for problematic subscriptions based on fuzzy criteria.

Also, mailing list programs don't store much information about subscribers - usually just an email address and maybe a name. A database, on the other hand, can store any arbitrary information associated with a particular subscriber or subscription. Our database keeps an event log for each, including subscription requests, email problems, analysis results, and manual changes by the database administrator. I've also set up additional fields that serve as flags for malformed email addresses, repeated email problems, and other things I might want to know about. All this makes administration significantly simpler.

Another advantage to a database is that it doesn't have to forget anyone. When a mailing list program like ListSTAR or LetterRip receives an unsubscribe request, it deletes the matching entry. At that point, the subscriber is gone: there's no indication the person was ever subscribed. Conversely, when someone unsubscribes from TidBITS, no information is deleted. The database just sets a termination date for that subscription and sends a note indicating the subscription has been cancelled. All information about that subscriber and subscription is retained.

Why? The database keeps this information in part because I feel it's essential to managing a list the size of TidBITS. I have queries and tools that look for patterns in the information, such as entire domains that suddenly start reporting email delivery problems (this usually indicates a server-level spam filter run amok) or a subscriber who unsubscribes and re-subscribes to TidBITS every day or every week (usually a misconfigured autoreply). This information lets me deal with some problems proactively, and provides more information to work with when subscribers have trouble.

List Profiling -- Other benefits of a database are more subtle. When the TidBITS list was hosted at Rice University, the LISTSERV system there worked well for us - in no small part due to the diligent efforts of Mark Williamson. However, we were often frustrated that we knew so little about the TidBITS mailing list. We knew how large it was at any given time, and we could retrieve copies to search and analyze. But that didn't tell us everything we wanted to know.

For instance, with a database it's trivial for us to figure out how many different domains are represented on the TidBITS list at any time (currently 127, including exotic locales like Niue, San Marino, Cuba, and Iran), which Internet providers have the most subscribers (AOL, EarthLink, CompuServe, and Netcom), and how many people in Japan (1,582), Switzerland (333), or Singapore (139) subscribe to the English version of TidBITS.

The subscription database provides other kinds of information as well. A big issue with services like AOL and cable television is the "churn rate" of subscribers: people who join a service and drop it after a certain period of time. A service's churn rate is an indirect way to measure how well the service retains subscribers, and hence how valuable those subscribers feel the service is. The churn rate for some cable television services like HBO has been estimated to be as high as 40 percent; AOL's churn rate has been placed over 50 percent at times.

<http://www.zdnet.com/intweek/print/960520/ upfront/doc16.html>

Since TidBITS is free, its churn rate should be different from a commercial service, and we've been pleased to learn that's the case. Since we took over the list in August of 1996, TidBITS's average thirty-day churn rate has been 1.2 percent, meaning more than 98 percent of the people who subscribe to TidBITS stick with it for more than 30 days. Similarly, TidBITS's average 90-day churn rate is 1.9 percent, and our average annual churn rate is 8.3 percent. Another interesting statistic is that more than 72 percent of the people who were subscribed to TidBITS when we took over the list 18 months ago are still subscribed today.

Although these kinds of figures are imprecise measures, they imply that TidBITS subscribers perceive TidBITS as a valuable service, and that they stick with TidBITS over time. People often tell us these things when they send us email, but it's good to know those sentiments are borne out over time across the entire range of TidBITS subscribers.

Follow the Bouncing Email -- One of the major problems with administering a list the size of TidBITS is the bounced email. We currently receive 12 to 15 MB of bounced messages and email errors in response to every TidBITS issue. Even though much of that material is the complete text of issues that have been returned to us, it's still too much to cull by hand. So I wrote a program, called Hired Thug, that does most of the work.

Hired Thug works on a simple principle: it looks for several dozen known patterns in the bounces we receive and tries to ferret out the email address that generated the error. Hired Thug then checks the found address against a list of email addresses that have had problems in the past. If the found address isn't in that list, Hired Thug appends it along with the date; otherwise, Hired Thug compares the date of the existing entry with the current date. If the two dates are sufficiently close together, Hired Thug finds the problematic address in the subscriptions database, suspends all associated subscriptions, and makes a note of the type of email error we received.

Every week, Hired Thug pulls 400 to 1,000 email addresses out of the errors and bounce messages we receive; but only one quarter to one half of those addresses are "repeat offenders" who have returned an error to us recently. A short-lived problem with your email provider won't cause your TidBITS subscription to be terminated, but sustained problems that cause repeated errors will result in suspended subscriptions. Also, if you try to subscribe to TidBITS and our confirmation message can't be delivered, we suspend the subscription immediately.

Because our database distinguishes between subscriptions that have been cancelled and subscriptions that have been suspended due to email errors, trying to resubscribe from an address we've suspended won't work. Instead (assuming your email is working correctly), you'll receive a notice that your subscriptions are suspended and you can't reactivate them. At that point, contact us at <editors@tidbits.com> and explain the situation. We can usually tell you the nature of the errors we've received and help you get your subscription running again.

Although mailing list software like LetterRip and Lyris (not available for Macs) now offer error-handling capabilities, they still use a "simple delete" methodology, so all information about the problematic address is lost. By tracking errors ourselves, we're better able to track problems like errant spam filters, mail loops, or mail processing changes made by large services like AOL or CompuServe.

<http://www.lyris.com/>

If you administer your own mailing list and want to use Hired Thug to process bounces, you can't. Hired Thug is tightly integrated with the TidBITS subscriptions system and can't easily be converted into a stand-alone utility. However, Vince Sabio's program SmartBounce offers similar features, and it's designed to work with most mailing list software. I wrote Hired Thug before SmartBounce was available, but if Hired Thug wasn't working so well and tailored precisely to our needs, we'd probably use SmartBounce.

<http://www.smartbounce.com/>

Show Me The Database -- When people learn that we manage subscriptions via a database, they immediately want to know why we don't let people manage their own subscriptions or let people send subscription commands from email addresses other than the particular one subscribed to TidBITS. Although these requests are well-meaning, we don't allow these things to preserve the confidentiality of our subscribers and prevent simple forms of list tampering.

For instance, if we were to set up Web-based searches through our subscriptions database, it wouldn't be long before a spammer used that capability to obtain thousands of email addresses. Similarly, if we accepted subscription commands on behalf of third parties, some people would use the capability to determine whether a particular address was subscribed, cancel someone else's subscriptions, or even add someone to the TidBITS list without their permission. If this sounds far-fetched, consider that we receive a few dozen bogus subscription attempts every week (with addresses ranging from Bill Gates to Bill Clinton), and occasionally someone attempts to subscribe an address twenty or thirty times under slight variants of the same email address. Although the mailing list database isn't tamper-proof (an impossible task even with digital signatures or photo ID) we try to make abuse difficult.

All For Now -- Ideally, all this complexity will be invisible to TidBITS subscribers, who should be able to sign up and begin receiving issues with minimal effort. In the future, we hope to offer more advanced capabilities and subscription options, but for now, we're pleased that the system is working well and that we can do it all from our Macs.

Show the full text of all articles

TidBITS Watchlist

Fill in Gaps in Pear Note

Other articles in the series Life Without Newton?!

TidBITS#420/16-Mar-98

Connectix Releases Speed Doubler 8.1

3Com Announces Palm III

What's Next After Newton?

Not Your Grampa's Mailing List