What follows is a true story. You’d never know it looking at him, but Adam Engst can be a manipulative individual.
The Experiment — Last April, Adam launched an experiment by starting the TidBITS Talk mailing list. TidBITS Talk was intended to provide a forum for public discussion of topics appearing in or related to TidBITS articles. He hoped TidBITS Talk would become a useful public channel for TidBITS staff and interested readers to discuss TidBITS itself, article ideas, or to respond to current events in the Macintosh and Internet communities.
Although it may not be obvious, the TidBITS Talk list is essentially a solo effort from Adam. He discussed the idea with the rest of us, and we said it sounded great, but the last thing we needed was more work. But Adam, appropriately enough, was adamant: he took the task upon himself and set off to make it happen.
Adam’s little project had many unknowns, including total readership, message volume per day, and administration time. Plus, he volunteered to moderate TidBITS Talk so it would remain focused, and he agreed to handle bounces and subscription problems. Since I maintain the TidBITS mailing list database and handle those subscriptions and mail problems, Adam’s plan sounded great to me, and I happily let him be fully responsible for TidBITS Talk. I didn’t feel manipulated at all.
The Results — Happily, Adam’s efforts paid off. TidBITS Talk has carried about 800 messages, or just over 8 per day. It quickly exploded to about 1000 subscribers, almost half of whom elect to receive messages as a daily digest. As moderator, Adam has kept discussion focused and within discrete threads, while keeping several hundred redundant and off-topic messages off the list. The list has attracted a particularly knowledgeable and articulate audience: messages have been surprisingly informative, complete, and well-reasoned. It has also made an outstanding sounding board: if we have an idea about something related to TidBITS or a question about a particular topic, TidBITS Talk subscribers have responded with sensible, diverse opinions. To join the list, visit the page below for details.
Things have also gone well technically. Adam’s nearly decade-old SE/30 and Fog City’s LetterRip Pro 3.0 are handling the load with aplomb. And, in the first 100 days of TidBITS Talk I’ve only dealt with one subscription problem, while Adam was at Macworld. I still didn’t feel manipulated!
Hook, Line, & Sinker — Shortly after TidBITS Talk debuted, Adam and I were talking on the phone. "You know," he said, with no trace of chicanery in his voice, "at some point we should vaguely think about an archive for TidBITS Talk. Something Web-based and maybe tied into our article database." "Yeah," I said. "Something for you to think about." After all, Adam was responsible for TidBITS Talk.
A week later, we were on the phone again, and (as we often do) surfing the Web simultaneously. "Say," Adam said, without even a hint of duplicity, "do you know of any good mailing list archives on the Web?" "Not really," I replied. "Most are pretty bad." Perhaps the best I’d seen was a now-defunct Frontier-Talk archive hosted by Acorn Software. Still on the phone, Adam began looking for Web-based mailing list archives. He didn’t care what the mailing lists were about; he just wanted to see how their archives worked. Before I knew it, I was surfing too, but for technical mailing lists, reasoning that geeks were more likely to create a good mailing list archive.
An hour later, we gave up, disappointed. Most mailing list archives are terrible, providing simple lists of messages, usually with no searching capability. We found only a few that were even usable. Ironically, even archives hosted by companies promoting their own products were awful or non-responsive. In a nutshell, the state of mailing list archives seemed abysmal. "It wouldn’t be hard to do better than this," Adam declared. "Damn tootin’!" I replied.
That’s right. I had played straight into Adam’s hands.
The Tools — Before long, I had designed a FileMaker Pro database to serve as a mailing list archive, intending to connect it to the Web using Lasso from Blue World Communications, just like our article database. Creating the database was easy – the persnickety work lay in finding an intelligent way to convert messages for display in a Web browser. Most other mailing list archives fell down in this area. Special characters must be converted to HTML entities, URLs and email addresses must be converted to links, all manner of bizarre characters and glitches must be removed, long lines must be re-wrapped, date stamps must be converted to Greenwich Mean Time… all while preserving as much of the original ASCII formatting as possible so quotes, signatures, and other specially formatted text remained intelligible.
All this is well beyond FileMaker, so I turned to the trusty workhorse HyperCard, in part because I had code to convert ASCII to HTML and create links, along with other code that would let a HyperCard stack serve as a POP mail client using Chuck Shotton’s scriptable TCP application NetEvents. POP capabilities were important, because the HyperCard stack would automatically retrieve and import new messages into the FileMaker database.
Adam watched all this with mild detachment. "Don’t kill yourself over this," he said. "Just do what you can – I’m sure whatever you come up with will be fine." What technique! What finesse! I dove even deeper.
The HTML — Once the basics of importing and processing messages were established, I turned to the truly exasperating work: an HTML interface. The thing I’d liked most about Acorn’s Frontier-Talk archive was its use of frames. Many Web sites use frames for the wrong reasons, often as a substitute for well thought-out site navigation. But frames can be a good way to provide non-linear access to large bodies of information. A mailing list might seem linear, but the Frontier-Talk archive challenged that assumption by keeping a search form constantly available in an upper pane, while displaying search results and individual messages in a lower pane. Instead of executing a search and then groping linearly through the archive for the information you wanted – or bouncing back and forth between a search form and a results page – the Frontier-Talk archive encouraged users to refine queries until they found what they wanted. Instead of treating the messages as the archive’s primary resource, it treated the information in the messages as the primary resource. I thought this was a fabulous idea, so I stole it.
I also thought messages in a mailing list archive should be "smart." Messages should know to which discussion thread they belong, and provide complete access to that thread regardless of the current search results. Messages should be able to call up any other messages from the same author or from the same date. And, since TidBITS Talk is in part about topics covered in TidBITS issues, a message should link directly to any TidBITS articles referred to in it.
Designing the HTML for the TidBITS Talk archive has been one of the most frustrating publishing experiences in my life. The HTML is the most convoluted I’ve produced – and the most tortuous sections exist only to work around shortcomings and quirks in popular Web browsers. Every Web browser – and even minor revisions of the same browser – interprets HTML differently. Form elements get different amounts of white space (or none at all), and properly formatted tables work in some browsers but not others. Alignment and sizing specifications are whimsical at best: frames and tables can be wildly different sizes in various browsers, or even in the same browser after reloading! And the behaviors change yet again with browsers on other platforms.
I struggled and compromised, and in June we unveiled the Web archive to the TidBITS Talk list, where it was well received. Despite the utility of the Web archive, the HTML design was a monkey on my back. It didn’t work as well across browsers as I’d have liked, and the interface violated almost every rule of onscreen presentation I’d learned from years working in computing, multimedia, and online publishing.
Here’s where Adam exhibited perfect mastery of his craft. The archive was "by far the most capable searchable discussion archive we’ve seen" and he proudly showed it off to anyone and everyone. Clearly, I thought with increasing panic, something had to be done. I went back to the HTML, removing clutter, improving the look and feel, and trying to make each element as smart as possible.
The TidBITS Talk Archive — So, today I’d like to introduce you to the TidBITS Talk archive.
The archive provides searchable, threaded access to all messages distributed to the TidBITS Talk mailing list. New messages are automatically added once per day and integrated into existing discussion threads appropriately. Archive features include:
A persistent search form that enables searching on subject, author, date, and the text of all messages.
Persistent navigation with single-click access to common queries, help, searching tips, and feature descriptions.
Common queries that enable easy access to currently active message threads, as well as messages recently added to the archive.
Message listings that differentiate discussion threads and provide immediate access to all messages by a particular author, in a particular thread, or contributed on a particular date.
Individual messages that provide immediate menu-based access to all messages in the same thread, as well as messages from that author or sent on that date.
TidBITS articles that are referenced within a message (via GetBITS URLs), available by name under the message introduction.
Fast-loading, understated design with minimal graphics.
Take a look at the recent discussions in the TidBITS Talk archive – it’s a great way to stay current even without subscribing to the list. We often discuss topics that don’t make it into TidBITS issues, and Adam has even posted bonus content to TidBITS Talk that wouldn’t fit into a regular issue. There are still some quirks to work out (such as MIME encoding in messages), but now I’m finally starting to think TidBITS Talk is a great example of a Web-based discussion archive.
You’re a sly one, Mr. Engst.