Behind the TidBITS Curtain
I’ve been pretty quiet about our recent transition to a new back-end infrastructure, mostly because I didn’t want to be talking about things that might change from week to week, and because I didn’t want to tempt fate any more than was necessary. Now, although we’re by no means finished with the changes we have planned, everything has stabilized sufficiently that I want to share what we’ve done.
From FileMaker/Lasso to LAMP — Our primary goal with this move was to transition our back-end databases from the highly evolved system that Geoff Duncan had built over many years to modern hardware and tools. Using then-current tools, Geoff had done wonders with FileMaker, Lasso, AppleScript, and yes, even HyperCard. However, performance wasn’t great, and only Geoff understood how the system worked and could modify it. Our long-planned content management system in Web Crossing hadn’t yet materialized, and in an effort to let Geoff extricate himself from day-to-day server babysitting, Glenn Fleishman offered to recreate Geoff’s system using a more-common and higher-performance combination of Linux, Apache, MySQL, and PHP, with some Perl code thrown in as well.
(Yes, that’s right, Linux. Glenn runs several beefy Linux boxes at digital.forest for his other sites, so that’s where we’re pointing tidbits.com now. We may someday purchase a second Xserve and move the system over to Mac OS X Server, where the same tools are also available, but that seems unnecessary at the moment.)
Glenn is an eternal optimist, and although his estimate of how long it would take to recreate the search and results features of the older database wasn’t far off, he didn’t realize just how many tweaky approaches we’ve accumulated over the years (heck, even I didn’t realize quite how much we were doing), so it took a bit longer to implement all our quirks. But at this point, we believe that the new system handles all the tasks the old one did, and we’ve thrown in some improvements that I want to call your attention to.
No More MailBITS — Long ago, MailBITS was meant as a sort of “letters to the editor” column (hence the name), but as the amount of email we received became greater than could reasonably be published, we turned it into collection of short pieces and directed most of the chatty messages to TidBITS Talk. I’ve long been uncomfortable with collecting entirely unrelated bits of information under the MailBITS rubric, so we eliminated MailBITS entirely and are now giving short articles equal status with longer ones (although shorter articles will still come earlier in the issue). We were already breaking the MailBITS column into its constituent chunks in the article database, so this move merely clarifies what is and isn’t an independent article in the email editions of TidBITS. (Take Control News and Hot Topics in TidBITS Talk remain collections, since their constituent items are all related.)
New Persistent URL Formats — You may have noticed that we’ve changed the persistent URL format that identifies every article and issue and article series we’ve ever created. Our previous format relied on the GetBITS CGI that Geoff Duncan wrote, along with a query for an article, issue, or series number. Since it was easy to change all these URLs in our new system we’ve implemented a new system suggested by Geoff that’s significantly more obvious. So, although our system will continue to respond appropriately to the old GetBITS URLs, the new URL formats to use when linking are:
<
ul>
<
ul>
Issue and Article Links — When we streamlined the boilerplate text in our issues as part of the transition, we initially removed the link at the top of the text edition that linked to the issue on the Web. After all, linking to the issue from the issue seemed redundant. However, a surprising number of people told use they used that link. Since all of our editions are now generated from Glenn’s database, it was easy to add not just a link to the issue, but also links for each article; you see those underneath the title of the issue and the title of each article.
Numbered URLs in the Text Edition — Everyone who subscribes to the full text edition of TidBITS has noticed that we started using square-bracketed numbers as a way of connecting link references to their URLs below the paragraphs. The reason for that is that we’re now using a variant of John Gruber’s Markdown format for writing our issues (Markdown was based in large part on the setext format we used to use, so it was an easy switch), and Markdown format uses the square-bracketed references as a way of identifying link text for embedding URLs in the HTML version of a file. In a fit of indecision, I decided to try leaving the references in the text edition for a few weeks to see what I and others thought. Although responses were mixed on TidBITS Talk, most of us on staff, including me and Tonya, disliked the way the references cluttered the text. In fact, we disliked it quite a lot, and as of this week, we’ve removed those references entirely in the text edition, returning to our previous method of including the URLs related to a paragraph directly below that paragraph.
In addition, the particular format we used for the URLs – with the square-bracketed number, then the URL in angle brackets, without a space between the two (because the space caused horrible wrapping on long URLs) – flushed out a bug in Microsoft Outlook 2003. Without a space or carriage return before the starting angle bracket, Outlook was incapable of recognizing the URLs as URLs, and wouldn’t make them hot. This problem will disappear for anyone using Outlook as of this issue; my apologies if you would prefer that we instead made Outlook itself disappear, but that’s a bit beyond our power at the moment.
A number of other people have reported all the links being broken in the issue, but that was unrelated to anything we did. It turns out that everyone with this problem used Yahoo Mail’s Web interface, and it was incorrectly parsing the closing angle bracket as part of the URL. I reported it to Yahoo and they said it should be fixed as of this week.
Luckily, for those who found that they preferred the way the references connected link text with URLs, the HTML edition of TidBITS (both in email and on the Web) now offers normal embedded links, where a link’s URL is embedded under the text. This change, along with the fact that the HTML version flows to the width of your email program’s window, makes our HTML edition an even better option, especially for anyone reading email in a Web browser. And even if you don’t like HTML-formatted email, never fear, because we use minimal HTML markup, with no graphics, tables, CSS styles, or JavaScript to get in the way of readability. And that leads to our next improvement…
Centralized Subscription Interface — So you’re thinking, “Maybe I’ll give that HTML edition a try. I wonder how I switch?” Good question, and before this week, the answer was significantly uglier than I liked. However, anticipating lots of people wanting to switch, I was able to work with Michael Landis of Web Crossing to create a centralized subscription interface. Before I point you to it, though, some background.
Although many people don’t realize it, by virtue of being subscribed to TidBITS, TidBITS Talk, Take Control Announcements, or any other of our mailing lists, you have an account on our Web Crossing server. You received your TidBITS account information when you signed up for the list, or when I added you manually during our big mailing list move several years ago (see “Important News for All Subscribers: Mailing List Migration,” 20-Dec-04). Don’t worry if you’ve lost that information; you can use your subscribed email address as your username, and you can always request a new password.
Needless to say (but I’ll say it anyway), you need to log in to our server to be able to manage your subscriptions; otherwise anyone could subscribe or unsubscribe you to our mailing lists. But whereas the subscription interface was awkward and confusing before, we’ve now reworked it so you can see and subscribe to all the public mailing lists we offer, including all the editions of TidBITS, TidBITS Talk, Take Control Announcements, and even our TidBITS translations. You can also unsubscribe from any mailing lists (including some which are private, such as the update notification lists for particular Take Control ebooks) to which you might be subscribed. And for those lists where digest subscriptions make sense, TidBITS Talk in particular, you can switch to and from digest mode. It’s all done with checkboxes; select a checkbox to subscribe, deselect it to unsubscribe.
Our new Manage Mailing List Subscriptions page handles everything; the only trick is logging in. If you have logged in before, our server should remember you, thanks to a cookie that prevents you from needing to login repeatedly unless you explicitly wish to log out because you’re using a public computer. If you haven’t yet logged in, or if you logged out after your last visit, enter your email address in the field at the top of Manage Mailing List Subscriptions page and click Update to load a login page where you can either login or request a new password by clicking the Problems Logging In? link. (The next iteration of this interface, which may appear as soon as later this week, will make it possible for people new to TidBITS to subscribe as well.)
We have more plans already in the works, and I’ll be sure to alert you to new services as they’re ready for the real world. Thanks for reading TidBITS, as always!