Next-Generation TidBITS Infrastructure in the Works
Has it really been a decade since the last redesign of the TidBITS Web site? (See “Designing a Modern Web Site for TidBITS,” 10 September 2007.) In fact, not everything has remained static over the past 10 years. We’ve moved the site from an Xserve to Rackspace to Linode and continually refined our tools and publishing practices. But at its core, the site has remained largely the same for a long time.
We’re currently working at full speed with our developer, Eli Van Zoeren, to rebuild nearly everything about our Internet presence with the goal of launching in early 2018. That includes the Web site and email issue design, along with our systems for article creation, content management, membership coordination, commenting and community discussion, account handling, email delivery, and more. We’ll be switching Web and email hosts too. Special thanks to our TidBITS members, whose contributions are making this transition possible!
I suspect that why we’re doing this is clear to most people, though perhaps not so painfully obvious as from our perspective. Our current Web site design looks and feels like it’s a decade old, and instead of seeming ironically retro, it makes the site hard to read on an iPhone and hurts our Google search rankings.
Why not update just the site design? Because of how it’s integrated into our Perl-based homegrown content management system, redesigning only the site would have been extremely difficult — it’s all intertwingled.
On the email side, a redesign would have been more straightforward, but we’ve struggled with deliverability problems over the years, so we’re looking forward to reworking our email system so we can outsource it to a company like SendGrid that specializes in sending massive amounts of email. I’ve spent far too much of my life fighting with email — it’s time to let experts deal with it.
Because of being written in custom Perl code, our content management system is brittle and difficult to tweak, improve, or explain to another developer. Even Glenn Fleishman, who wrote it originally, has had trouble fixing problems since so much time has passed since he was immersed in Web programming. So we’re headed to WordPress. It’s not perfect, but it’s robust, well-maintained, and highly extendable. If WordPress is good enough for the New York Times, it’s good enough for TidBITS.
The big win of WordPress is its extendability, both in terms of existing plugins and custom code. For instance, we plan to move TidBITS member management from eSellerate to the Paid Memberships Pro plugin for WordPress, with payments processed through Stripe. (In fact, I set up the TidBITS Content Network site using WordPress and Paid Memberships Pro, so I have some experience there already.) Plus, Eli can write custom plugins for TidBITS-specific behaviors such as building Watchlist and ExtraBITS articles for the email issue.
The big unknown right now is how our article creation process will change. This won’t affect readers, of course, but is how Josh and I spend a lot of our time. We’ve been writing in BBEdit and committing new and updated articles to a Subversion repository that our content management system then reads. Aspects of it are great, but it is finicky to set up and use, which gets in the way of fluid collaboration with writers. Since we already rely on Google Docs for collaborative editing, we hope that WordPress’s Chrome add-on for importing posts from Google Docs will eliminate the need for the copy-paste dance. If that doesn’t
work, we’ll try Wordable before investigating a custom solution.
Another advantage of moving to a new system is that it gives us a chance to clean up accounts. Until we sold Take Control Books to Joe Kissell (see “Take Control Books Acquired by Joe Kissell,” 1 May 2017), TidBITS and Take Control shared an account database. That’s no longer true, and we’re taking this opportunity to remove all users whose accounts were associated only with Take Control. We can also reduce the amount of data we manage since TidBITS-only accounts won’t track book ownership.
We’re also hoping to eliminate some confusion that comes about because our commenting system works with TidBITS accounts, but doesn’t require them, and because our Mailman-hosted TidBITS Talk mailing list uses entirely separate subscription accounts. All services going forward should rely on the same account information.
Speaking of commenting and Mailman, we’re replacing both. The commenting system works well but suffers from some annoying formatting limitations and other bugs we couldn’t fix easily, and Mailman requires too much work to maintain. The current plan is to use Discourse for both per-article comments and general TidBITS Talk discussions. Discourse has a Web focus (and lots of interesting capabilities there) while still allowing email-based discussions, and we’ll see how it works.
Running WordPress and Discourse will require separate hosting accounts, but the combination should still be vastly simpler to deal with than our current CentOS-based virtual private server. While I can monkey through many Unix tasks, I find administering a production Unix server highly stressful and time-consuming. We’re running our staging version of WordPress at ArcusTech, which provides a fully managed server, and we’ll install Discourse at Digital Ocean because ArcusTech doesn’t support the Ruby on Rails environment Discourse needs.
In the end, we’re looking forward to providing you with a Web and email presence that’s more visually appealing, mobile friendly, and modern. It will also be simpler — we plan to drop features that no longer make sense or that require too much work. For instance, the hard-wrapped text versions of the email edition require a lot of hand-coding while writing, complicate the email management interface, and look lousy in most of today’s email clients. Simultaneously, though, it should become easier for us to add new capabilities because we’ll have a platform that supports tons of plugins and that many developers understand.
You will certainly notice these changes, and you may need to log in to your TidBITS account again, tweak email filters, or make other minor adjustments to how you interact with TidBITS. But we’ll do our best to alert you to important things in advance and make the transition as easy as possible. We hope that you’ll appreciate all the changes from the front end as much as we’re looking forward to the new back end.
I, for one, look forward to working with my new WordPress overlords.
But will it still support Setext? I still have Easy View. Yes, I've been reading Tibits that long. Keep up the good work.
Alas, no — that's one of those aspects of the current system that's brittle and requires manual intervention every week, since it's devilishly hard to wrap text properly with an algorithm, among other things. Josh and I spend non-trivial amounts of time on every article making sure it will look good in straight text. So we're looking forward to an HTML-only approach, and one that will look a lot better and more modern than what we have now.
Thanks for reading for so many years! The main thing that's important is that we'll keep producing the kind of content you're accustomed to getting from us.
I think a responsive design with clean UI, good mobile support.. will more than make up for the absence of Setext. Was mostly yanking your chain on that one.
I was wondering the same thing about setext. I've been reading since late 1991, and, frankly, I didn't know Easy View was still around, though you might be using it on a really old machine...
The last version of Easy View was 2.6.2, I think, probably from mid-1995. It never made the transition to Mac OS X at all, so it would require something that could run Classic apps. Akif is a great guy and is still around, though I haven't talked to him in years.
I am looking forward to decommissioning my battleship made of chicken wire and plaster! It was a creature of its time—and that time has passed. Pre-congratulations on this upcoming move!
And in fact, many people who have written for other publications have praised the TidBITS Publishing System for how much better it was than what other sites used. But 10 years is a long time, and other content management systems have improved radically in that time.
I'm sorry to hear you're moving to WordPress as all the WordPress websites I've seen are downright ugly with a capital UG. Thus, I've done my best to stay away from any WordPress site. Besides, IIRC, they were hacked in both 2016 & this year 2017. Doesn't exactly instill confidence in their system!
WordPress powers something like 25% of the websites in the world. Did you miss this part of the article where he wrote that the New York Times uses it? WordPress sites don't have to look any particular way.
Because it powers so many sites, WordPress is more of a target for hacking; a professional hosting service can promptly take care of installing patches. WordPress's code itself is pretty solid, the many, many plugins made by 3rd parties are much more prone to contain vulnerabilities, TidBITS should be very cautious about which ones to use.
Discourse is a good web forum. I'm skeptical that any web/email hybrid can handle email well but I hope I'm wrong.
Plugins are indeed the concern when it comes to WordPress, since they provide so much flexibility, but can also cause reliability and security problems. Our developer is extremely aware of this, and is pushing us hard to use only the minimum necessary plugins.
We'll see how well Discourse works for email. I'm quite unhappy with Mailman right now because it's generating a non-trivial support load and because I don't think it does much of anything to encourage more people to read TidBITS.
Excellent article, I love hearing what's going on, and great that you are upgrading the TidBITS systems. I am particularly glad to hear that the hard wrapping is being removed from the email edition, it is far from ideal when reading on my iPhone. However, I do like the text-only nature of it. I think you use Markdown (which could be considered the spiritual successor to SeText) to write and format the articles? Is there an option that a text version of the email edition could just include the original Markdown source? I can live with an HTML email edition if not, but would love a Markdown-formatted text one!
Regarding Mailman, I'm impressed you've run it this long. I ran a few mailing lists with it for several years, and while it is very good for its era, the support overhead was significant, even with tiny lists, nothing like the size of TidBITS Talk. Is there a reason you aren't just going to host the new mailing list/forum directly with Discourse? Wouldn't that cut down on the support burden even more than maintaining your own Discourse installation elsewhere? Great choice by the way. I've used a few forums running Discourse and it's an excellent system. Not interacted with it primarily by email, but I think it was built to support that from the beginning.
All in all, exciting news! I'm sure it's a lot of work, but should set you up for a much reduced support burden, and lovely modern website and email for us all to enjoy. :-D
If our current plans come to pass, we'll be dropping Markdown in favor of writing in Google Docs. We've used Markdown until this point because it's easier than writing HTML, but it's still harder than just writing formatted text. Plus, sizing and laying out graphics isn't possible in Markdown.
Discourse hosting is expensive, which is why we aren't considering the commercial hosting. Plus, because we're going to be embedding Discourse within each article for per-article commenting, that might be easier when we control the installation.
Looking forward to unveiling it all for you!
All makes sense, thanks for the further info. I'm looking forward to seeing it! ;-)