Anyone who visited the Take Control site using Google Chrome from March 9th through 17th might have seen a dire-sounding warning that “Your connection is not private,” much like the screenshot below. That statement was far more extreme than the actual situation, but without knowing the back story, you might have been scared off from using our site. We want to apologize, explain what happened, and how we fixed it.
Certificate Background -- First off, the cryptographic foundation for secure Web connections relies on an ecosystem of which “digital certificates” are a vital part. You can judge your connection secure when you load a Web URL that starts with https, the page loads without errors, and the browser shows a lock icon next to the URL in your browser’s address bar. (The screenshots below show what this looks like in Safari, Chrome, and Firefox.) Secure connections prevent bad guys from eavesdropping on your traffic, so it’s safe to transmit credit card details and other sensitive information.
The entire system is based on trust, backed up by verification. (Interestingly, the saying “Trust, but verify,” is a translation of a Russian proverb that was used regularly by President Ronald Reagan.) At the top level are major browser makers. Apple, Google, Microsoft, and Mozilla all set stiff requirements that third-party verification organizations must meet to be included in — and thus trusted by — their respective browsers.
These third-party verification organizations are called “certificate authorities,” (CAs) because they provide independent authority that a certificate is valid. A top-level, or root, CA has to meet stringent rules for browser makers to bundle its “root certificate” into their browsers. Only several hundred organizations worldwide meet the bar, and browser makers require audits and continuously evaluate reports of any problems.
Root CAs issue what are effectively countersigned certificates: a company like TidBITS that wants to have a secured Web server creates a request with some text portions and some cryptographic ones, and a CA evaluates whether we are a legitimate party for the domain that we want to secure. If we pass its validation, the CA issues a certificate cryptographically signed by it using its root certificate.
You can also work with delegated, or intermediate CAs. Root CAs create intermediate certificates derived from their root certificates to allow other parties, like DNS and Web hosting companies or firms that sell corporate hardware, to create valid certificates that meet the same standards as the root.
The result is a chain of trust, with a CA’s root certificate at one end, often an intermediate certificate from a reseller in the middle, and an individually issued certificate at the other end. By embedding root certificates, browsers (and operating systems) make it possible for a user to know that the connection to a particular Web site is secure: the CA that sold us our certificate vouches that we are who we say we are, and the top-level CA vouches for the issuing CA. In turn, users trust the browser makers to support only trustworthy root certificates. (The browser provides an “out-of-band” element of trust: we’re not using a potentially insecure Internet connection to check that the CA root certificate is valid, because that trust is already baked into the browser.)
This certificate system is based on public key cryptography, with the public certificates associated with private keys held by each entity in the chain. As a result, certificates go beyond verifying identity to enable a Web browser and server to exchange a session encryption key without risk of interception and thus encrypt all the traffic that passes over the connection.
Historically, the process of getting a certificate for your Web server was both expensive and onerous — you could easily drop $300 to $500 a year in the not-so-distant past. Since 2010, when we first went down this road, we’ve worked with a CA called StartCom, which offered substantially less expensive certificates. Getting a certificate from StartCom required multiple authentication steps and scans of legal documents that confirmed our personal and corporate identities, and the process always ended with a phone call to check the details.
Once I had the certificate from StartCom, I had to install it, which involved copying it and StartCom’s intermediate certificate to our server and modifying Apache and Sendmail configuration files to point to it. The certificate was good for two years, so I’ve had to repeat the process three times so far, and honestly, I hate it. I had to take copious notes so I could recreate the steps, and since I’m merely adequate at Unix administration, I was always worried I’d mess something up badly. Nevertheless, I managed to do it, with the last renewal coming in August 2016. (Our long-running on-demand system admin, Glenn Fleishman, also used StartCom until this fussiness got to him a few years ago.)
StartCom Problems -- Here’s where our problems started. It appears that StartCom was quietly acquired by a Chinese CA called WoSign in late 2015. The specifics are still murky, but between the details of the acquisition not being made public and various bad actions on the part of WoSign, Apple, Google, and Mozilla all announced last fall that their Web browsers would no longer trust new certificates issued by StartCom.
That didn’t affect us until 9 March 2017 when Google released Chrome version 57, which also stopped trusting StartCom’s old certificates — including ours. Suddenly, Chrome 57 users saw that “Your connection is not private” interstitial warning before our site loaded. If someone went past that by clicking the Advanced link and the Proceed link, all pages on the Take Control site were labeled as “Not Secure.” You can imagine my consternation! Neither Safari nor Firefox complained because they still trust StartCom’s old certificates. Luckily, because Chrome updates itself over time, and needs to be relaunched before the new version loads, relatively few people saw the problem.
(Thanks to Twitter user @shepgo for alerting me to Chrome’s behavior, since even though he and I weren’t able to figure out what was happening then due to the fact that my copy of Chrome hadn’t updated itself to version 57 yet, when we got a customer complaint the next day, I was able to put it all together fairly quickly.)
Despite these warnings, nothing had actually changed with the security on our site or with the encryption for traffic to and from our site. (I verified this with TidBITS security editor Rich Mogull.) However, Google no longer wanted to use its reputation to vouch for a user’s privacy, because it had stopped trusting StartCom to behave reputably. A more accurate version of the message could have said, “Google cannot verify your connection is private,” but that doesn’t light a fire under users. In essence, Google, along with Apple and Mozilla, were punishing StartCom and WoSign for bad behavior by ensuring an exodus of their customers. It worked.
(StartCom could have dealt with this problem in a positive and proactive way by alerting all its certificate holders about the problem and suggesting a move to other valid CAs until it could reorganize itself and regain trust. It did nothing along these lines, which is why this situation became a crisis rather than a technical task I could have handled in a more relaxed fashion.)
Hey Folks, Let’s Encrypt! -- Once we figured out what had happened, we switched to using Let’s Encrypt, a non-profit CA that provides free certificates. Let’s Encrypt’s certificates are good only for 90 days and must be renewed after that time, although that can (and should) be automated easily.
Happily, Let’s Encrypt has radically simplified the process that I had so laboriously sweated through with StartCom every two years, thanks to client software that supports ACME (Automatic Certificate Management Environment). There are numerous ACME clients that you can run on your Web server, but Let’s Encrypt recommends the EFF’s Certbot, which is relatively easy to install and run on a wide variety of operating systems and with different Web servers. Instead of making you jump through identification hoops, Certbot validates that your Web server is authoritative for your domain by making sure it can confirm a connection to the server using the domain names you claim that you control. This is apparently sufficient for the chain of trust.
If you’re familiar with installing Unix apps, you’ll have no trouble with Certbot. I got Glenn to help since I always worry that I’ll do something that will kill the server. In the end, however, it turned out to be almost shockingly simple. We used wget to grab a copy of Certbot, changed some permissions, and when we ran it, it installed itself. Actually getting and installing a certificate just involved running Certbot again with a flag that told it to configure everything for Apache. It read our configuration files, asked a few questions about what domains to support, acquired and installed a certificate, and updated our Apache .conf files.
Although I didn’t set up Certbot to renew our certificate automatically because I want to see it work the first time, I’ll do that after the first manual renewal by creating a cron job that schedules Certbot to renew the certificate on its own. (Certbot provides a “dry run” option for automatic renewal that lets you test whether any problems would crop up.)
Throughout all this, I was running the Qualys SSL Server Test, and even after switching to Let’s Encrypt, we were getting only a B rating. Some more research revealed that our Apache .conf files weren’t setting choices of SSL ciphers optimally, so the server could potentially use some ciphers that weren’t as secure as others. When I fixed the SSL cipher lines according to these recommendations, our site started getting an unqualified A rating. I also had to delete some extraneous lines in the .conf files that still referred to the StartCom certificate.
The main thing I haven’t yet done is replace the StartCom certificate in our Sendmail configuration with the Let’s Encrypt certificate. I believe I know how to do that, but I need to do more research into how Sendmail will realize that Certbot has renewed the certificate.
Though I spent hours figuring out what happened and how to resolve the problem, it was doubly worthwhile in the end, since working with Let’s Encrypt is cheaper and easier than with StartCom, and the additional configuration changes I made further hardened our site’s security.
One last thing. StartCom and WoSign were the bad guys in this situation, and their customers are the ones who suffered. But dubious behavior on the part of CAs isn’t unheard of, and Google is going to start punishing Symantec, the largest CA in the world, for a variety of questionable practices. If you get your certificates from Symantec or a Symantec-owned reseller, you might look into alternatives.
So, if you’re at all unhappy with your certificate provider, and especially if you bought your certificates from StartCom, I recommend that you give Let’s Encrypt a try.