Info-Mac Archive Mirror Network Improved
Since the dawn of the Macintosh (really!), a small group of volunteers has been toiling away to provide services to the Macintosh community. Known as the Info-Mac Network, this non-profit organization publishes the Info-Mac Digest, a moderated mailing list of all things Macintosh (currently on hiatus while the group works through some problems with digest scripts after moving to a new server) and the Info-Mac Archive, the oldest (and for many years the largest) archive of freely distributable Macintosh software and information.
In recent years, Info-Mac has had a hard time keeping up with the many companies who found they could make money from similar services via advertising – hence the rise of CNET’s Download.com, VersionTracker, MacUpdate, Tucows, and others. For a long time, Info-Mac was saddled with an extremely old server that didn’t support much additional disk space, and as with any volunteer organization, getting anything done quickly is uncommon. I’m certainly not one to point fingers here; as the president of Info-Mac Network, I’m more to blame than anyone else for not recruiting more volunteers to help with our to-do list. (If you’re interested in helping, drop me a note outlining your skills and I’ll add you to our list of people to call on for particular tasks.)
However, thanks to Glenn Fleishman’s misfortunes with distributing his book Real World GoLive 6 in PDF format (See "Publish (Electronically) and Perish?" in TidBITS-672), we’ve taken a significant step forward in making the Info-Mac Archive significantly more useful for everyone.
Pricey Bandwidth — Despite predictions from pundits like George Gilder that bandwidth would become essentially free, it hasn’t really happened. Glenn’s experience is the most extreme I’ve heard of, but it’s not uncommon for developers to complain about the cost of maintaining a high-speed Internet connection for distributing updates to their programs. If a product has several thousand users, and the developer notifies them of the update, that can result in a huge number of downloads in a short period. And that, as Glenn found out, can be devastating.
So bandwidth can be expensive. But what about the commercial file distribution sites? Although it’s often not obvious, they don’t actually host the files themselves (with the notable exception of Tucows). Instead, they just point at the developer’s site, so working with the commercial file distribution sites doesn’t change the bandwidth equation at all, other than by increasing demand for the file, hopefully.
This is where Info-Mac comes in. Although our server is kindly hosted by MIT and has quite a bit of bandwidth because of that, we don’t let people download files from the main server directly. Instead, Info-Mac encourages other organizations to set up mirror sites of the Info-Mac Archive, thus spreading the load much more widely. In the early days of the Internet, reducing the geographic distance your download traveled was also important, although that concern has fallen by the wayside in most parts of the world now.
Mirror, Mirror, On the Net — Info-Mac currently has 22 mirror sites, 7 in the United States and another 15 internationally. As a user, how do you know which one to pick, and as someone making a file available, how do you provide a reasonable interface for your users, without making them pick from 22 links? Years ago, a Macintosh developer named Fabrizio Oddone proposed a new URL scheme that worked with a utility he wrote called QuickestMirror to solve the problem. For a variety of reasons, QuickestMirror and the special URLs never caught on.
This mirror problem is the gauntlet Glenn picked up. I suggested to him we could upload his Real World GoLive 6 PDF file to Info-Mac, but that the user experience wasn’t great because of all the mirror sites. I’d been thinking about the possibility of a CGI that would take a path to a file in the Info-Mac Archive and download it from a random mirror, and when I mentioned it to Glenn, he wrote the necessary script in Perl in a few minutes.
Because of the Perl script, developers can now offer users a single download URL that connects each user to a random mirror site. From a developer’s perspective, here’s how it works. If you click the link below, the script takes you to the top level of a random Info-Mac mirror site. Most Info-Mac mirrors are FTP sites, so it’s likely that your Web browser will launch your preferred FTP client software.
The next trick is to point the script at a specific file. For that, just add a specific path= argument to the link, as I’ve done below for an old issue of TidBITS. Of course, you must find your file on one of the mirror sites by hand (and I strongly recommend using the short directory names rather than the longer ones that appear on some sites – they’re purely to make the file listings more human readable). If you need to search for your file, try the Info-Mac HyperArchive at MIT, and then locate the short directory name on one of the other mirrors.
If you wish to narrow the list of archive sites used, you can append "&archives=us" or "&archives=intl" as I’ve done with these links. It might be a good idea on a download page to provide two links, one for those in the United States and one for people in other countries.
<http://www.info-mac.org/cgi-bin/mirror.cgi? path=per/tb/tidbits-663.etx& amp;archives=us>
<http://www.info-mac.org/cgi-bin/mirror.cgi? path=per/tb/tidbits-663.etx& amp;archives=intl>
Current Limitations — I won’t pretend this is the ideal distribution solution for every situation. First off, you really do get a random mirror from our list, and it’s entirely possible that one could be down temporarily (or permanently – if you run into a dead site, please let us know at <[email protected]>). It’s also possible the random mirror might be slower than a hand-picked mirror. We’re happy to consider improvements to the script from any Perl experts out there; visit the link below for details. Possible improvements include smarter selection of mirrors, load balancing, and improved user feedback.
Second, although our mirrors update frequently, you might want a file that a particular mirror doesn’t yet have; in this case, clicking the download link again should send you to another random mirror for your requested download.
Third, since Info-Mac is a volunteer organization, our archivists, Christopher Li and Patrik Montgomery, aren’t always able to download, verify, virus-check, and post submissions as quickly as might be ideal. We’re working on improving and speeding our submission process, but it could take two or three days for any given submission to appear. Don’t assume it will be instantaneous.
Fourth, and finally, although we currently have plenty of free disk space and have been able to lift our previous submission size limit, thanks to the new server MIT gave us recently, lack of disk space and aging hardware are ongoing expenses that have caused us problems in the past and will no doubt do so again. It would be helpful if developers using the Info-Mac Archive to distribute shareware or demos would donate the equivalent of a single license to help cover our costs. So, if a particular utility costs $15, a $15 donation from its developer would be welcome. We won’t require such donations, but if Info-Mac helps you avoid large bandwidth charges, donating the amount of a single license seems like a reasonable way to chip in.
Future Plans — We have lots of things we’d like to do with Info-Mac, given unlimited time and resources, but since both are in exceedingly short supply, I won’t promise anything. Info-Mac is what it is, and if it’s useful for users and Macintosh developers, I can be happy with that, and I hope you can as well.