Have you ever encountered an URL like this before?
Probably not. It’s a new Uniform Resource Locator (URL) scheme that’s designed to improve access to information that’s replicated on multiple Internet servers. Using fab URLs, you can gain access to a number of identical servers without tracking down their individual URLs. Although the first implementation is available only for the Macintosh, the scheme itself is not tied to a specific platform.
Purpose — Have you ever tried to download a program from a popular Internet site, only to lose the connection partway through or have the transfer speed drop to unacceptable levels? Probably, and especially so if you live in a country where Internet access isn’t ubiquitous or reliable. In many cases when you’ve been frustrated by failed download attempts, there may have been other responsive sites that contain exactly the same files. In Internet jargon, these sites are called mirror sites.
If you are a Macintosh user, you’ve probably tried to download a program from the Info-Mac archives, grab software written by Peter N Lewis, or get the latest incarnation of Netscape Navigator. In each of these cases, the files you want are available on numerous mirror sites.
But what good does having mirror sites do if you don’t know which one to pick? The primary concern is network performance, so most people usually recommend that you "choose the mirror nearest to you;" however, empirical evidence and scientific inquiry show clearly this advice isn’t sound. It turns out that geographic proximity is not a valid indicator of network efficiency. This and other important results are found in a most interesting paper by Mark E. Crovella and Robert L. Carter, which I found almost by chance with AltaVista. More about this paper later.
How Does the Fab Scheme Work? The fab scheme maps an individual fab URL to a list of URLs. A simple URL of the fab variety looks like this:
For the time being, Internet applications like Web browsers and email programs won’t have the foggiest idea what a fab URL is, so they’ll launch a helper application (as defined in Internet Config) that can deal with the unknown URL type. That helper application is called QuickestMirror, a small application that contains a list of URLs for each fab URL.
Most Internet applications support Internet Config, although the degree of support varies widely. For instance, Netscape Navigator requires an AppleScript kludge like this to let QuickestMirror handle fab URLs.
tell application "Netscape Navigator"
register protocol "QMir" for protocol "fab"
(In theory you can copy this script into Apple’s Script Editor and run it only once, but in some cases Navigator does not remember what protocols have been registered; if that happens to you, check out Stefan Anthony’s Netscape Preferences Fix.)
When you click a fab URL, the Internet application passes the URL to QuickestMirror, which looks it up in the local URL list, constructing a list like this:
The list of URLs that QuickestMirror displays is dynamic, because its goal is to give you the best mirror site for each query. Faster or more accessible sites drift to the top as QuickestMirror adjusts itself to your network connectivity.
This is the big win of the fab URL scheme: when you click a fab URL, QuickestMirror finds the mirror site that’s responding best, creates a URL for that site, and returns that URL to your Web browser or FTP client.
Where Is the URL List Stored? In the current implementation of QuickestMirror, the URL list is stored locally, inside QuickestMirror itself. This design decision has several advantages. First, local storage doesn’t require additional software on the server side, encouraging more widespread adoption. Second, local storage avoids security problems in obtaining mirror lists via the Internet.
One could easily devise a technique for updating the local mirror lists. For instance, a special file containing the mirror list could be made accessible from the root level of the site in question. That way you can update the URL list using the fab scheme itself – that is, the special file becomes part of the mirrored resource and is replicated like any other file. Then, when you connect to a site via a fab URL, QuickestMirror could check that file to see if it had been updated, and if so, synchronize the local URL list with the remote one.
The Crovella-Carter Paper — The Crovella-Carter paper mentioned earlier requires a background in math and statistics, so I’ll summarize their findings here with a few short points.
The number of hops between two Internet hosts is not strongly correlated to round trip latency. So, an Internet server won’t necessarily be slow or unreliable just because a number of machines or routers sit between you and it.
The extra cost incurred at runtime by dynamic latency measurement, as compared to prior static knowledge of hop distances, is well-justified based on the resulting improved performance. In other words, it can really pay off to spend a little time at the beginning of a connection to figure out which mirror site is responding best.
Selection based on dynamic latency measurement is preferable to random selection. So, figuring out which particular site in a mirrors list responds best is better than just picking a mirror site at random.
Dynamic policies consistently outperform static policies; furthermore, the difference between static policies and dynamic policies increases with larger file sizes. Even random server selection is preferable to choosing any static server for documents larger than 5K. In short, selecting a mirror site dynamically based on which one responds better can be beneficial even when transferring small files, but it really shines when working with large files.
QuickestMirror uses a dynamic policy that never places an undue burden on the destination hosts.
Further Reading — For a deeper understanding, I recommend that you read RFC 1738 (Uniform Resource Locators).
Also, more information is available about QuickestMirror and the fab URL scheme.
Finally, Walter Ian Kaye pointed me to a document detailing a Virtual Server protocol that would enable a user to connect automatically to a least-loaded server among the list of mirror or alternate servers available for a particular service. Using this protocol, Internet clients would no longer need to know of the existence of mirror servers available for a particular service.
Mirroring Cracked — There’s no denying that mirroring of popular sites and Internet resources is a powerful concept. It’s worked well in the past with sites like Info-Mac, and it stands to work well in the future, especially as the Internet expands and the need for stable, centralized services grows. With that growth, however, we need tools like the fab URL scheme, QuickestMirror, and the Virtual Server protocol to help make mirror sites more accessible and easier to use.
[Fabrizio Oddone is a software developer and the author of QuickestMirror.]