Border Wars on the Net

UUNET Technologies, a major, top-level Internet service provider with a multi-million dollar nationwide network, recently announced plans to phase out arrangements with other networks to carry Internet traffic free of charge across its network, unless the other networks had substantial, national investments in infrastructure. UUNET said, in effect, "We won’t peer except with our peers."

What’s Peering? "Peer" has two meanings: as a technical term, peering is the act of exchanging data across networks, typically at specific, discrete locations. The other definition of peer as an "equal" we know pretty well; however, in this sense, UUNET defines business equals as companies that invest significant amounts of money into building a T3 (45 megabits, or Mbps, per second) or faster network with at least four geographically dispersed peering hookups. UUNET’s description is a much more specific and higher-end than was previously the norm.

Prior to UUNET’s announcement, peering wasn’t an automatic right for any organization that could put equipment next to other networks’ routers. Even so, any organization with reasonable size, savvy, and stability could negotiate peering arrangements that would substantially improve their customers’ ability to reach other parts of the Internet.

The Internet is not Monolithic — Let’s step back briefly. The Internet strongly resembles a mass hallucination in that there’s a constant delusion that it’s something that it’s not. The net is no single entity; instead it is many entities so intertwined – sometimes this is described as "fully meshed" – that using a machine on any individual network that comprises the Internet feels much like being on every network connected to the Internet.

Without this mass hallucination, we users would have to know a lot about every network’s idiosyncrasies; as it is, standard distributed systems and standard protocols allow us to sidestep knowledge of any given company or organization’s network. (If you’ve ever worked in a large-scale network or what was called an "enterprise" before it became an "intranet," you know that identifying resources is quite difficult.)

It can be hard to remember that the Internet is not monolithic. Web Week’s recent cover story on the UUNET announcement made the enormous simplification of referring to "the Internet backbone" when the author meant "a collection of locations scattered around the United States and the world where networks interconnect."

<http://www.webweek.com/970519/news/small.html>

What’s Peering Got to Do with It? The Internet is based on the notion of peering. Even when the National Science Foundation Network (NSFNet) was in operation (see TidBITS-275), the Internet was comprised of multiple national and regional networks with their own infrastructures that all exchanged data at a few common points. These different networks with different customers – educational, governmental, military, and private sector – all exchange data bound for customers of other networks.

<http://www.nsf.gov/od/lpa/news/press/pr9645.htm>

There are many formal peering points where several or dozens of Internet Service Providers (ISPs) and Network Service Providers (NSPs) exchange data. These include NSF-subsidized Network Access Points (NAPs), which were promoted to help the private-sector Internet grow and have a common place to test and implement next-generation networking technology, like the multi-gigabit-per-second networks currently in testing.

<http://www.merit.edu/nsf.architecture/NAPs.html>

The goal of these peering points, and the reason to have a lot of them, is to enable customers of one network to easily reach customers of another. If people can’t reach Netscape’s servers easily, this is a big problem for Netscape, Netscape’s NSP, the customers who can’t reach Netscape, and the customers’ NSPs, who will switch to other services. Easy access to all networks – making the Internet feel like a single entity rather than lots of scattered individuals networks – is a big marketing bonus for all companies on the Net and all users of it.

These peering points essentially involve having connections to co-located network equipment – "co-location" is a fancy term for "putting routers and/or other equipment in the same physical location." Generally, a phone company maintains the network hardware and connections, while the NSP or ISP plugging into it pays for setup and for a connection from its location into the peering point. This costs several thousand dollars a month at a bare minimum.

NAPs and Multihomed Networks — The Network Access Points are one way of peering, but there’s another way. Any company can purchase service at its own location from multiple NSPs. Suppose I was a large ISP with three T1s to handle the traffic from tens of thousands of customers. (A T1 is a standard unit of bandwidth you can purchase from phone companies: 1.544 Mbps.) I could have each of those three T1s running to a different NSP, like MCI, Sprintlink, and UUNET. I might even be able to convince those NSPs to peer with my network, so that inbound and outbound traffic for my mail and Web servers (plus traffic from my dial-up customers) takes the most efficient path. This is called multihoming, where the network points at multiple networks at the same time – or has "many homes." This can be an advantage for each NSP, since in- and out-bound data from my large ISP can go over different routes when its network is overloaded, avoiding additional congestion.

The difference between multihoming and a NAP is that the NAP acts as a gateway. A multihomed network accepts and sends data only for its local networked machines and clients. A gateway exchanges any and all data across networks. It’s in the interest of NSPs to allow peering at multihomed locations, since they’re paid for each connection and, in some cases, for bandwidth used. It doesn’t make clear business sense for an NSP like UUNET to sit at a gateway like a NAP with dissimilar companies, because that puts UUNET in the position of allowing free transit of data over their multi-million dollar networks and getting nothing in return.

UUNET’s move means that ISPs and regional networks, which in the past simply plugged into a NAP or NAP-like structure, now must have a direct connection to UUNET and/or other big NSPs to take advantage of a fast and distributed network infrastructure. General reporting on this subject indicates that other major NSPs will follow UUNET’s lead. It’s not cheap to be in a NAP, but in the past it’s been far cheaper than establishing multiple high-speed connections to multiple networks.

In Seattle, for example, there are at least two regional NSPs providing NAP-like services. They have feeds from several national NSPs and pay for each feed directly to the NSP; they resell access to this pool of peered service to local businesses and ISPs who can get the NAP-like advantage without having to maintain the pool or deal with peering relationships. Everybody makes money and the customers pay around the same (or sometimes less) than they’d pay to a national NSP.

The Impact — UUNET’s action and subsequent response by other NSPs might drive some ISPs out of business, but it seems unlikely. It will primarily affect NSPs that only peer in one or two locations and resell "downstream" bandwidth to customers. Those NSPs will now need to pay substantially more to achieve a similar level of service from their NSPs, and they’ll have to build and maintain more of their own local infrastructure, too.

The real impact of this change is that the new hugely different scale between biggest and medium size NSPs and ISPs is apparent. Even a year or so ago, the cost for entry into the NSP market was low. The NAPs provided a leveling influence. And, nobody’s service was great. Now, the Internet is a big pond with some big fish who can offer levels of service orders of magnitude above the smaller fish: most of the big NSPs are running or plan to install nationwide backbones running at 155 Mbps or faster.

In an announcement coming close the UUNET news, nine of these big fish formed IOPS.org, a group which has the purpose of communicating meta-information about the Internet across networks – finding and coordinating problems, planning for the future, and so forth. In actuality, it’s a commercial Internet guiding committee that will have the power to control the future of the Net’s implementation and growth regardless of what the Internet Engineering Task Force (IETF), The Internet Society, or other groups might recommend.

<http://www.iops.org/iops-release.html>

Of course, just because they might try to control the Net’s future implementation doesn’t mean they will. Even as the Net has become almost totally commercially funded and driven, netiquette has survived to the point that Congress is currently discussing anti-spam legislation.

The real agenda of IOPS.org may turn out to parallel with UUNET’s announcements. Currently, no settlement fees are charged as data is exchanged across networks. That is, if UUNET sends a terabyte more information across MCI’s network than vice-versa, UUNET doesn’t pay a cent. Of course, MCI controls its own destiny and can shed routes and packets as needed to provide service to its own customers; this happens constantly and automatically.

But as more talk appears about how oversold and overtaxed phone networks and the Internet are, these kinds of announcements will lead more directly to industry-wide implementation of metering or levels of service for which we will pay. Already, several ISPs have introduced deals in which more monthly fees buys you premium access and better service and response. $20 gets you an Internet dial tone, but $40 might buy priority service for your packets (a coming innovation called RSVP) as well as a better modem-to-user ratio.

<http://www.isi.edu/div7/rsvp/rsvp.html>

Bob Metcalfe, inventor of Ethernet and all-around smart guy, has been predicting major Internet outages and the coming of metered services as economic realities start to hit. Up to now, companies have faced enough competition that they couldn’t unilaterally implement fees that were out of line with the rest of the industry. A shake-up that appears to be coming should free the top-end players from these fee restrictions, as they can more clearly demonstrate what being a big fish can do for their customers.

In the end, it’s possible that all of us will be barnacles, firmly secured to the blue whales that will rule this new ocean, while all that’s around beside them is road krill.

[Glenn Fleishman founded one of the first Internet/Web hosting businesses and has since lived most of his life on the Net. He recently finished six months as Amazon.com’s catalog manager, and is now a freelance technology guy and an unsolicited pundit.]