The Economy of IP Bandwidth

Warning: This post is a bit geeky. If you’re not technically minded, you may want to check out the next one


Mediating the Network

I’m often asked by what do I mean by “mediating the network”. I’ve covered the topic a few times before. However, this time I want to be a bit more specific.

Let’s first review what needs to happen in order for ISP to maximize the “billable capacity” of their networks in a tiered Internet. Today, almost all network operators have a flat-rate best-effort model for service. Australian providers, and very recently, Time Warner’s Cable network represent some rare exceptions. Under the current broadband billing paradigm, consumers pay a single monthly fee for as much bandwidth as they can consume. In the case of the Time Warner trial, and Australian ISPs, they pay for the amount of best-effort bandwidth they consume by the byte. The additional charges amount to the absolute quantity of bandwidth at varying qualities consumed as opposed to charging for the quantity and quality of that bandwidth.

The Current "Tiered" Internet

Some net neutrality proponents believe that there is no tiered service on the Internet as it exists today. I would, of course, respectfully disagree. Large and small corporations today purchase tiered services from a number of providers. One of the most notable examples are Content Delivery Networks (CDNs). These networks promise to reduce transit bandwidth costs by caching frequently used content closer to the edge of the network nearest the consumer. They also effectively tier the network by taking advantage of a “bug” or, in their case, a “feature” of TCP and the queue management systems that are actively deployed today. The current dominantly used version of TCP, “Reno”, has some interesting behavior on the network. It works in such a way that with shorter round trip times, or delay, those TCP flows from the CDN are able to obtain a greater share of the bandwidth at bottleneck points on the network. This has been well understood for a while. Peer-to-Peer (P2P) networks actively use this strategy very effectively – often collectively obtaining more than 80% of the available bandwidth on at broadband congestion points.

Therefore, the Internet is already “tiered” as it is today. Websites often use CDN services to distribute “high bandwidth” content like video because it streams or downloads faster off those distributed servers. This may represent “left hand lane” best-effort and not “true diamond lane” service. It is, however, better service than you get from your typical hosting provider. The cheapskates, on the other hand, often use P2P networks. They rely on broadband consumers downloading software, which shares their uplink connection's bandwidth to obtain the lowest cost distribution. Consumers don’t want to download and share their computer or bandwidth unless they get something in return. Unfortunately, for Hollywood, they most often get a freebie in the form of pirated content in exchange for sharing. The P2P applications can get even a greater share of bandwidth than the CDNs because they frequently have even shorter round trip times.

An interesting development has occurred over the past couple of years. New versions of TCP such as those from Netli (now Akamai), FastSoft, and nuMetra can also grab far more bandwidth from those congested bottlenecks. There are also UDP-based solutions on the market from companies like Rivulet and EdgeStream. The TCP systems can also operate very stealthily with higher level protocols masquerading on port 80 so they look like fairly ordinary HTTP traffic. This gives the ISPs an even greater problem. They are basically left with having to ever increase their capacity without a commensurate increase in revenues under the current pricing regime. Thus. we have experiments with the Australian pricing model by the likes of Time Warner.

The questions becomes whether, from a competitve standpoint, these new TCP and UDP implementations are playing by the rules. Being overly aggressive and not polite on the network can be bad for both the ISPs and consumers who are not using that aggressive protocol. Competively, the CDN providers must move to this model for the simple reason that they all must offer best "left-hand lane" service they possibly can. Therefore, from a business standpoint, the polite versus aggressive debate become pretty academic. Every driver knows, particularly those in Southern California, the difference between diamond lanes and the left hand lane to the right of it.


Mediation enables diamond lane access for the media grid. CDNs or Tier 1 networks with only "left hand lane" access will be left in the dust. As I'll discuss below, the Open Source Peering Agreement (OSPA) creates a uniform diamond lane system for all ISPs on the Internet and allows the current CDN providers to make use of it in exchange for conforming to the standards of the agreement.

Bandwidth Quality, not Quantity

Bandwidth has quality characteristics that can be monetized. If we look at it two dimensionally, packet loss and latency, rather than in a single dimension, bytes, other much better pricing models could emerge for the ISPs. Obviously, the big one, today, is high latency video on the Internet. Back in 1996, a mathematician by the name of Tom Kurtz, provided the basis for an economic and theoretical framework to describe the behavior of packets in bottlenecks using simple priority, resulting in my interpretation of the ISP opportunity, shown in the diagram below (jlt [at] numetra [dot] com (email me) if you want to discuss it):


In essence, as one the ISPs we’ve been working with described it to me, they need a general purpose “bandwidth spigot” to fairly (whatever that means) distribute the packet loss and latency for the bottleneck appropriatly for each application. This “spigot” is analogous to the trickle (low volume & pressure), high volume, and high pressure narrow stream feeds most of us have in our automatic irrigation (water) systems outside our homes. The combination of the ISTP’s TCP plus some secure AQM technology in the bottleneck with some various and sundry other features allows us to create the right kind of “spigot” so that ISPs may maximize the billable capacity of their network. By the way, this approach does not violate the principles of net neutrality, as long as ISPs agree to set their mediation policies to adhere to common carriage principles. The nuMetra design for this system will operate in all world jurisdictions by allowing the ISPs to handpick those mediation policies for their network.

A central tenet of ours has been to recognize that "fairness" in whatever flavor you mean can only be enforced inside the network. Relying on software developers who develop congestion control algorithms to be "polite" has no commercial value. The creation of these polite TCP implementations should be left to academia. The commercially viable and valuable work is in mediating those bottlenecks.

ISPs Need a Flexible Business Model and Solution

In a way, many ISPs really don’t care about Hollywood’s piracy problem. Some believe, such as Time Warner, that they can cost factor all of those P2P users, and high volume video consumers into the pricing equation and make more money from their broadband service by getting the consumer to pay by the byte. We'll see if the consumer bites on that value proposition.

Whatever new or existing network equipment they may be providing to do will almost certainly not have the quality of bandwidth pricing flexibility that a system like the one I describe above. In my mind, ISPs should go for systems that maximize flexibility and monetizable capacity while helping Hollywood with anti-piracy and offering the consumer a superior experience.

The IP Bandwidth Food Chain

I hope I’ve made it clear that P2P is like the “hobos” of the great depression on today’s IP bandwidth train. Normally, they let them on, only if they could hop a ride on the caboose. ISPs, today, are allowing them in the coach passenger cars. In the current best-effort networking paradigm, they sit at the top of a pyramid. They also offer a really lousy customer experience. Joost's problems are just one example of this. Fundamentally, P2P systems are for hackers and teenagers to pirate content. The complexity of using something like BitTorrent makes it all but impossible to easily watch what you want when you want it. Web-based distribution systems like Hulu and YouTube offer a much better and more satisfying consumer experience. In terms of the IP bandwidth economy, the consumer and distributor both get what they pay for.

If you’ve read our wiki and have been following what we’re doing, you probably recognize that the Tier 1 ISPs are next down on the bandwidth food chain. In essence, they never have to pay for IP Transit, so they should have the second lowest cost structure for delivering video across the Internet. CDN-only operators, and Tier 2 ISPs then Tier 3s are next down pyramid in terms of cost structure. The diagram looks something like this:


Our Open Source Peering Agreement (OSPA) may change the pricing dynamics for the current peering and transit agreements by creating a new world order of Tier 1s. Level 3, a true Tier 1 provider, for example, just announced pricing for its CDN services that are comparable to that of IP Transit. They’re pushing the commoditization effort of CDNs forward rapidly and beginning to focus on higher value-added services. Implicitly, they and other Tier 1s recognize these distributed “grids” are the only way video is going to be well supported on the current and next generation web.

Since under the OSPA, “new world order” InterStream Tier 1+ ISPs must share their “transit keys” with one another so that common carriage is enforced by allowing ISPs to unlock each other’s premium bandwidth services at the bottlenecks. The OSPA, in essence, allows the GPL to enforce common carriage practices across the Internet. If what net neutrality really means is common carriage, then I’m totally in favor of it as long as ISPs are allowed to put those hobos in the caboose.

The Key Questions

We’re now left with a couple of questions: How will the new “Tier 1” network evolve under the InterStream approach? Will ISPs be willing to adopt open source?

The answer to the first question is pretty obvious. We’ll have exactly who we’ve got today or we’ll get some intersection, or perhaps even a subset of the current group. It could remain as the same oligopoly or, more likely evolve into a much more complex ecosystem. We’ll see as we finish the process of defining the OSPA as more ISPs participate.

We believe ISPs will readily adopt a mediation system based on open source. They have a natural incentive for the patent infringement protection it offers them as well as dramatically reducing their capital and operational costs. I’m not the only one in Silicon Valley who feels this way. I expect the vast majority of the ISPs and network operators are looking to help improve their profit margins as well as be willing to support common carriage over their networks.

We’ll see over the next several months how the OSPA will reshape IP bandwidth economics. I expect the same breakthroughs will occur with net neutrality as long as the net neutrality proponents stick to their knitting with a common carriage message and not push a bill that could have disastrous unintended consequences.

Jeff Turner


Post new comment