[governance] How the 'Net works an introduction to peering and transit

Thu Sep 4 18:25:40 EDT 2008

How the 'Net works an introduction to peering and transit

By Rudolph van der Berg | Published: September 02, 2008
http://arstechnica.com/guides/other/peering-and-transit.ars/1
(Four Pages: 1/2/3/4)

Whose pipes?

In 2005, AT&T CEO Ed Whitacre famously told BusinessWeek, 
"What they [Google, Vonage, and others] would like to do is 
to use my pipes free. But I ain't going to let them do 
that…Why should they be allowed to use my pipes?" 

The story of how the Internet is structured economically is 
not so much a story about net neutrality, but rather it's a 
story about how ISPs actually do use AT&T's pipes for free, 
and about why AT&T actually wants them to do so. These 
inter-ISP sharing arrangements are known as "peering" or 
"transit," and they are the two mechanisms that underlie the 
interconnection of networks that form the Internet. In this 
article, I'll to take a look at the economics of peering of 
transit in order to give you a better sense of how traffic 
flows from point A to point B on the Internet, and how it 
does so mostly without problems, despite the fact that the 
Internet is a patchwork quilt of networks run by companies, 
schools, and governments. 

The basics

At the moment, the Internet consists of over 25,000 
Autonomous Systems (AS). An Autonomous System can 
independently decide who to exchange traffic with on the 
'Net, and it isn't dependent upon a third party for access. 

Networks of Internet service providers, hosting providers, 
telecommunications monopolists, multinationals, schools, 
hospitals and even individuals can be Autonomous Systems; all 
you need is a single "AS number" and a block of provider 
independent IP-numbers. These can be had from a regional 
Internet registry (like RIPE, ARIN, APNIC, LACNIC and 
AFRINIC). Though one network may be larger or smaller, 
technically and economically they have equal possibilities. 

(Most organizations and individuals do not interconnect 
autonomously to other networks, but connect via an ISP. One 
could say that an end-user is "buying transit" from his ISP.) 

In order to get traffic from one end-user to another 
end-user, these networks need to have an interconnection 
mechanism. These interconnections can be either direct 
between two networks or indirect via one or more other 
networks that agree to transport the traffic. 

A <--> B (direct) 
A <-->C<-->D<-->…<-->B (indirect) 

Most network connections are indirect, since it is nearly 
impossible to interconnect directly with all networks on the 
globe. (The likes of FLAG and AT&T might come close, but even 
they can't claim global network coverage.) In order to make 
it from one end of the world to another, the traffic will 
often be transferred through several indirect 
interconnections to reach the end-user. The economic 
arrangements that allow networks to interconnect directly and 
indirectly are called "peering" and "transit": 

Peering: when two or more autonomous networks interconnect 
directly with each other to exchange traffic. This is often 
done without charging for the interconnection or the traffic. 

Transit: when one autonomous network agrees to carry the 
traffic that flows between another autonomous network and all 
other networks. Since no network connects directly to all 
other networks, a network that provides transit will deliver 
some of the traffic indirectly via one or more other transit 
networks. A transit provider's routers will announce to other 
networks that they can carry traffic to the network that has 
bought transit. The transit provider receives a "transit fee" 
for the service. 

The transit fee is based on a reservation made up-front for 
the number of Mbps. Traffic from (upstream) and to 
(downstream) the network is included in the transit fee; when 
you buy 10Mbps/month from a transit provider you get 10 up 
and 10 down. The traffic can either be limited to the amount 
reserved, or the price can be calculated afterward (often 
leaving the top five percent out of the calculation to 
correct for aberrations). Going over a reservation may lead 
to a penalty. 

Figure 1: peering vs. transit
[see:
http://media.arstechnica.com/guides/other/peering-and-transit.media/diagram-1.gif
] 

These mechanisms are pictured schematically in the diagrams 
above. Diagram I shows peering between two networks. Diagram 
II shows transit over two networks. Diagram III shows transit 
over three networks where there is a peering agreement 
between networks C and D, and A and B both pay for transit. 
Diagram IV shows how A pays to C, and B and C pay to D for 
transit. 

--

Peering

When a network refuses to peer for another network, things 
can get ugly. I once heard the following anecdote at a RIPE 
meeting. 

Allegedly, a big American software company was refused 
peering by one of the incumbent telco networks in the north 
of Europe. The American firm reacted by finding the most 
expensive transit route for that telco and then routing its 
own traffic to Europe over that link. Within a couple of 
months, the European CFO was asking why the company was 
paying out so much for transit. Soon afterward, there was a 
peering arrangement between the two networks. 

Given the rules of peering, we can examine how an ISP will 
behave when trying to build and grow its network, customer 
base, revenues, and profits. To serve its customers, an ISP 
needs its own network to which customers connect. The costs 
of the ISP's network (lines, switches, depreciation, people, 
etc.) can be seen as fixed; costs don't increase when an 
extra bit is sent over the network compared to when there is 
no traffic on the network. 

Traffic that stays on the ISP's network is the cheapest 
traffic for that ISP. In fact, it's basically free. 
Peering costs a bit more, since the ISP will have to pay for 
a port and the line to connect to the other network, but over 
an established peering connection there is no additional cost 
for the traffic. 

Transit traffic is the most expensive. The ISP will have to 
estimate how much traffic it needs, and any extra traffic 
will cost extra. If the ISP is faced with extra traffic 
(think large-scale P2P use), its first priority will be to 
keep the traffic on its own network. If it can't, it will 
then use peering, and as a last resort it will pay for 
transit. 

Figure 2 
[see:
http://media.arstechnica.com/guides/other/peering-and-transit.media/diagram-2.gif]

Every ISP will need to buy some amount of transit to be able 
to interconnect with the entire world, and to achieve 
resilience, an ISP will choose more than one transit 
provider. Transit costs money, and as the ISP grows, its 
transit bill will grow, too. In order to reduce its transit 
bill, the ISP will look for suitable networks to peer with. 
When two networks determine that the costs of interconnecting 
directly (peering) are lower than the costs of buying transit 
from each other, they'll have an economic incentive to peer. 

Peering's costs lie in the switches and the lines necessary 
to connect the networks; after a peering has been 
established, the marginal costs of sending one bit are zero. 
It then becomes economically feasible to send as much traffic 
between the two network peers as is technically possible, so 
when two networks interconnect at 1Gbps, they will use the 
full 1Gbps. But with transit, even though it is technically 
possible to interconnect at 1Gbps, if the transit-buying 
network has only bought 100Mbps, it will be limited to that 
amount. Transit will remain as a backup for when the peering 
connection gets disrupted. The money an ISP saves by peering 
will go into expanding the business. 
Another important limitation of peering is that it is open 
only to traffic coming from a peer's end-users or from 
networks that have bought transit. A transit provider will 
not announce a route toward a network it peers with to other 
networks it peers with or buys transit from. If it did 
announce the route, it would be providing free transit over 
its network for its peers or, even worse, buying transit from 
another network and giving it away freely to a peer. This 
situation is illustrated below (blue is peering, red is 
transit). 

Figure 3
[see:http://media.arstechnica.com/guides/other/peering-and-transit.media/diagram-3.gif]

Network G can see all the networks because networks E, D and 
H buy transit from it. 

Network A can see network F and its customers directly, but 
not network B through network F. 

Network C can see Network B through its peer D, but not via 
its transit customer F. 

Traffic from C to H will go trough E, but not through D. 

The higher up in the network you are, the more networks you 
can see without needing to pay someone else for transit. In 
the example above, a network like G is sometimes said to be a 
Tier 1 network, because it buys transit from no one, yet 
still has access to the whole network. 

It's a common misconception that the benefit an ISP derives 
from peering depends upon the direction of the flow of 
traffic. According to this way of thinking, if YouTube peers 
with an ISP, this benefits YouTube more than it does the ISP 
(since YouTube sends so much data but receives comparatively 
little). But in practice, the flow of traffic is not an issue 
for an interconnect. Whether it goes to or from the network, 
companies still need the same Cisco equipment. 

In practice, it is actually quite likely that the ISP side of 
an ISP-YouTube relationship would see the greatest savings 
both in absolute costs and as a percentage of total traffic 
costs. Most ISPs have less traffic (and buy less transit) 
than YouTube and its parent Google have. Their buying power 
therefore is less than that of YouTube/Google, so their price 
per Mbps/month for transit is likely to be higher. Given that 
the amount of traffic saved from transit is by definition 
equal for both YouTube and the ISP, it follows that the ISP 
is saving more money. 

--

Hot potato, cold potato

Another source of contention and confusion is arguments 
between "hot potato" and "cold potato" routing. Hot potato 
routing is the practice of handing over traffic at the 
earliest convenience (hot, hot! Here, you take it!), while 
cold potato routing is where you hold onto traffic as long as 
you can before handing it over to another network. 

There are long debates in the networking world about which of 
these is the best solution. Hot potato routing may overload a 
link to an interconnection point with many peers, or it might 
force a global network provider to carry traffic all the way 
from Europe to South America at its own cost if it has peered 
with another network, whereas it could have sold transit. 
Some transit providers have solved this problem by splitting 
their networks into several regional Autonomous Systems and 
only peering locally (not globally) with each of its AS 
numbers. 

Cold potato routing may give the originating network greater 
control over quality, except that it is making a guess on the 
status of the network beyond its own routers. In a cold 
potato scenario, it's difficult to factor in changes that 
happen over time, as guesses are made based on the past. Hot 
potato routing, on the other hand, assumes that the other guy 
knows best how to route traffic on his network, and it also 
assumes that if the other network gets overloaded at a 
location, it will have the biggest incentive to upgrade or to 
restructure its interconnects. 

Pay to peer?

Would it be advisable to pay for peering? There has been 
significant debate on whether it is beneficial to pay for 
peering, but I think that peering should typically be free. 
When two networks peer, they both save the same amount of 
traffic from transit. 

As stated previously, the monetary benefits of not having to 
use transit depend upon the transit price that each network 
pays. The network that saves the least is the network that 
has the best transit deals. If, for both networks, a peering 
agreement is cheaper than buying transit, then the choice of 
who should pay for the peering agreement becomes completely 
arbitrary. 

One could say that the network that saves more money should 
share the savings with the network that saves less, but on 
what basis? The peering in itself is already there. Paying 
money for it or sharing the benefits doesn't make it better. 
The only reason the smaller party pays more is because it is 
in a less fortunate position when it comes to buying transit. 
If, through renegotiation of transit contracts, it is all of 
a sudden better off, it would still be hard to convince the 
other network to reverse payments. Worse still, it would in 
fact be sponsoring the other network to attain even lower 
overall traffic costs. If the two networks at the same time 
compete for the same customers, it would now be sponsoring 
its competitor. 

There might be situations where a peering might be beneficial 
to network A, but the savings are too little for network B. 
In such a case it might look good to A to pay B for a peering 
agreement to increase B's savings to such a level that both 
parties will profit. Though this might sound good at first, 
it could have unintended consequences for network A. If the 
traffic between the two networks grows to such a level that 
both parties benefit equally from the peering, B will still 
want to try to keep the payment for the peering; it's 
essentially free money. 

Another problem with pay to peer is that networks would have 
an incentive to understate their transit costs in order to 
become a receiving party. This makes it less likely that both 
parties would reach a peering agreement, because one party is 
lying about its benefits and the other is not willing to pay. 
This is hard to check for either party. The best thing a 
network can do is hope that when it's economical for this 
network to peer for free, it is the same case for the other 
network. If not, the transaction costs of other arrangements 
are probably too high. 

Peering Locations

Peering will happen at a location that is most convenient for 
both networks. When two networks decide to peer in one 
location, that location immediately becomes a valuable place 
at which to peer for other networks, too. This increase in 
value causes more and more networks to cluster together at 
certain locations. In the history of the internet, we can see 
that at first, these locations were at the sites where 
academic networks interconnected, and later on at large 
co-location facilities. In order to facilitate peering, 
Internet exchange points (IXPs) were established at those 
locations. In Europe these IXPs are typically not-for-profit 
associations, while in the USA they operate as private 
businesses. 

Putting a single switch in between all the parties who want 
to interconnect makes it possible to reach all parties with 
one connection (public interconnect), instead of having to 
dedicate a line and a port on a switch for each 
interconnection. This does require IXP's to be neutral and 
uninvolved in the business of their customers; the process of 
peering and transit is up to the networks, and the IXP is 
just responsible for the technical functioning of the switch. 

This doesn't mean, however, that peerings will take place 
only through the IXP. There will still be direct 
interconnects that bypass the exchange (known as private 
interconnects), where the exchange can act as a backup for 
that interconnect (and a transit connection often acts as a 
backup for that backup). 

When more and more networks roll out in the location of the 
Internet exchange point, this location becomes valuable not 
only for peering, but also for buying and selling transit. 
This will attract transit providers to the location in order 
to peer with other networks that sell transit and also to try 
and sell transit to networks needing it. The increase in 
transit providers will cause more competition and, therefore, 
a lowering of transit costs, which will, in turn, increase 
the attractiveness of the location for other networks through 
the combination of more peers and lower transit costs. 
As networks grow, some of them will exchange more and more 
traffic with networks that are not yet present at the local 
Internet exchange. If the costs of buying a direct connection 
to another location where networks are present is lower than 
the costs of transit, then the network will expand toward the 
low-cost location. This is quite clear in Europe, where 
medium and large networks will almost always be present at 
the IXPs of Amsterdam, London, Frankfurt, and Paris. In these 
cities, there are many networks to interconnect with and the 
price of transit is at its lowest. 

The irony is that in some of these towns, transit prices have 
dropped to such lows that it's no longer economical for some 
smaller networks to interconnect at an IXP, since the transit 
fee saved is lower than the monthly fee for the IXP. 

In a nutshell, the economics of interconnection are: 

Peer as much as you can, to avoid transit fees. 
Use the savings from peering to expand your business and 
network. 

Use the expansion of your business and network to become more 
attractive for others to peer with and to reach those that 
are attractive to peer with. 

Establish IXPs in order to further lower the costs of 
peering, to bring together as many networks as possible, and 
to create locations where there is competition between 
providers of transit. 

Repeat. 

--

Transit economics

Providing transit has its own rationales and economic 
mechanisms. Transit providers charge transit fees in order to 
recoup their investment in the lines and switches that make 
up their networks. The price of transit will be a combination 
of the costs of running the network, plus the amount of 
transit the transit provider has bought, minus (maybe) the 
traffic that is destined directly for peers and customers of 
the transit provider. 

Being a pure transit provider with only Autonomous Systems as 
customers puts a network in a weird spot. Such a network's 
business case is built on being the intermediary in the flow 
of traffic, so it tries to charge all of the other autonomous 
systems for their traffic. The problem for a pure transit 
provider is that its customers are always looking at ways to 
lower their transit fees, and lower transit fees can be had 
by switching to a competitor or by not using the transit 
provider at all. So disintermediating the transit provider is 
standard behavior for the transit provider's customers. 

How can the transit provider prevent its customers from going 
to competitors or from cutting it out of the loop? The first 
way is to keep prices down. If a transit provider is the only 
provider of a link between Geneva and Amsterdam, it will have 
to be very aware that its price stays low. If it's too high, 
the customers may opt to cancel their transit contracts and 
either build their own links or compel a competitor to step 
into the market and start competing. 

The other trick is to actively work to keep competitors from 
entering the market. How do you persuade people not to enter 
the market? By keeping margins low, even as growth rises. 
Fiber is a fixed-cost investment, because traffic can be 
supported for little or no extra cost. Though it's tempting 
to let profits rise with the growth of traffic, the network 
will actually have to lower its traffic price every month in 
order for margins to remain the same, thereby keeping intact 
the barrier to entry for a competing network. 

A couple of cooperating ISPs can also be dangerous to the 
business plan of a pure transit player. These networks could 
cooperate in creating a backbone between their networks in 
order to carry traffic to and from eachother's systems. For 
instance, Dutch, Belgian, French, and Swiss ISPs could work 
together and bypass a Trans-European transit provider. So a 
pure transit play is under constant threat even from existing 
customers who resell traffic. 

An interesting tactic that I once heard about was from a 
content-heavy hosting provider who was trying to buy transit 
from residential ISPs. ISPs have a high inflow of traffic; 
hosting providers have high outbound traffic. Because 
incoming and outgoing traffic are bundled into the same 
price, the hosting provider rightly had determined that there 
would be ISPs willing to resell upstream capacity they didn't 
use. For the pure transit player this might be seen as a loss 
of income. 

In the end, pure transit is debatable as a real business 
model. An average end-user is bound to its ISP by numerous 
switching costs (change of e-mail address, lack of knowledge, 
time, hassle, etc.), but this customer lock-in just does not 
apply to transit. The Border Gateway Protocol propagates a 
change in transit provider within seconds, globally. 
Autonomous Systems can switch within seconds and there is 
little a transit provider can do to differentiate itself from 
rivals. Add to this the effect of competitors and mutually 
assured destruction, and one can understand that there is not 
much money to be had in this business. 

Tough at the top: word about Tier 1 networks

Tier 1 networks are those networks that don't pay any other 
network for transit yet still can reach all networks 
connected to the internet. There are about seven such 
networks in the world. Being a Tier 1 is considered very 
"cool," but it is an unenviable position. A Tier 1 is 
constantly faced with customers trying to bypass it, and this 
is a threat to its business. On top of the threat from 
customers, a Tier 1 also faces the danger of being de-peered 
by other Tier 1s. This de-peering happens when one Tier 1 
network thinks that the other Tier 1 is not sufficiently 
important to be considered an equal. The bigger Tier 1 will 
then try to get a transit deal or paid peering deal with the 
smaller Tier 1, and if the smaller one accepts, then it is 
acknowledging that it is not really a Tier 1. But if the 
smaller Tier 1 calls the bigger Tier 1's bluff and actually 
does get de-peered, some of the customers of either network 
can't reach each other. 

If a network has end-users (consumers or businesses), it's 
probably in a better business position than a Tier 1 or a 
pure-play transit provider, since having end-users provides 
stability to a business. Autonomous Systems can switch within 
seconds, but end-users are stickier customers. Churn is less 
of a problem and revenues are therefore more stable and 
easier to base decisions on, since prices don't have to drop 
on a monthly basis. So an end-user business, combined with a 
bit of transit is, therefore, ideal for a network provider. 

Can peering and transit lead to a steady state?

Economists often ask if peering and transit can lead to a 
steady state, i.e., a situation that can sustain itself by 
generating enough money for investments while also providing 
a dynamic and competitive environment. 
I personally think the answer is yes. Experiences in recent 
years have shown a big boom and bust in long haul networks. 
However, I do believe these are the result of over-investment 
and not problems with the model of peering and transit. Five 
overprovisioned networks on the same route are too much for 
any business case. So yes, if investment is done prudently, 
and if the owners of transit networks understand that they 
will have to lower prices continuously or face mutually 
assured destruction, then it is possible to have a stable 
state. 