[governance] FW: [ciresearchers] FW: [IP] A Ridiculous Failure of Critical Infrastructure

Wed Dec 2 13:06:03 EST 2009

Before I get slammed by folks with a lot more technical knowledge than
myself (the Internet is "pipes" etc.etc.) here below is a longish technical
discussion on the email I sent from a colleague working in providing
Internet service (and associated e-services) to indigenous communities in
remote Northern Ontario...

Worth taking a look at also in the context of the Internet for All/Right to
the Internet discussion....

M

-----Original Message-----
From: Adi Linden [mailto:adilinden at knet.ca] 
Sent: Wednesday, December 02, 2009 9:55 AM
To: brian.beaton at knet.ca
Cc: 'Michael Gurstein'; 'O'Donnell, Susan'; 'Adam Fiser';
jeanniecarpenter at knet.ca; 'Penny Carpenter'; terenceburnard at knet.ca;
jamieray at knet.ca
Subject: Re: [ciresearchers] FW: [IP] A Ridiculous Failure of Critical
Infrastructure

Let's see... Sioux Lookout is at the end of a spur. The fiber extends 
from Dinorwic (or even Wabigoon) along Hwy 72 to Sioux Lookout. That 
single fiber bundle on the poles beside the highway carries all internet 
data for the two largest commercial internet service providers (Bell DSL 
and Shaw Cable) as well as all wireline long distance calls and a good 
portion of cellular calls.

There are a few alternatives, as the local school board has their own 
independent wireless backhaul to Dryden. Then there is TBayTel who has 
their own backhaul to Thunder Bay for cellular voice and a limited 
amount of data.

Even though these alternative routes exist, they have nowhere near the 
capacity to replace the Bell fiber in the event of a fiber cut.

A very different scenario that happened a while back, there was a fiber 
cut near Thunder Bay. It disrupted the Bells network connectivity 
between Thunder Bay and Sudbury. As a result it took out many of the 
K-Net circuits, but also affected telephone service and long distance as 
well as internet service throughout the region.

Even though alternate fiber paths exist, the ATM technology within Bells 
core seems to be statically configured. So paths are build through the 
ATM cloud for each circuit. When a major event like this occurs, there 
are no automatic failovers in place. Now my understanding of ATM and 
routing is that alternate routes and automatic best route selection is 
well within the possibilities of that technology, for whatever reasons 
this is not done. So when this fiber cut occurred, Bell eventually 
agreed to reroute circuits, a manual process. But due to the sheer 
number of circuits involved (not just K-Net and some higher priority 
issues such as wireline phone, 911, etc) the fiber cut was repaired 
before any K-Net circuits saw an alternate route. This outage did last 
many hours.

I think part of the redundancy / alternate path issue is the cost of 
running fiber. Either you have provider building backhaul to suit their 
needs only, or you have a provider building a single huge capacity link. 
Neither is the solution.

The provider building for specific purpose only will not be in a 
position to handle someone elses traffic. They will not have the 
capacity, nor will they have the interconnect and peering arrangements 
to offer any alternate paths to 3rd parties on short notice.

The provider building with resale in mind causes everyone to flock to 
the single link available. So you may have several service providers 
offering service, facilitating local peering, doing all the things that 
make the internet a self healing network. Yet they all share the same 
bundle of fiber strung along the same pipeline, railroad bed or highway 
corridor.

So a fiber cut results in everyone going out at the same time still. But 
with many regions way underserved as it is, how do you prioritize? I 
think providing a non-redundant service in unserved regions takes 
priority over building redundancy. Providing people living in remote 
areas with equitable access in terms of cost and quality should be a 
priority. Keep in mind that building out these networks, all these spurs 
will eventually be connected. That in itself will create a mesh and with 
it redundancy.

Adi

Brian Beaton wrote:
> Michael . interesting post and welcome to our world of 
> "end-of-the-line
> spurs" without any redundancy available . this telecom pot needs to be a 
> lot deeper to deliver essential services .
> 
>  
> 
> Sure sounds like a great plot for the continued growth in urbanization
> and being taken care of within concrete ghettos .
> 
>  
> 
> Lots of spins on this story .
> 
>  
> 
> Brian Beaton, K-Net Coordinator
> Keewaytinook Okimakanak
> Box 1439, 115 King Street
> Sioux Lookout, ON, P8T 1B9
> T: 807-737-1135 x1251
> F: 807-737-1720
> IP and ISDN video conferencing
> E: brianbeaton at knet.ca <mailto:brianbeaton at knet.ca>
> W: http://knet.ca <http://knet.ca/>
> 
> ----------------------------------------------------------------------
> --
> 
> *From:* ciresearchers-owner at vancouvercommunity.net
> [mailto:ciresearchers-owner at vancouvercommunity.net] *On Behalf Of 
> *Michael Gurstein
> *Sent:* Wednesday, December 02, 2009 10:30 AM
> *To:* ciresearchers at vancouvercommunity.net; 'CRACIN Canada discussion'; 
> ci-research-sa at vcn.bc.ca
> *Subject:* [ciresearchers] FW: [IP] A Ridiculous Failure of Critical 
> Infrastructure
> 
>  
> 
> A bit of a warning re: a privatized infrastructure for the critical
> elements of an Information Society especially as things are being pushed 
> "into the cloud" at an alarming rate.
> 
>  
> 
> So what happens when, banking, government, retail, education, health,
> telecommunications etc.etc. are all effectively on the Internet/cloud 
> including their transactions, information flow and records and the 
> Internet goes down as below... and your friendly local privatized 
> service provider (whose ownership is buried under a dozen layers of tax 
> shelter dummied corporations ultimately housed in a post office box in 
> the Caymans), and their outsourced help desk doesn't/won't answer even 
> if you can find an operating (non-Internet based) phone ...
> 
>  
> 
> Or have I missed something.
> 
>  
> 
> Hmmmm....
> 
>  
> 
> M
> 
>  
> 
>  
> 
> -----Original Message-----
> *From:* Dave Farber [mailto:dave at farbe i r.net]
> *Sent:* Tuesday, December 01, 2009 12:10 PM
> *To:* ip
> *Subject:* [IP] A Ridiculous Failure of Critical Infrastructure
> 
> On 12/1/09 2:54 AM, "Matt Larsen - Lists" <lists at MANAGEISP.COM
> <mailto:lists at MANAGEISP.COM>> wrote:
> 
> Some kind of combination of failure between Charter and Qwest has left 
> tens of thousands of people in Nebraska without Internet and has
> disrupted the Internet and phone services for thousands more.    Right
> now, the outage is going on 12 hours and there is no ETA for repair in 
> sight.
> 
> The word coming down is that the outage is on a Qwest fiber, but it 
> looks to me like both parties should be on the hot seat for not having
> the ability to route around the problem.    There was a four hour outage
> on Charter a week ago that was caused by a fiber cut in Gothenburg, 
> Nebraska. That one killed everything west of the cut, but it was small 
> potatoes
> compared to this one.   Is this truly the level of performance that we
> can expect from our major Internet backbone providers?   It took me
> about 10 seconds to re-route my traffic to a backup provider - you 
> would think that a couple of multimillion dollar companies would be able
to
> sort out a problem of this nature in a reasonable amount of time.   The
> small CLEC that I use for my backup connection had enough capacity to 
> route around the problem and was even able to lend me a little bit 
> after 5pm when the traffic on their network (mostly businesses) 
> dropped off. It isn't rocket science to figure out how to route around 
> an outage.
> 
> Almost as frustrating is that there was NO news about the outages 
> anywhere except on the social networking sites (Facebook, Twitter). 
> One TV station in Hastings, NE put up a short story on their website, 
> but I got more news from the tweets and FB posts that people where
> posting from their cell phones than I did from anywhere else.   None of
> the network outage sites have any news about this.
> 
> Could this be a harbinger of things to come?   I am feeling pretty
> thankful right now that I have a choice in backbone providers and that I
> kept a second one.   Diversity is a good thing, and this is a great
> example of why we need competition and multiple options for Internet.
> 
> Matt Larsen
> vistabeam.com <http://vistabeam.com>
> 
> Archives <https://www.listbox.com/member/archive/247/=now>
> <https://www.listbox.com/member/archive/rss/247/>
> 
> 	
> 
> [Powered by Listbox] <http://www.listbox.com>
> 
> !DSPAM:2676,4b1597ab25621176811185!
> 

-- 
Adi Linden
Keewaytinook Okimakanak / K-Net Services
Box 1439, 115 King Street
Sioux Lookout, ON, P8T 1B9
Phone: 807-737-1135 ext 1257
Fax: 807-737-1720
adilinden at knet.ca

____________________________________________________________
You received this message as a subscriber on the list:
     governance at lists.cpsr.org
To be removed from the list, send any message to:
     governance-unsubscribe at lists.cpsr.org

For all list information and functions, see:
     http://lists.cpsr.org/lists/info/governance