[governance] The Story Behind San Francisco's Rogue Network Admin

[Yehuda Katz]

The Story Behind San Francisco's Rogue Network Admin
Paul Venezia, InfoWorld 
PC World / Business Center / Security / Data Protection /
Saturday, July 19, 2008 5:20 AM PDT

Art. ref.:
http://www.pcworld.com/businesscenter/article/148669/the_story_behind_san_franciscos_rogue_network_admin.html

-

The Story Behind San Francisco's Rogue Network Admin:

Last Sunday, Terry Childs, a network administrator employed by the City of San
Francisco, was arrested and taken into custody, charged with four counts of
computer tampering. He remains in jail, held on US$5 million bail. News reports
have depicted a rogue admin taking a network hostage for reasons unknown, but
new information from a source close to the situation presents a different
picture.

In posts to my blog, I postulated about what might have occurred. Based on the
small amount of public information, I guessed that the situation revolved
around the network itself, not the data or the servers. A quote from a city
official that Cisco was getting involved seemed to back that up, so I assumed
that Childs must have locked down the routers and switches that form the
FiberWAN network, and nobody but Childs knew the logins. If this were true,
then regaining control over those network components would cause some service
disruption, but would hardly constitute the "millions of dollars in damages"
that city representatives feared, according to news reports.

Apparently, I wasn't far off the mark. In response to one of by blog posts, a
source with direct knowledge of the City of San Francisco's IT infrastructure
and of Childs himself offered to tell me everything he knew about the
situation, under condition that he remain anonymous. I agreed, and within an
hour, a long e-mail arrived in my in box, painting a very detailed picture of
the events. Based on this information, the case of Terry Childs appears to be
much more -- and much less -- than previously reported.


A Man and His Network:
 
It seems that Terry Childs is a very intelligent man. According to my source,
Childs holds a Cisco Certified Internetwork Expert certification, the highest
level of certification offered by Cisco. He has worked in the city's IT
department for five years, and during that time has become simply
indispensible.

Although Childs was not the head architect for the city's FiberWAN network, he
is the one, and only one, that built the network, and was tasked with handling
most of the implementation, including the acquisition, configuration, and
installation of all the routers and switches that comprise the network.
According to my source's e-mail, his purview extended only to the network and
had nothing to do with servers, databases, or applications:

"Terry's area of responsibility was purely network. As far as I know (which
admittedly is not very far), he did not work on servers, except maybe VoIP
servers, AAA servers, and similar things directly related to the administration
of the network. My suspicion is that you are right about how he was "monitoring
e-mail"; it was probably via a sniffer, IPS, or possibly a
spam-filtering/antivirus appliance. But that's just conjecture on my part."

Like many network administrators who work in the rarified air of enterprise
network architecture and administration, Childs apparently trusted no one but
himself with the details of the network, including routing configuration and
login information. Again, from the source's e-mail:

"The routing configuration of the FiberWAN is extremely complex. Probably more
so than it ought to be; I sometimes got the feeling that, in order to maintain
more centralized control over the routing structure, [Childs] bent some of the
rules of MPLS networks and caused problems for himself in terms of maintaining
the routing.

"Because the system was so complex (and also because he didn't involve any of
the other network engineers in his unit), Terry was the only person who fully
understood the FiberWAN configuration. Therefore, to prevent inadvertent
disruption of this admittedly critical network, he locked everyone else out. I
know most of the networking equipment ... does use centralized AAA, but I get
the impression he may have configured the FiberWAN equipment for local
authentication only."

Childs' attitude toward other administrators is by no means unusual in the IT
industry. This is generally due to the fact that admins who are tasked with
constructing and maintaining networks of this size and scope care for them like
children, and eventually come to believe that no one else could have the
knowledge and skills to touch the delicate configurations that form the heart
of the network.


Sole Administrator:

A key point made in the e-mail is that Childs' managers and co-workers all knew
that he was the only person with administrative access to the network. In fact,
it was apparently known and accepted in many levels of the San Francisco IT
department. Again, quoting from the e-mail:

"This is where it gets tricky for the prosecution, IMO, because the localized
authentication, with Terry as sole administrator, has been in place for months,
if not years. His coworkers knew it (my coworkers and I were told many times by
Terry's coworkers, "If your request has anything to do with the FiberWAN, it'll
have to wait for Terry. He's the only one with access to those routers"). His
managers knew it.

Other network engineers for the other departments of the City knew it. And
everyone more or less accepted it.

No one wanted the thing to come crashing down because some other network admin
put a static route in there and caused a black hole; on the other hand, some of
us did ask ourselves, "What if Terry gets hit by a truck?" If a configuration
is known and accepted, is that "tampering"?"

My source appears to believe that Childs' motivation was the antithesis of
tampering, and that Childs did everything possible to maintain the integrity of
the network, perhaps to a fault:

"He's very controlling of his networks -- especially the FiberWAN. In an MPLS
setup, you have "provider edge" (PE) routers and "customer edge" (CE) routers.
He controlled both PE and CE, even though our department was the customer; we
were only allowed to connect our routers to his CE routers, so we had to extend
our routing tables into his equipment and vice versa, rather than tunneling our
routing through the MPLS system."

-

Dedicated Engineer:

Like so many other high-level network administrators, Childs seems to have
taken his job extremely seriously, to the point of arrogance and perhaps to the
point of burnout.

"Terry was very dedicated to his career as an engineer. He is a CCIE (probably
the only one in the City government), and spent much of his free time studying
and learning more -- the MPLS for the FiberWAN, VoIP some of the departments
are rolling out, other new technologies for our 311 and E911 systems, etc. He
worked very hard, evenings and weekends in addition to full-time 8-5 work, and
rarely took vacations. His classification is "professional," so he doesn't earn
overtime pay, only comp time -- which like many of us he never really had the
opportunity to use. He was on standby more or less 24-7-365; whereas in the
private sector, in a company of 20,000 or more employees, you'd expect to find
multiple engineers rotating that standby status, I'm pretty sure he was always
the guy on call."

This attitude is, again, not uncommon among high-level IT administrators.
Neither is the fact that they tend to eschew what they perceive to be
unnecessary questioning and bureaucratic "nonsense."

"Terry also, obviously, had a terrible relationship with his superiors. I
should point out that he's not just a network engineer -- he was the lead
network engineer for the entire City. His bosses were all managerial rather
than technical, and while the other engineers did not actually report to Terry,
they did defer to him in any technical matters. Even the network architect left
it to Terry to actually figure out implementation. Terry felt that his direct
superior was intrusive, incompetent, and obstructive, and that the managers
above him had no real idea of what was going on, and were more interested in
office politics than in getting anything done.

"[Childs] complained that they spent more time doing paperwork -- change
requests, documentation, etc. -- than actually implementing or fixing anything
(a common complaint among engineers, I know). He complained about being
overworked (which he was, and which his colleagues are even more now) and that
many of his colleagues were incompetent freeloaders (also not entirely without
basis).

"You could see him getting red in the face whenever he started talking about
his department. And once you were on Terry's bad side (which thankfully I never
was), that's where you stayed, and you'd get only the most grudging assistance
from him from then on. Whether any of his complaints were valid or not, I can't
really say, but I don't think that's as relevant as how Terry felt."


Keys to the Kingdom:

If Childs' sole proprietorship of the FiberWAN network was normal operating
procedure, how did the tensions between Childs and his managers come to a head?
Why was Childs arrested on Sunday? There have been reports that the city's
newly-hired head of security may have pushed for Childs to open the FiberWAN
doors to other admins. My source doesn't know for sure, but offers some
insight:

"I don't know much about his actions in the last few weeks. It's been a couple
of months, at least, since I've even spoken to him, and even then it was
probably only in reference to some specific request or ticket. But I can
imagine that being the subject of disciplinary action by his supervisors for
"performance" issues would be absolutely infuriating to him. I can imagine that
his response would be, "How can you say my performance is poor when I've been
doing what no one else here was willing or able enough to do?"

If Childs was pressured to give up the keys to the network that he had built
and cared for so long, would he go so far as to explicitly prevent anyone else
from tinkering with his charge?

"I can imagine that [Childs'] response to a demand to open up authentication to
the FiberWAN would be, "Why? So you can screw it up and bring the Citynetwork
crashing to a halt?" I can even imagine that, under so much pressure, he'd take
steps (deleting or hiding config backups, for instance) to make sure he was the
only one in control."

These tales offer significant insight into what may have occurred between
Childs and the FiberWAN network hostage situation. Rather than a case of a
rogue administrator attempting to cause damage to the network by locking out
other administrators, this may be a case of an overprotective admin who
believed he was protecting the network -- and by extension, the city -- from
other administrators whom he considered inferior, and perhaps even dangerous.
One important fact seems to be in Childs' favor, if reports that the network
has continued to run smoothly since his arrest are true.. My source
corroborates this.

"As for the impact of [Childs'] actions to the rest of the City, the mayor's
statement basically has it right. The network is completely up and running. No
servers that I'm aware of are affected. No one has had any downtime (yet). But
until they get back into those routers, they can't make any changes. I don't
know yet if Terry's lockout applies only to the FiberWAN or also to the other
routers, firewalls, switches, etc. in the City network."

--

Laying the Blame:

My source doesn't appear to harbor any ill will towards Childs for this
situation, and even believes that the city may be worse off with Childs out of
the picture, and that some of the blame should be shouldered by Childs'
superiors.

"It's a real shame. The city is losing a good network engineer -- probably the
best, technically, that they've ever had. Ultimately he has no one to blame but
himself, but it's too bad his superiors weren't better about establishing and
enforcing policies about authentication, backups, auditing, cross-training, and
separation/rotation of duties.

"You'll note the papers have referred to the new information security manager.
It's only been a month or so since the City even had an information security
policy, and even that is a bare, unmodified template from CCISDA that's
awaiting discussion and alteration by a committee that hasn't been formed yet.
(When I asked Terry if we could get a copy of the City's network security
policy some months ago, he told me, "I've been trying to get them to approve
one for years. I've written ones up and submitted them, but they don't want to
do it, because they don't want to be held to it.")"

He also points out that by forcing the issue, the city may have significantly
reduced its ability to use and control its own network.

"The one impact they haven't mentioned is that Terry was one of only two
engineers assigned to special projects and to do major routing changes and
perimeter firewall configuration. The service level, even after they regain
control of the network, is going to be way down, until they can fill his mighty
big shoes."

My source had many good things to say about Childs, but did not shy from
negative comments, noting that Childs has a bad temper and can be very
defensive.

"As for Terry's character, I can imagine this happening. He takes great
personal and professional pride in his work -- to a fault. He can be very
defensive if someone suggests there's something wrong with the way his network
is set up, and that's been a problem for us (as his customer) a couple of
times. Terry has a bad temper.

"He's the sort of person who, while his bile is up, won't budge an inch -- and
then will call you a couple of hours later and acknowledge that maybe your
suggestion was right, after all, or maybe here's an even better way to handle
things."


The Inner Sanctum:

Later in the e-mail, my source offered some insight into what may be at the
core of the issue: Childs was so paranoid about the security of the network
that he even refused to write router and switch configs to flash, which would
mean that if the device was powered off, all configurations would be lost.

"At one point he was concerned about the security of the FiberWAN routers in
remote offices, so he had them set up without saving the config to flash. "If
they go down, I'll get alerted, and connect up to them and reload the config."
Great, except we have power outages all the time in this city, some of those
devices aren't on UPSs, and what happens if you're on vacation? And what about
the 15 to 60 minutes it might take you to connect up and reload? He eventually
conceded and (ahem) decided that disabling password recovery was sufficient
security."

If Childs did this with some or all of the switches and routers comprising the
FiberWAN network, then password recovery without significant network disruption
becomes a bigger problem. Without first-hand knowledge of the state of those
routers and switches, there's no good way to know, unfortunately.

If the details given to me in this e-mail are accurate, it would appear that
this case is not nearly what it seemed originally. Perhaps it comes with the
pressure and responsibility of the job, or the belief that the network they've
built is simply too complex for mere mortals to comprehend, but it's not
uncommon for highly skilled network administrators to become overprotective of
their networks, or for networks of significant size to become an extension of
the person who built them.

It certainly appears that Terry Childs believed San Francisco's FiberWAN
network was his baby, and that by refusing to allow others to access the inner
sanctum was in the best interests of the city, the citizens, and perhaps most
importantly, himself.

---
-30-
____________________________________________________________
You received this message as a subscriber on the list:
     governance at lists.cpsr.org
To be removed from the list, send any message to:
     governance-unsubscribe at lists.cpsr.org

For all list information and functions, see:
     http://lists.cpsr.org/lists/info/governance