All Circuits Are Currently Busy — A Look Back 20 Years After AT&T Network Crash
Bug-iversary Alert! Tomorrow is the 20-year anniversary of the “crash” of the AT&T Long Distance Network. On January 15, 1990 faulty software was installed on the AT&T Electronic Switching System (Number 4 ESS): a one-line bug incapacitated the entire system, disabling switches throughout half the network.
Known as one of the most serious telecom bugs in history, more than 75 million calls were not connected during 9 hours, an estimated $60 million loss.
Dennis Burke of California Polytechnic said it best: “The Jan. 1990 incident showed how bugs in self-healing software can bring down healthy systems, and the difficulty of detecting obscure load- and time-dependent defects in software.”
Speaking of “load defects,” AT&T — after signing up to be exclusive U.S. provider of iPhone service — has recently come under fire for the quality of its network coverage. Businessweek‘s top headlines read:
- An AT&T Mystery: Abrupt New York iPhone Shutdown
- AT&T Mulls Plans to Deal with iPhone Data Demand
- Can AT&T Meet iPhone Network Demands?
In light of this bug-iversary, I can’t help but wonder if more testing should have been done before AT&T took on the massive data demands of modern 3G smartphones? What do you think?
I’m not sure a competent load-test paradigm has been designed. Can AT&T (and now T-Mobile) really simulate the millions of people dragging down their data networks?
With ESS, we’re talking about voice rather than data. These days they’re practically the same thing, but back then there was a lot more voice traffic. And the nice thing about voice is that it’s more predictable and consistent in its needs.
Voice traffic can be well modeled with Markov chains, and simulation is also pretty straightforward. The phone companies were some of the first to really use Markov chains for this purpose.
Data modeling and testing is tougher, but high scale load testing is still possible. However the leaders in that science are the router vendors: Cisco, Nortel, Juniper, etc.
[...] All Circuits Are Currently Busy — 20 Years After AT&T Crash [...]