When IT “goes wrong”

This is the first in a short series of three commentaries that look at when, where and why IT fails in its support for business operations and objectives.

We have all seen what happens when IT goes wrong …
• the system is not available (usually timed to cause the most inconvenience!) …
• there is a process or calculation error in an invoice (the million-dollar or million-pound consumer utility bill; the final demand letter for "-$0.01c" or "-£0.01p") …
• viruses, worms, trojans, and spam infect PCs and become the tools of trade for fraud and service disruptions …
• projects go wildly out of control, over budget, deliver late results, or don’t deliver the right solutions to the business …

This commentary takes a brief look at some visible examples when IT has gone wrong. The next commentary will look behind the scenes at some of the business and IT processes involved – where IT goes wrong. The third and last commentary will look at why IT goes wrong – some of the root causes, and what you can do to reduce your chances of suffering from the same problems.

Let’s look at some high-profile examples of the situations we listed above. This list is obviously a minute sample from many failures that have made the front pages over the past few years. They are by no means the only high-profile failures, but are sufficient to illustrate the points we want to make during our three commentaries. Nor, for that matter, does the number of failures hitting the front pages give any inkling of the massive number of unpublicised IT problems that organisations suffer. But, again, we’ll use our small sample to provide some useful examples for discussion.

As you read through these examples, reflect on the profound financial and social impacts the failures have had. Also reflect on the fact that this list is a mere drop in the ocean of IT failures and ask why these disasters keep happening – surely we should have learned better by now!

• A pharmaceutical company wrote off nearly $17.2 million in missing funds due to IT "discrepancies". A short time later the CEO and CFO were replaced.

• A drug company was forced into bankruptcy by a series of operational and project blunders.

• A late, over budget system was introduced to “streamline and simplify” importers’ dealings with a government agency. Within days there was a severe backlog of containers in sea ports during a critical period for importers in the lead-up to a holiday season.

• A telco spent $500m+ on billing software. It is still not right, and the same telco has announced a replacement programme. The software made the front pages when it sent erroneous final notices to the relatives of long-dead customers.

• A large bank was pushed by a software vendor into early adoption of an untested new version; the software took out the automated teller machines, then allowed cardholders to withdraw cash without debiting their accounts.

• A government agency applied its annual regulatory changes to old and unstable core systems. The systems first overcharged members of the public, then made too many refunds, then overcharged those who received incorrect refunds, and finally got it right on the fourth attempt.

• A major new stock exchange system was 11 years late and 13,200% over budget

• A new emergency services system was introduced on time and on budget, but the system and its backup locked. The emergency service (in one of the world’s largest cities) reverted to a manual system that restricted the ability of the service to respond quickly and as a result placed lives at risk. (This debacle was repeated by an emergency service in another large city on the other side of the world only months later.

• An IT infrastructure upgrade increased in cost by 150% (from about $US 2Bn to about $US 5Bn – only 18 months into a ten year project.

• The opening of a new airport was delayed 16 months by late delivery of revolutionary software. As a result the airport's planners’ bond rating was demoted to junk and the organisation lost $1.1 million a day in interest and operating costs.

• Most of the desktop computers in a government welfare agency were paralysed for four days when a failed operating system upgrade took them offline. The outage, covering 75 percent to 80 percent of the agency’s 80,000 PCs, was one of the largest in the country’s history. The outage disconnected staff e-mail, benefits processing, and connectivity to critical information and systems.

• In the same country a “computer crash” in another organisation prevented pensioners from collecting benefits payments.

• In yet another organisation in the same country a call centre and its systems for processing applications for welfare payments ran so slowly that “up to two thirds” of callers (in at least one region) were unable to get through, and there is evidence that once through, payments took up to six weeks after applications were lodged. (Presumably, if people are needy they actually do need the payments as soon as possible!)

Next: Where IT goes wrong