Some IT Disasters

From the dawn of age of the large-scale, electronic, re-programmable computer to today’s 64 core speed demons, the history of Information Technology has been characterized as much by the blunders which have been the result of misjudgements and incompetence as by the achievements of genius and far-sightedness.

Such blunders have sometimes ended in costly mistakes for the people involved, sometimes in farce and sometimes even in ultimate triumph. against the odds

This collection of howlers recounts some remarkable stories and examines the kind of problems that can lead to disaster, ranging from the daftness of Governments and shortsightedness of corporations, to inadequate training.

The Mailshot in the Foot

In the distant days of 1993, when hundreds of small banks closed or merged and many streamlined their operations there was a lot of pressure on banking employees to devise new ways of attracting and holding onto customers.

Dave M had worked in the City of London for a number of years as a senior programmer and was highly respected in the industry.

That July he had joined a financial institution in the City as a senior computer programmer and just about his first job was to meet his new IT boss, and the Head of Communications.

He had to explain to them the workings of a new mailshot program devised by the person whose job he had taken. The bank was pretty keen to get things right since a month beforehand, a rival high-street bank, the then National Westminster Bank (now Nat West), had been forced to confess to irate customers that all sorts of personal information about them – such as their political affiliations – was stored on its computers.

There was no need to worry, Dave assured his new employers. The program was pretty simple stuff and easy to work. It would automatically select 2000 of the bank’s richest customers and invite them to buy all sorts of wealth-creation services.

Alas, there was precious time to tinker or properly test the system; and anyway the Head of Communications was confident the previous incumbent had already fully tested it, and that the program worked perfectly.

What he wasn’t told is that the programmer had indeed tested the system but had used an imaginary customer called ‘Rich Bastard’: unfortunately (for Dave that is) all 2000 letters were ultimately addressed and mailed out to ‘Dear Rich Bastard.’

No one realized what had happened until a day later when an ex British army general phoned the head of customer services and threatened to disembowel him with a prized bayonet he’d brought back from a foreign campaign as a souvenir several years before.

Dave, the hapless programmer didn’t even have time to dunk his morning biscuit into his mug of tea before he was frog-marched out of the building (which has to remain nameless) by three burly security guards. He now works as a vet.

The British Government and the Need for First-Aid

Although the British Government’s ill-starred National Health Service Modernization Programme may have all the hallmarks of one of the worst IT disasters in history with its $10 billion (and rising) overspend, it is reassuring to know that other Government departments are just as inept.

Last year the Cabinet Office was forced to pull one of the public service videos it placed on YouTube after it managed to violate its own copyright. The video called Transformational Government was taken down within 24 hours and was replaced by a box which stated:

‘This video has been removed at the request of copyright owner COI Television [the Government’s Communication office] because its content was used without permission.’

COI Television is an integral part of the Cabinet Office, so their own initiative fell at the first hurdle due to poor communication between its in-house communications unit and, er, itself.

The embarrassing case of the missing video came just a week after the Cabinet Office was praised for embracing modern means of communicating through the web.

A red-faced Cabinet Office spokeswoman said that the posting on YouTube was a ‘small-scale trial or experiment. Trials are meant to flush out a whole range of issues precisely such as these. Rights for online distribution are notoriously complex. We are in discussions to rectify the situation.’

Two weeks later the Home Office topped the COI episode by admitting that 2,700 people had been wrongly labeled criminals following errors in the database at the Criminal Records Bureau (CRB).

The blunder was uncovered by The Mail on Sunday newspaper, which discovered that the innocent 2,700 – ranging from an 87-year-old great grandmother to students – were refused jobs, passports and university courses after wrongly failing CRB checks.

The police checks are usually carried out to vet people who apply to work as teachers, or any other jobs involving children or vulnerable adults. But in some cases innocent people failed a check because their own details were similar to those of someone else with a criminal record on the CRB database.

Many of those who were incorrectly identified as having criminal convictions, ranging from theft to violent robbery, have successfully appealed but still had to go through the humiliation of a trip to their local police station to undergo a fingerprint check.

The Home Office said it would make no apology for erring on the side of caution saying that those wrongly labeled only accounted for just 0.03 per cent of the nine million checks carried out since the system was installed in 2005.

Software Conspired To Help America’s Most Wanted

If the UK Government acted as if they were keen to increase the number of violent criminals on its computer screens, the FBI appeared to be having difficulty in tracking the ones that America already had.

In 2005 the FBI struck off its problem-plagued multi-million dollar Virtual Case File System (VCFS). The VCFS was a custom made piece of software that was supposed to let the Feds search and share information across multiple criminal databases in the hunt for dangerous villains in the wake of 9/11. Unfortunately it did nothing of the sort.

The failure of the VCFS, part of the Bureau’s Trilogy modernization service, can be put down to the simple fact that the technology was out dated before it was completed.

None of the nine-strong team of project managers had the foresight to question what the tech environment would look like or what technology would likely be in use once the build had ended.

In planning the VCFS in 2001, the FBI developed their own custom developed software, even chose to implement their though cheaper, far more flexible off-the-shelf applications were coming into favour among larger US corporations.

‘The pace of technological innovation out-witted the original version of the application. There are far more products on the market that were not around when Trilogy began, ‘ officials admitted last year.

Much to its credit, the Bureau is going to give the idea another go. Although it sensibly jettisoned the case file idea, it has instead settled in favour of an information management system called Sentinel – so that should please the taxpayer.

Being web-based, operators hope that Sentinel will be quick, robust, easy to administer, intuitive and, most importantly, future-proof. Additionally, it is expected to have the added benefit of integrating different database technologies (relational and freetext) – something unaccountably missing from the old technology.

IT plays silly-burgers at McDonalds

Six years ago the ubiquitous American fast-food chain McDonalds undertook a project so vast in both scale and scope, it proved impossible to fulfill.

Codenamed ‘Innovate’, McDonalds wanted an intranet platform so sophisticated that McExecutives at company headquarters in Oak Brook, Illinois would be able to see in real time how many millions of apple pies were being sold globally, if the 3 minute service goal was being observed say at any one particular restaurant in Istanbul or if the chip fryers in a McDonalds in Manhattan were on the blink.

The company hoped that a global ERP application would improve the speed of its day-to-day operations.

Accordingly its accountants drew down $1 billion for the rather ambitious scheme.

Twelve months later it was scrapped. Innovate had cost the firm $170 million in R & D and consultancy fees, though some analysts suspected that several other pet technology projects were buried in the multi-million dollar loss.

A dry internal memo sent out that December from Vice Chairman Jim Skinner to employees said the decision to kill Innovate was based on the company’s recent financial difficulties.

‘We have been screening our initiatives based on their customer impact and their ability to deliver financial benefits to the system in the near term. With that in mind, we have made the very difficult decision to immediately stop all work on Innovate.’

That McDonald’s first foray into large-scale, real-time data systems failed comes as no shock to some experts such as Professor Andrew McAfee at Harvard Business School.

‘I doesn’t surprise me a bit that a company like McDonald’s, with its relative lack of experience in this area, spent so much money and has so little to show for it. It happens all the time. The fact that McDonald’s isn’t exactly on the cutting edge in terms of technology and there’s little executive-level appreciation or understanding of technology surely made matters worse,’ added McAfee.

When Great Just Isn’t Good Enough

But even those who have an appreciation of technology, sometimes get it wrong. Take Google for example. Several times last year, staffers managed to make company look rather dumb.

Not only did they accidentally delete the company’s official blog but also their neglect of site security allowed an unauthorized user to temporarily take over the Web address.

This faux pas could have caused chaos for the company since the official blog is one of the company’s main communication vehicles and often contain corporate information such as news reports, analyst recommendations, and investor decisions.

Luckily for Google, the 19-year-old University of Texas student who snapped up the address did not have an inclination to cause harm. His only posting read ‘Google, fix your blog pleeasssee! P.S. Just to clear things up, I’m not associated with Google at all. I just wanted to take advantage of this before someone else with less worthy intentions did.’

The Google errors are particularly damaging to a company whose stated mission is to ‘organise the world’s information and make it universally accessible and useful.’

Not only could the user have distributed misinformation about Google, he could have used the site to propagate malware, and done substantial damage at breakneck speed.

It was the latest in a line of embarrassing gaffes made by Google employees while handling company data.

While no large firm is immune to mistakes of this type, Google experienced more than its fair share last year Outsiders began to ask questions about the competence of some Google staff after an employee accidentally posted a confidential financial forecast on its Web site, which negatively affected the company’s stock price.

This in turn prompted the company to file a note with the U.S. Securities and Exchange Commission saying the information was out of date, confidential, and in any case shouldn’t be relied upon for financial planning purposes.

A month later Google again scrambled to remove presentation slides from its Web site because they contained sensitive commercial information about possible future products.

The slide presentation had been put on the web to complement those given by Google’s top executives, including chairman and CEO Eric Schmidt, at the company’s annual meeting with Wall Street analysts.

What Would Happen If I Pressed This Button?

It is true to say that  computer technology in government hands has sometimes resulted in terror, despair and a heightened fear of sudden and unpredictable death, but mostly, it seems, to its own electorate. And since governments are charged with protecting and uplifting our lives (and still have a none-too-impressive record on IT cock-ups), we’ll end with a notable selection of their technology blunders.

Seven years ago the UK Government’s school inspection agency Ofsted was forced to delay the launch of its flagship software project after a series of embarrassing failures. The electronic inspection notebook – known as Wizard – was developed by an in-house team of software engineers to make life simpler and easier for the downtrodden school inspector. It was intended to enable them to electronically file reports, which would then be emailed, to Ofsted’s head office.

But after various templates failed to work, the project was dubbed ‘unworkable and useless’ by contracting companies who employed inspectors on a short-term basis.

‘There were so many significant problems with the software,’ one inspector told me. ‘The program was easy to install but hopeless to use.’

Then there was the time when 80,000 staff at the Department for Work and Pensions was unable to use their PCs for a whole week after a routine software upgrade knocked out 80 per cent of the PCs in the sprawling department. This disruption was on top of the failure of a £450 million IT project at the Child Support Agency and the collapse of a £500 million benefits system.

Still it’s not all bad news for us in the UK. Our Government assures us that a move to e-government could save £300 million over 5 years. What a relief.