To simplify enormously, one could say there are two main ways of developing new IT systems – ‘big bang’ and ‘iterative prototyping’. With big bang, you may do a small pilot just to iron out the bugs, then you put in massive resources (often hundreds of consultants), build the system and roll it out to the users.

Big bang satisfies the politicians’ need to show things are happening and it reassures the civil servants that the whole thing will be delivered ready for them to press the button and off it will go. It also provides admirably large levels of billing for the consultants right from the start of the project. But already in 2000, after reviewing twenty five government IT projects, the Public Accounts Committee recommended that, ‘an incremental, as opposed to ‘big bang’ approach to IT projects should be adopted, with regular milestones, each delivering an auditable business benefit’

The main problem is that ‘Big bang’ tends to be inflexible to the needs of users. The system is designed and built using hundreds of consultants and at massive expense. If anything has been misunderstood during the design (as always happens) or if circumstances change during the build phase (as always happens), big bang systems developments are usually unable to adapt and so tend to bulldoze their way forward ignoring those bits of reality that do not fit their preconceptions.

With iterative prototyping, on the other hand, you start small with a basic working system – ideally based on an already working system – in a couple of test locations. You involve users and gradually build on more functionality. You go through a whole series of iterations – involving the users and adding even more functionality – until you have a system that meets user needs. Then you scale it up and roll it out. With iterative prototyping you can involve users, continuously adapt the system and make friends. But iterative prototyping can look messy to the politicians and civil servants because you don’t appear to have as much central control as you do with big bang. Moreover, the really big consultancies try to avoid iterative prototyping as it starts small and so doesn’t provide huge fees up front.

One of the great unsung advantages of iterative prototyping is that it is impossible to spend many tens of millions before you find out if the system is going to work or will have to be radically adjusted in order for it to work: the main problems are usually discovered after just hundreds of thousands have been spent. With big bang, as scores of previous government efforts have shown, it is usually not till tens or even hundreds of millions of our money have gone up in smoke that the problems are discovered.

A frightening number of large organizations, particularly government departments, simply have no idea how to buy large computer systems developments nor how much they should pay. This makes them easy prey for the large systems houses eager to sell massive programmes to their clients.

A project manager at one large US-based IT consultancy once talked about its AFAB (Anything For A Buck) culture and said that in his long career he had only ever met one client who had skillfully managed his IT systems suppliers. When we discussed selling to the public sector, he just laughed and talked about how easy it was to get them to buy whatever the consultancy wanted to sell.

At an IT industry conference, the lead legal counsel from one of the world’s top three systems suppliers talked about British government buyers of IT being ‘inconsistent, inexperienced and incompetent’ and joked about how this allowed the consultancy to earn huge amounts of money.

Too many IT systems are still developed based on what we call ‘Time & Materials’ contracts. On Time & Materials contracts, the supplier is paid on the basis of how much time and resources they use – so the more time and resources they use, the more money they get. Thus they have no incentive to finish any systems project within a tight timeframe or with carefully controlled use of resources. If more were on contracts with a fixed price and penalties for cost and time over-runs, then many more IT systems might see the light of day faster and at much lower cost than is the case now.

Many other IT systems projects may appear to be based on a fixed price and fixed timeframe contract. However, when you look at the small print, you tend to find a whole series of ‘Get out of jail free’ clauses – clauses that, at the slightest excuse, allow the system provider to claim that the client has made changes to the specification and so the suppliers continuously increase the cost of the project and the time taken. These projects are often called ‘partnerships’, although they are rather unequal partnerships as in every single case the consultants get the money while the government gets the systems disaster.

At an IT conference, a commercial lawyer, who had served as an adviser to the Customs and Revenue and the police, described these partnering contracts as ‘a licence to permit cost-plus and death by change control’. He meant that on supposedly fixed price government contracts, you could always increase the price as much as you wanted by continuously claiming that you had to make changes for which the client should pay extra on a ‘Time and Materials’ basis.

Hence we see most systems providers making stupendous profits year after year in spite of the fact that many of their projects end up costing hundreds of millions more than initially budgeted and taking years longer than planned, if they ever work at all. And, of course, with government IT projects, every few hundred million squandered on the latest fiasco mean a few hundred million less that can be spent on providing essential services like health, policing and education.

British New Labour are pouring so much money into new IT systems as the ‘silver bullet’ for reforming the way public services are delivered, that most systems consultancies are faced with a major problem – where to find competent consultants with relevant experience to staff their huge new government projects.

Some consultancies are seeing their business double, triple, even quadruple every year as they get ever more public sector work. This places a massive strain on these companies to meet the promises they have sold. In fact, the experienced consultants these companies so desperately need, don’t exist. So the IT systems providers are having to make do with whoever they can get.

I have seen many IT systems consultancies, while doing public sector work, start up their projects with as few experienced people as possible, then when the project is running, quickly move the few experienced staff on to their next project. Naturally, the results of this are disastrous as project after project overruns on time and cost due to the lack of experience of the people left working on them. Then when public-sector clients start to raise their concerns about lack of progress, the consultancies try to buy time by muddying the waters – for example, by blaming their clients for the problems, accusing people they are working with of delays and incompetence, claiming their clients did not make their specifications clear and so on.

The consultancies know that few of their public-sector clients have the knowledge or willpower to hold them to account for their failure to deliver and so usually get away with enormous time and cost over-runs. In many cases, they even succeed in charging their public-sector clients extra money to fix the problems that were caused in the first place by the consultancies’ blatant lack of experienced staff.

Although their products or services may be very different, most organizations do fundamentally similar things. They make or buy something, put it in a box and sell it. Or they make or buy something put it in a shop or website and sell it. Or they transport passengers. Or they handle people’s money. In the case of government departments, they take some skill, knowledge or service (cancer surgery, taxation, social services) provide it to citizens and manage their own costs and budgets. So most organizations, in the private or public sectors, need similar things from their systems – acquiring products or skills, attracting customers or identifying the people they should serve, managing information about customers or citizens, cost control & billing, HR management, financial accounting and so on. For example, the £450m ($630m) system for the Child Support Agency is unlikely to be hugely different from the types of systems used by debt collection companies. Yet, as the hundreds of millions price tag, the system’s failure and the ensuing administrative chaos all show, a whole new system was built rather than an existing system being adapted. Moreover, at the start of 2006, the Government considered handing many of the CSA’s responsibilities over to private debt collection companies who use precisely the kind of IT systems that the CSA should have adapted for its use in the first place. In another case of extravagant wheel-reinvention, the Ministry of Defense (MoD) abandoned an IT systems development project after spending 8 years and £41m ($57m) to achieve nothing. The MoD then bought an adapted off-the-shelf replacement system for just £6m ($8). Similarly, after three years of largely futile work on developing a new IT system, a local council found to its surprise that small changes to its existing system would give results as the new system had so far failed to provide.

It is simply not in the IT systems consultancies’ interests to provide you with a fast, cheap adaptation of existing systems when they can earn hundreds of millions by convincing you that you need your own specially designed new system.

The profit margins made by IT suppliers and consultants are eye-watering.  A junior consultant, what we in the business call ‘billing fodder’, will probably be paid about £600 ($840) a week but can be charged out to clients for at least £4,000 a week ($5,600), giving the consultancy a margin of over 600%. Similarly, a project manager being paid £1,500 ($2,100) a week can be billed at £15,000 ($21,000) a week. Moreover, now that many consultancies do their basic programming in countries like India, where wages are a fraction of those in the West, some IT suppliers will be achieving profit margins of over 1,000%.

 A slightly depressing aspect to the whole British Government IT systems debacle is that our representative, the Government, has not only utterly failed to manage its IT systems projects in a professional way, it has also failed to enforce its own legal rights as a paying customer of the IT systems consultants.

In both the U.S. and Britain, case law around the duties and obligations of IT systems suppliers and consultants has undergone a seismic shift over the last ten years or so. In the early 1990s, buying systems was very much a case of Caveat Emptor (buyer beware) and organizations seldom had any right of recourse against their IT suppliers, however disastrous the results of those suppliers’ work. A series of over one hundred legal judgements have since changed this to something that more resembles Caveat Vendor (supplier beware). For example, IT consultants must inform clients whether their business analysis actually meets their operational requirements, whether its systems will meet clients’ operational requirements, whether both the client’s and suppliers’ staff are capable of successfully completing the project and whether the buyer’s original budget is sufficient to meet their objectives.

Unfortunately, government departments and their expensive lawyers do not seem to have a clue about their legal position when setting up contracts with IT systems suppliers. In an analysis of over 500 IT systems contracts, many of them done by government departments, one group of specialists did not find one single contract that effectively took account of the ‘duty of care’ and other legal obligations of IT systems suppliers. Moreover, when systems suppliers clearly fail to meet their legal obligations, the Government virtually never seeks to enforce its legal rights.