The Database Mole Secrets of successful IT projects

“The project was two years late and three years in development,” he said. “We had 30 people on the project. We delivered an application the user didn’t need. They had stopped selling the product over a year before.” – Sid, quoted in the 1994 Standish Chaos report.

Some time ago, I worked on a team that was attempting to define an IT strategy for a large multinational company. Management incorrectly thought the company was unique in that it had a high number of failed IT projects. In fact, only one in four IT projects can be termed successful, if the benchmarks of success are adherence to scheduled completion time and budget, and realization of the project goal, whether that be saving money, growing the business, or what have you. I have to explain this because many of you may never have come across one.

We spent a long time looking at current and recently completed projects within the company’s facilities. As the company had shied away from a single corporate strategy, I had an education on the different ways an IT project could be undertaken. One common thread was that projects were almost invariably late and over budget, despite their innovative designs and use of pioneering technologies.

Later I got wind of a group of projects that the company’s business users universally loved. They weren’t flashy but they did what they promised, meeting all the criteria for success. They also all came from the same out-of-the-way IT department.

When I got inside the building and was being shown around this particular department I noticed something curious. The programmers were slightly plumper, and had a tendency to baldness. Many wore tweed jackets, even the ladies. A software house ran the department under contract. I eventually discovered, to my astonishment, that the staff was programming in Cobol with a Sybase back end, using a fancy windowing interface to disguise the fact.

They implored me not to give up their secret, and I can happily say they were churning out successful applications with almost no visibility to senior management when I left the company some years later. For all I know, they are still there.

I discovered during that exercise that there are a number of reasons given for the failure of IT projects, most of which are bogus and designed to distract attention from the real causes. In contrast, there are just a few reasons why they succeed. The experience taught me that, instead of attempting to improve our success rate by identifying what had gone wrong with certain projects, we should look at how and why other projects succeeded.

Arguably, Cobol should not play a real part in a modern IT project, although, like SQL, it is grounded in and inspired by the processes of commerce. There are, however, certain rules for software development that consistently hold true:

• Decide what needs to be done – Sit on your hands until everyone buys into what they have to do and how long they have to do it. Then operate rigid change control.
• Don’t be a pioneer – Insist on and stick with technology you know will work.
• Code, then recode – When you’ve got the code working, throw it away and rewrite it properly.
• Discourage virtuosity – Never let a programmer do something you cannot understand.
• Encourage results – Reward work that meets scheduled completion dates and specifications. Politely ignore excuses as to why things that should have happened didn’t.
  
• Don’t overload the lifeboat

Scope-creep is a euphemism for the stacking of requirements on a development project that looks like a runner in the corporate environment. Until a project gets sign off, it is generally ignored within the enterprise, but afterward it becomes the focus of frenzied attention. From a management viewpoint, it could be the company’s last chance to achieve business efficiencies through technology.

From the IT management perspective, resisting the pressure to turn a molehill into a mountain is difficult. Imagine yourself being in charge of a lifeboat, beating away the drowning with an oar to prevent the boat from sinking. When you try to argue that the addition of functionality cannot be achieved without resources, you have a similar reaction to deliberately bashing someone’s fingers with an oar. The rejection of new changes into your project plan is necessary, however, since an overloaded, constantly changing, project will go nowhere.

Don’t be fooled by an impressive-looking requirements document from the business side. The true nature or complexity of the company’s business processes is not often shared, even when management knows what it is.

I was once involved in a corporate re-engineering exercise that spent roughly $17 million on a requirements-gathering exercise. For months, well-groomed young graduates in dark suits earnestly identified processes and procedures. The resulting report filled a large steel filing cabinet. I was the first to, er, skim through the document, and I believe I was also the last.

I took one department’s requirements back to them and challenged the staff to admit, without blushing, that the document reflected reality. After a bit of blustering, they confessed that their contributions were mostly fabricated to make their role in the project-and the company-seem more important. For them, re-engineering was nothing more than redundancy.

Determining what the IT development staff should do is the hardest part of a project, and no code should be cut until that job is done.

Use boring, proven technology

Using a proven technology may sound radical. I always hear gasps of shock from J2EE enthusiasts when I mention this rule. IT departments fall prey to the idea that perhaps, this time, if we try something new, things will be better. It is also comforting to blame old technology for the failure of IT systems.

Beware of misguided attempts by business management or the project sponsor to decide on the technology before the project starts. This is like choosing the wedding dress without benefit of a suitor. Invariably, this happens when management is successfully seduced by the technology supplier’s sales claims.

If the customer is the enemy without when defending the due process of selecting technology, the development team is the enemy within, because developers have their own agenda.

Many developers have mixed feelings about a conservative choice in technology. The industry values programmers for their current skills, so their prime objective is to hone those skills to increase their market worth. If the project they are working on doesn’t require a so-called hot skill, they will use their spare energy to make sure it does.

It may not be logic that fires the hunger for the latest technology. One of the strange side effects of testosterone is an irresistible urge to play with the latest technology, and it is hard to distinguish real productivity from unashamed self-stimulation.

It takes courage and willpower to adopt a relatively conservative approach to development work and select the technology appropriate for the application. When I plan a project, I generally construct a “playpen” area in which all the latest technologies are used to develop a non-critical project component, and then let the team take turns developing it.

The staff gets to put all sorts of skills and technologies on their CVs, and the pressure is off to prematurely adopt fancy technology for the project’s serious deliverables. We can then inform the project sponsor that we are using glitzy technology just like in the advertisements, and everyone is happy.

Code, then recode

Throwing away code, just when you’ve gotten it to work, may seem unkind or unnecessary. It is actually a deeply cleansing experience, like sloughing off dead skin. I discovered this by accident when the bad luck fairy struck and no one had made backups.

Initial feelings of despair were followed by a curious lightness of spirit, as all the false turnings and cul-de-sacs of everyday coding were forgotten. We knew what needed to be built, having done it once, stumbling in the half-light of the systems analysis.

When we keyed it all in again, it was half its previous length and ran twice as fast. It didn’t take long to do it either, and we hummed as we worked. I’ve since discovered that the same applies to any creative work. Tentative work should always go on the fire.

Discourage virtuosity

The principle that you should never let a programmer do anything you cannot understand is a classic one first articulated by C.A.R. Hoare, inventor of the Quicksort algorithm. It is ignored at your peril.

My worst experience with this behavior was when I was supervising a freelance Sybase programmer who created a reporting system for a financial services company. He used dynamically compiled stored procedures that were created in response to the exact slice-and-dice query required by the financial analyst.

He almost got the code working to the satisfaction of the business, and then demanded a doubling of his contract rate. We parted on bad terms, and I was left with one of the scariest and most opaque bits of application code I had ever come across in my waking hours. To make matters worse, the business sponsors were baying for impossible improvements, bug fixes and extensions to the scope of the project.

Reward results

The idea of rewarding a programmer’s productivity may seem unfair and unreasonable in a dignified profession. On the contrary, it is as difficult for programmers to gauge their own productivity as it is for them to test their code, and a monetary reward sends a clear signal of a job well done.

There is always one programmer on a team who works hard and does heroic coding, but achieves very little. The objective of producing an application that satisfies the business requirement is lost in the abstractions and obfuscations of the code.

It is as if they get sucked into a Tolkienesque universe. There have been times when I’ve half expected to go to work and see nothing but a pair of feet sticking out of the screen. Now that physical violence against programmers is frowned upon, one way to jolt them back to reality is to be selective but generous in the distribution of bonuses.

Been there, done that

Unless it is thought that I speak with the sanctimony of virtue, I must confess that over the years I have either watched or participated in almost every mistake that can be made in IT. One of my worst experiences occurred when the company I worked for got a contract with a government department embedded deep within what was then called the Ministry of Agriculture.

Our sales rep got overexcited after a meeting with government officials and priced out the job as a trivial application involving the automated sorting and grading of vegetables with its associated simple accounting. Instead, it turned into a monster application that was to be released nationwide as a major government initiative.

The scope of the project grew faster than the development work could be completed. Interfaces and dependencies grew like field mushrooms. To make matters worse, the hardware, selected by government dictat, was unreliable, and the macros used for process control were almost impossible to debug. The software, a version of BASIC, was proprietary to the hardware and innocent of database functionality.

The day of the demonstration loomed. The government mandarins and representatives of the farming community were soon scheduled to come and admire the efficiency with which the application could read barcodes; grade, weigh and sort produce; feed the data into the computers; and produce some impressive paperwork. We were up to our armpits in unfinished code and non-functional interfaces. Then we were summoned to an important meeting.

With trepidation we put on our best suits, as if going to our own execution. To our surprise, the ministry representatives were extraordinarily apologetic. They broke the news to us that their department was to be closed down in a government reorganization and that the project would be canceled.

They mistook my flushed face with crazy and wet eyes for signs of sadness. “Don’t worry, Phil,” I was told. “You’ll be paid in full, so you won’t suffer too much.”

Perhaps there is a patron saint for software development after all.