It must have been the lobster we had for supper, but I dozed over my book, and slept fitfully. In my dreams, I bumped into Bill Gates.

“Hi Bill”, I greeted him amiably. “Long time no see”.

“It was 1982 to be precise”, he replied, with that lopsided, curiously adolescent, grin I remembered so well. “And, I’ve been thinking, I was wrong to shout at you when we had adjacent stands at that conference.”

Blimey, I thought to myself, the lobster must have been off.

“My fault entirely”, I replied magnanimously, “for explaining rather too enthusiastically to that senior executive from Ford Motor Company that PCDOS was just CP/M with a bit of make up and frilly knickers.”

“I didn’t mind that so much as your long and amusing speech about Concurrent CP/M being years ahead of PCDOS.”

“It was”, I protested, “but you caught up eventually with NT. A shame about Xenix and OS/2 though”

“Never mind all that”, said my old friend colouring up remarkably, “I need your advice on SQL Server”.

“I must be dreaming!” I expostulated.

“You are”, he replied. “You can’t really imagine that the real Bill Gates wants your advice, can you?”

“I suppose not. Even in 1982, you weren’t too keen. So you really are just an imaginary Bill, dreamed up after drinking too much Vouvray 2002 with my Lobster…”

“Just be thankful it wasn’t a large glass of Chardonnay, otherwise you’d have gotten Larry Ellison.”

“…and if I’d had finished the bottle, it would have been Beelzebub himself, in a haze of sulphurous smoke..”

“OK, enough pleasantries” said Bill, adding after a thoughtful pause. “I have a problem. I spent five years with one of the biggest development teams in history, creating SQL Server 2005. It is now 2007 and a good-sized chunk of the industry is still using SQL Server 2000”. He paused again. “So, we started giving it away as SQL Sever Express, thinking that PostgreSQL and MySQL would just vanish as a result, and that existing SQL 2000 users would happily convert over to Express, and then up to the full SQL 2005. We were wrong on both counts. Everybody admits SQL 2005 is the most comprehensive database in history. It has everything. So what is wrong?”

“Simple. Look at this book I fell asleep over. This is typical. A book on administering SQL Server and it was written by nine people. The entire story of the “Decline and Fall of the Roman Empire” was written by one man, and here it takes nine people just to tell the poor souls who have the unfortunate task of administering the behemoth of a system how to do it. It doesn’t quite say whether they wrote it serially, collapsing exhausted with writers cramp and passing on the metaphorical baton to the next volunteer, or whether they just partitioned off the immensity of the task between them. I find this deeply scary.”

“Scary?” asked Bill, puzzled

“Yup. I’ve got to plough through all seven hundred and thirty-five pages before getting back to re-reading Beauchemin and Sullivan’s 964 pages, and that’s before I get stuck into all the changes in NET 2 that I missed. And I have a stack of books on PHP5 to tackle. If this trend continues, Microsoft’s next offering will require each administrator to read an entire bookshop. Your SQL Server 2005 Administrators Pocket book might fit in the pocket of a Behemoth, but I would walk with a strange limp if I managed to stuff it in my trouser pocket. Authors wrote the great books on SQL Server 7 single-handed. Today’s tomes are as thick as doorstops and are written by about nine authors. By simple mathematical extrapolation, books on the next version of SQL Server will be 2000 pages long and written by a small army! “

“Well, you have to keep up-to-date in any job…”

“Well, I used to think that doctors have the worst job of keeping up to date, but they don’t find the entire human body changing its plumbing every five years. Sure, the drugs change, but aspirin is still aspirin. IT people are finding the entire field of knowledge changing radically every five years. We’re getting information overload.”

“Come now, Phil, it is just that you are an old fart, who can’t keep up any more. You should be growing roses and knitting antimacassars, thinking about nice comfy chairs and joining the bowling club.”

“There may be some truth in what you say…but I don’t think anyone is keeping up. Nobody likes to admit it. It is like owning up to having pitifully inadequate secondary sexual characteristics, or the brainpower of a two-toed sloth. We all like to keep up the myth of our mental adequacy, but it is a myth. It took man 100,000 years just to progress from the first stone tools to discovering how to make metal objects – and in evolutionary terms, we are complete strangers to the computer keyboard! It is a tall order for us to suddenly become the priests of the mystical power of Information technology. It would be all right if we were developing extra brain-lobes to keep up with the galloping complexity, but, if anything, our brains have shrunk slightly since civilization began.”

I warmed to the subject. “The huge breakthroughs in Information Technology were breakthroughs because they were simple, and they reflected the way that business was already being conducted. When the relational database was introduced, people found that it was uncannily like the existing ledgers, and the systems put in place by the nineteenth century logistical experts after the Crimean war, and refined ever since. The Spreadsheet is simply an accountancy summary table, and could be recognised and comprehended by any accounting clerk from the time of the Industrial Revolution.”

“SQL was a staggering advance, because it was so simple. The clever and difficult stuff happened under the bonnet. For one, you told the system what you wanted but not how to go about doing it; for another, you didn’t have to keep a huge number of details in your head. It was just the same with Email or the Internet; dead simple, and easy to understand. The real skill of technology is to keep it conceptually simple. So what have you been doing for the last fifteen years, Bill, but adding complexity to a system whose chief virtue was its simplicity? It requires no thought to create a tower of Babel, a vast beast riddled with Rampant Featuritis; just effort”.

We fell into a gloomy silence for a while, before I continued.

“Whatever anyone asks for in SQL Server is put in, until there are eight different ways to export a query to a file, four ways of extracting data from a spreadsheet, and a Byzantine intricacy in the handling of XML – which, by the way, started out as a dead simple way of representing small databases, before it got the complexity treatment. It is so easy to be obliging and add stuff in. The great skill is in keeping the product simple at the same time. One can imagine it happening. Some persuasive guy contacts Microsoft, suggesting a feature and, without checking if it is already there, a team gallops around in all directions developing a splendid do-it-yourself technology with a new TLA, that ends up being little more than an invitation to program the feature yourself: when all that was really required was a button.”

“And there, frantically rushing around, trying to keep up are the DBAs and Database Developers. You give them the choice between a database like SQL Server 2000 that is complicated, but works and does the job they want, and one that is wildly complicated, but works and does the job they want, and then are surprised when they want to stick with the one that is merely complicated.”

Bill looked at me impatiently. It was obvious that he’d heard this line of talk many times before. “OK,” he said wearily, “So where next? What can one do about it?”

In a blinding flash, it all became clear to me; the vision of a product that hid its complexity behind a simple logical interface to the data. Here was a product where Microsoft no longer needed to tell us to do our own programming to get the features we needed. Here was a SQL engine that could be described in a short book, probably written by Joe Celko, or Bill Stanek.

I was about to tell Bill how to go about it when I felt myself being shaken, and the veil of sleep lifting. “Stop! Stop!” I shouted, “I must tell Bill what to do!”

The next moment I was awake in my chair, eyes glazed, textbook still open in my lap. My wife was standing over me, looking at me severely, and asking who this ‘Bill’ was, who ought to be told what to do. In panic I tried to recall what it was I was going to tell him. Surprisingly, I managed to recall at least a part of the advice that I had to offer.

All data objects, whether Word files, Excel files, XML, relational databases, text files, HTML files, registry files, LDAPs, Outlook and so on, are really relational databases, spanning the spectrum of metadata complexity. All we need to do is to build on all the ideas of ODBC and JDBC to provide the same logical interface to all of them. If we can do that, then SQL Server can treat them all the same logical way, as linked databases, and extract the metadata, create the entities and relationships in the same way, and use the same SQL syntax to interrogate, create, read, write and update them.

The complexity would be in the implementation, hidden from the poor DBA and developer. He would just know that he could create a Word, Access, XML or Excel file in the same easy way he could create a SQL Server database, and transfer data between these esoteric media, being obliged to do no more than specify the medium. Now, there’s a start….

The rest of my advice got lost in the struggle to regain consciousness. Its loss is not important; as I’d be much more interested to read your suggestions on how SQL Server could be made simpler.