Having dominated the news for the last month or more BP’s problems with one of their oil exploration endeavours is a problem that will affect the whole planet.

Now I dont know the whole story, either how it all happened or what is being done to restore some sort of control (I have heard about shredded tyres and golf balls being pumped into the hole but am not sure if that was a satirical program mocking the situation or an actual event . golf balls . seriously!? ).

I started wondering what the other oil companies are thinking; My first thoughts of the Directors at Shell – “Phew, glad that isn’t us!”, followed by “How’s our share price? How’s theirs?”. I would also put a £ to a penny that the thoughts change as they go down through the organisation until you get to the person in the same position as the BP employee currently ‘holding the baby’, where they are “What if that was us? How would we cope in that situation? Would we struggle like that?”.

If they aren’t then they certainly should be, its only luck that separates the two positions.

Now this scenario, whilst probably having a less severe impact on the environment, applies to you too. Have you ever been on a tech forum and read a question along the lines of “I came in this morning and the server wont start” or “I can see my database in SSMS but it is marked suspect” or “My main database server has a failed hard-drive in the RAID array” – someone having their own BP moment? Sometimes you may have volunteered a piece of advice or assisted in some way to help them out of their predicament or you may have sucked air in through your teeth and quietly been thankful it was them and not you. How often do you think about whether/how you would be able to cope if you were in the predicament? If you regularly go through your disaster recovery process in order to prove your the planning and expectations are likely to be an effective guide to getting back to business then you should be both pleased with yourself and confident that you wont get any nasty surprises if the worst happens. If you haven’t or don’t run exercises to use the plan as written to recover a system to a backup location then you should.

Now.

Seriously, right now, come back when you have done it.

If you use only the DR plan to run your practice to recover a system then you are not only getting incredibly valuable practice at running restores, moving logins, re-configuring applications but you are also checking that the plan is effective guidance. In the case of a real event it might not be you doing the recovery work, for what ever reason. You need to know that the plan will enable any tech-aware person to carry it out. It like the old programmers interview question “Explain, in writing, how to make a cup of tea.”. The theory goes that it is a simple process but has to have things done in the right order and with sufficient supporting information that it becomes a good demonstration of someone’s ability to take a process and effectively understand and document it.

If you agree with:

• Put water in kettle,
• Turn on kettle,
• Put tea bag in pot,
• Put milk in mug,
• Pour water from kettle into teapot,
• Wait 5 mins,
• Pour tea into mug,
• Add sugar, stir, drink

then you are pretty good but will be drinking cold tea. You have to make sure the kettle is plugged in ..

Your DR plans need to cover a lot of detail too. I know one site that went in for a DR practice and it ended in 5 minutes. The scenario was to lock the main building, imagining it was destroyed in some accident, and follow the plan to ship out to the secondary location owned by the company. It all came to a premature conclusion when it was realised that the key to the secondary location premises was kept in the safe in the primary location. Full Stop, EndEx. Now the rest of the plan was solid, the technical processes were workable and effective and the business was not, essentially, at risk in this instance – had it been an emergency someone would have sanctioned access to the building by other means and the process would have continued. Its a wrinkle you don’t want to have to deal with when tension and pressure are at a high already though. So, this exercise was total failure? No, the practise was a success. Yes, it was a success; it showed up a problem at a non-critical moment and means that the same issue wont impede a future execution – either if its for real or another practise.