There is an old joke that circulated amongst IT professionals during the 1980s – this joke goes as follows. A man goes up to an ATM puts his card in the machine and requests some cash. The machine accepts his card and PIN but doesn’t give out any cash. He goes into the bank and tells a cashier what has happened. The cashier replies – “that’s strange because we just had brand new software installed this morning”. This joke is probably not funny if you bank with RBS in the UK.
I normally write about IT security issues so – why is it that this entry is about managing change. Well - security is about confidentiality, integrity and AVAILABILITY. Good IT security ensures that you have access to the information that you are entitled to whenever and wherever you need it. One of the most frequent causes of non-availability is poorly managed changes. In the world of software – a change is often a change for the worse.
The older the software system the more difficult it is to patch and most of the retail banking systems are very old. The people that originally wrote it may be long gone; the change you are applying is probably on top of many previous changes. You did your best and it looks like it should work but unfortunately you didn’t fully understand the complex interactions that now exist within the program. So you test it, and your test contains all the expected cases plus all the previously detected bugs that have been fixed. However these tests don’t include every possible case and so when it goes live – whoops the impossible happens and the system crashes. If you are lucky this unlikely event only causes minor damage. If you are unlucky – as seems to have been the case with the RBS systems – this unlikely event causes major damage. It becomes the nightmare of IT security: a low probability, high impact event.
Now you have to recover from the problem. Can you roll back the software to the last working version? Are you able to restart or re-run the failed transactions? How can you make sure that you don’t repeat the successfully processed transactions? You need to have planned for all of these contingencies BEFORE you applied the change. You need to have tested your plan BEFORE you applied the change.
Now it may well be that RBS did all that it could and should have done – only a detailed investigation will reveal whether there were avoidable shortcomings. Nevertheless RBS’s experience should be a reminder to all of us in the IT industry to be careful about managing change to IT systems. It shows the need for IT professionals to really understand the impact they have on the business.
The fundamental role of IT within an organization is simple to describe: It must provide the IT services that business requires in the way business wants them – nothing more, nothing less. Unfortunately, many corporate IT departments tend to concentrate more on technology than on the needs of the business. This is a major paradigm shift for many IT professionals. To explain this business led approach to managing IT services KuppingerCole has written a research note "The Future of IT Organizations".