Keep on moving

Lyndon Bird of the Business Continuity Institute looks at the need for comprehensive BCM plans.

In such a highly regulated industry, financial services firms can’t help but have comprehensive BCM plans in place. But though their compliance might be good, the effectiveness of their BCM measures might be unproven. And the time-critical and sensitive nature of their processes, mean they have little option other than provide massively secure IT environments. The tendency to equate sophisticated IT provision for resilience and ultimately recovery of systems with good BCM is also an oversimplification.

If we look back at the most significant impact on the world’s financial infrastructure, the 9/11 attacks on New York, we learned a number of lessons and very few of them were about technology. Primarily they were about people and organisations.

The first lesson was the realisation of the sheer magnitude and geographic scale that could be involved. The oil industry learned a similar lesson from different sources following hurricane Katrina. Most organisations assumed that they could plan for a building and perhaps a police cordon area of 500m or so. With most of Manhattan closed down some organisations had over 10 locations simultaneously affected so if any of their plans assumed those facilities would help, the plans failed.

Everyone learned a lot about the people involved and how they would handle a crisis. In this situation, even if you are directly unaffected, your priorities would be at home and with family, not working hundreds of miles away to help recover the systems. Many people never fully recovered from the trauma and this has long-term affects on businss continuity in their firms.

The direct response companies like IBM and SunGard of course had many more invocations than they could ever have planned for and the whole viability of that industry in dealing with a concentrated terrorist attack came into question, leading many organisations to consider having their own dedicated solutions. To some extent it also lead to an over-reaction from the US regulatory authorities on their demands for recovery site geographical separation of hundreds of miles.

Communications was a key lesson to be learned for many organisations. Do not take the infra-structure for granted, do not assume you will always have telecoms, internet or even power. Do not assume your people can get to recovery sites; roads might be blocked and other means of travel disrupted. The lack of co-ordination and understanding between corporate firms and public authorities was highlighted and has lead to calls for much better integration between BCM and emergency planning.

Having said that, in the past six years many firms have improved their BCM significantly. However this is still heavily predicated on technology recovery, and to be fair among most large firms there were not massive gaps prior to 9/11. In particular, however, firms have improved their capability in communication (where virtual command centers are becoming fashionable) and so called ‘work area’ recovery.

However, many observers still have concerns about the effectiveness of the actual crisis response and more particularly the HR elements. Without people the most automated operation in the world will still fail, so having the right people doing the right things in the right places at the right time is still the most important element of BCM. This is always key to effective recovery. Defining roles and responsibilities, exercising people (and deputies) and undertaking multi-functional drills are the thing that makes recovery likely to be successful.

Pandemic planning has also highlighted weaknesses in many plan assumptions. Switching to home working during a pandemic is clearly questionable and need to be given much more thought. There is nothing to suggest that it is even fully viable despite such technology as Citrix and the load and capability of national and international infrastructure is not really known. There are certain to be major back and middle office functionality that cannot be handed via home working.

In some ways the best BCM for the financial sector is to design the need for recoverability out of systems and networks. Virtualization is becoming a popular option from a system management and efficiency perspective and it does simplify recovery. However at a simplistic level not putting all your eggs in one basket is a good approach so having distributed systems might reduce risk. Business now demands a level of service uptime that was previously unheard of. In that sense de-centralization helps by both limiting the damage one single incident might create while creating opportunities for better guaranteed service availability across fault tolerant networks.

However often this is not the case as a proliferation of servers and systems geographically spread can create a management headache to properly monitor and map which parts of which business process are supported by which part of the infra-structure. Mapping your mission critical systems across the hardware/networks that support it is essential – or you might well lose one server which will stop an entire key business process.

It is well accepted that firms might draft business continuity/disaster recovery plans but might not carry out comprehensive simulations of an actual disaster. This is mainly because these exercises take a lot of organising, they are not just about turning up at a recovery center, loading backups and going for a drink. The time, resource, and cost involved discourages many organisations from doing more than token test.

Tests need to be planned and scheduled in a way that gradually increases the complexity and people involvement. Testing small things in isolation like call out procedures is fine; testing IT recovery site operations can be done on a limited scale, training/testing the Incident management team can be done. Individual business units can test short-time work-around measures. However, a major test should not be scheduled until you are pretty confident about the viability of the individual components.

In conclusion, it is fair to say that no financial services company can survive a major interruption to its ICT for an extended time. In some cases no interruption at all is acceptable. Ultimately adequate insurance can only be delivered by IT, but IT does not own the BCM process. It is a key component of delivering the implementation strategy – ultimately the responsibility lies with the Board. Deciding on the level of risk the company is prepared to take is a Board decision and deciding what systems, services, locations and business processes are vital post-disaster are business decisions.