When disaster strikes

When Hurricane Katrina hit the Gulf Coast of the United States in August 2005 its effects were enormous. Leaving almost 1900 people dead and causing an estimated $81.2 billion in damage, it was the most devastating storm in US history.

Ninety thousand square miles of territory were declared a federal disaster zone and upward of three million people were left without power. For John Hairston, COO of the Gulf Coast-based Hancock Holding Company, there were a huge range of potentially insurmountable problems. He tells the how Hancock Bank got back on its feet following the storm.

Hancock underwent a massive recovery effort in the wake of Hurricane Katrina. In those first days after the storm hit, our primary focus was finding and making safe our associates and their families. Over 1000 of our associates live in directly effected areas and with no communication, we naturally feared the worst. Many lost their homes, schools, churches, and entire communities. Essentially, all they had left was the bank. The first mission was to identify who was not accounted for, and get people on the trail of finding them. It took five weeks to identify the last missing associates. We were extremely lucky that even with 1900 casualties in the storm, not one Hancock associate was lost. Banks from all over the country, along with faith-based groups, supplied clothing, food, and water to our associates. Our various suppliers also provided badly needed aid that we then dispersed to the Hancock families. Once all this was in place, we were able to focus our entire attention on righting the ship.

After moving to find and protect our associates, we were able to concentrate on recovery. Fortunately, we had sent an advance team of IT professionals to our recovery site in Chicago before the storm. Afterwards, additional IT associates were sent to join the team and get the core systems up and running. These systems were operational in 24 hours and the core processing was current within five business days. Another top priority was getting our operations associates into functional workspaces. We lost four of five corporate facilities in the storm, including the HQ, data center, check processing center, deposit and loan operations centers, call center, underwriting and mortgage center, human resources, finance, legal, compliance, security, and just about every other corporate function.

The only corporate facility that survived was the training center. We converted this building to our recovery command center the day after the storm. Remember that with no communications, we had to have multiple command and control areas. The on-site ‘ground zero’ areas were managed in the former training building. The rest of the Company was directed from Tallahassee and Baton Rouge. The workspaces of almost 1000 associates had been destroyed and we quickly needed to find somewhere for them to go. In a matter of days, we converted workspace in various cities, including Purvis (near Hattiesburg, MS), Baton Rouge, Denham Springs, Tallahassee, Chicago, and Atlanta. Over 800 associates commuted daily by plane or bus to these interim centers. The spirit and drive of the Hancock associates was astounding. In those awful first few days, never once did I hear a complaint. What I heard was “where do we start, and what can I do?” I have photos of executives cooking food, trust officers sweeping glass, and commercial lenders carrying computers and printers many blocks away to a deployment area. I am rarely an emotional person, but I still get choked up when I remember the performance of the team during the darkest of days.

As we were trying to regain control we faced a number of extremely tough challenges. The most frustrating of these lay in restoring telecommunications, acquiring fuel and getting access to cash reserves. In the aftermath of Katrina, commerce was completely a cash and barter economy. ATM, debit, and credit cards were useless because 100 percent of power and telephone connectivity was lost over the entire market. Everyone needed cash, and they needed it right away. The Federal Reserve in New Orleans was lost, thus our source of cash was gone. Our security folks mustered teams and went out to the flooded ATMs and branches to empty the vaults of wet and stinking money. Our gaming customers also wanted to get cash off their sites. We took all this cash, and literally learned how to ‘launder’ money. We bleached and washed it in a washing machine, then we dried it in a drier, ironed it, and moved it through the branches. We did this with millions of dollars, money that was essential to keep the local economy moving. Due to the lack of fuel armoured couriers couldn’t deliver more cash, so this was all we had.

Obviously the fuel shortage presented plenty of problems of its own. We had stockpiled fuel in advance of the storm, but had expected more to be available by day four after the hurricane hit. This was not to be. A few well-meaning government officials redirected several fuel tankers to serve hospitals and jails. Consequently, our suppliers stopped sending fuel because they’d heard it would be confiscated. Before long, there was no fuel and panic ensued. The closest we came to civil unrest was due to fuel shortages. We finally decided to start our own delivery service and began to run hundreds of fuel cans on trailers back and forth from Florida and Alabama. This had to be done in the middle of the night with armed protection. Without fuel, we couldn’t power generators or transport our associates to and from work, so securing supply was of the utmost importance.

The last and perhaps the biggest challenge was telecommunications. For the first time on such a massive scale, telecommunication work order control was taken over by the Department of Homeland Security. No task, regardless of how critical, could be done without the permission and prioritization of DHS. Essentially, an office in Virginia decided what BellSouth would do first. The decision-makers weren’t here, so they didn’t know enough to really make good decisions on what circuits provided the greatest value. This took months to ultimately resolve.

Luckily we had disaster recovery plans in place prior to the hurricane. We had backup facilities in several markets, a recovery center for IT in Chicago and a recovery center for check processing in Atlanta. Were it not for these provisions, regaining operational effectiveness would have been impossible.

But even with these contingencies in place, many decisions still had to be made on the ground. In these cases, it was a tremendous help having associates who understood the core values and priorities of the Company. With communication in disarray, associates were left to make decisions they had never had to make before. They used common sense and compassion as a guide to help make good calls. Hancock’s core values of teamwork, personal responsibility, honesty, integrity, commitment to service, strength and stability provided a framework for all decision making. We cashed millions of dollars of ‘checks’ where the checks were written on little yellow note pads and napkins. When a house gets swept away, so do check books, forms of identification and bank statements. We helped thousands of people who didn’t even bank with us – or didn’t then and do now. In some communities we were the only bank open for weeks, even months. Our values guided us to make decisions when management wasn’t available.

Luckily, most financial institutions are never called on to implement their disaster plans. One thing we have been able to take from our experiences is a clearer understanding of which elements are the most critical to get right. You need to have leaders in the Company who can take command and make decisions. There also need to be strong guiding principles to assist them. In the wake of the hurricane, our HR Director stepped up to lead the technology team. He is now our CIO. Another case saw one of our trust officers and the chief auditor leading the team to retrieve all the critical files from destroyed buildings. It’s too late after a disaster to get people to think that way – they have to be already in place with an attitude to do whatever it takes. Secondly, there must be people who understand how each area executes their recovery plans. The people in those areas have to be drilled over and over on exactly how the plans work. There simply isn’t time to go over these things when the worst happens. Thirdly, ample recovery facilities must be close by. Ours were too far away, resulting in our associates having to make extensive daily commutes for months after the disaster. Finally, there are a few areas that must have rock-solid recovery plans. IT, ATM, call center, internet banking debit card, credit card, check processing, statement rendering and loan document creation areas require robust, well tested plans. We have invested millions in improving these areas in case we take another hit while our facilities are being rebuilt.

It’s also vitally important to protect against the loss of company and customer data. Even before Katrina, all data was backed up both on and off site. After what we learned following the disaster we also now redundantly mirror all data in Chicago and Gulfport. This is done live, so that we have an up-to-date copy available if a sudden disaster occurs.

The hurricane also served to underline the importance of technology in recovery efforts. Without IT we simply wouldn’t have been able to operate. While the core systems area performed marvellously, the server-based applications’ recovery took too long. As a result we have invested heavily in converting those applications to a virtual server technology that is easily ported to another hardware platform. We also have redundant platforms like Exchange and VPN access. The bandwidth between our recovery centers and Chicago was too light, so that has been expanded and single points of failure eliminated. Overall, we were delighted with the resilience of the IT team. They worked twenty four seven to get the job done, sacrificing their comfort and sleeping on office floors in the process. We are building a new, immensely strong data center to ensure they never have to suffer like they had to after Katrina. We’ve also made sure it’s well away from the beach!

One of the most valuable lessons that we took from our post-Katrina experience is that Business Recovery Planning is a core competency and should have the appropriate investment in capital, contracts, and expertise. Also, we learned the importance of volume testing. It might not be enough that a piece of equipment in a recovery center works. A few days worth of load must be placed on it to ensure it can handle the pressure of a catastrophic situation.

One of the most disappointing aspects of the post-disaster period was the government response. Not enough was done to safeguard the continuity of fuel supplies and too many important decisions were made at too great a distance. It would have been better to make calls about blockades and telecommunications priorities at ground zero rather than in an office 1000 miles away. By contrast, we were elated to confirm the strength and toughness of the people who live in the Gulf South. We were also surprised and touched at the outpouring of support from all over America, especially from faith-based groups who fed and clothed us for so many weeks.