How one electronic payments provider relied on tested redundancy procedures to assure business continuity

Financial institutions should ask specific questions of their providers.

It isn’t enough for Financial Institutions (FIs) to ask if a payment provider has a back-up plan; they must dig deeper by asking specific questions addressing key areas to assess whether the vendor can provide uninterrupted services during emergencies.

ChoicePay, a leading electronic payments provider that recently experienced a natural disaster in their headquarters city of Tulsa, Oklahoma, recommends that financial institutions request to see a comprehensive risk mitigation plan from their ePayments provider. This plan should cover the following topics: power, crisis communications, disaster recovery site(s), back-up procedures, testing and external audits. Providers should have a comprehensive business continuity plan in place to support each payment channel offered such as IVR, contact center, online and web services, and kiosk and walk-up agent locations.

While security and disaster recovery are often lumped together, these are two very different subjects. A sound security infrastructure will always be a critical factor in vendor selection, but as more ePayments providers achieve PCI compliance, whether or not a vendor has made the investment to develop a sound business continuity and disaster recovery plan has greater influence. FIs should ask as many questions as necessary to verify that a payment provider can protect sensitive financial information on a daily basis as well as assure business continuity in the event of a natural disaster.

ChoicePay’s systems were recently put to the test when a severe ice storm paralyzed 30,000 square miles of Oklahoma. Established and tested failover systems were in place that made it possible for the company to remain fully operational to take payments throughout the week-long emergency, with no effect on service levels to its billers or consumers.

Weathering the storm: 600,000 households and businesses out of power
On December 10, 2007, Oklahoma was hit with one of the worst power outages in state history. At its height, more than 600,000 homes and businesses were out of electricity as a result of falling limbs and downed power lines. Businesses, schools, churches, homes – almost every place and everyone were without power for heat or light. Eight days later, more than 20,000 homes were still without electricity. The destruction from three days of ice storms and freezing rain could be seen along every street and neighborhood. All 77 counties in Oklahoma were declared federal disaster areas.

While this event was taking place in Tulsa, cities like Orlando and Tampa – where ChoicePay serves hundreds of thousands of customers – enjoyed 80-degree temperatures. Life elsewhere carried on as usual, so bills had to be paid. During the most severe weather and in the days that followed, ChoicePay never lost the power needed to process a payment or answer a customer call. ChoicePay was able to rely upon their disaster recovery plans and emergency infrastructure to protect data and maintain 100% uptime. The investment to establish a state-of-the-art data center protected by a redundancy strategy paid off for ChoicePay and the clients it serves nationwide.
  
Secure data center and tested recovery strategy
“The ChoicePay headquarters is located in the CityPlex Towers in Tulsa, which also houses a hospital,” explains their Information Security Architect, Scott Williamson. “To accommodate the needs of the hospital, our building (and data center) receives power from two separate power grids supplied by two different power companies, so we never experienced an extended outage. The data center is also protected by a data center-wide battery backup system and electrical backup generator fed from a diesel reservoir with three 10,000-gallon tanks. We test these redundant systems weekly to ensure optimal performance during disaster scenarios such as the one on December 10th.” 

Additionally, Internet connectivity for real-time integration and credit card processing is provided by dual providers with independent fiber optic paths into the building. Tulsa is located in the heart of “Tornado Alley”, so the data center has been hardened to withstand a direct hit from an F5 tornado, with winds in excess of 260mph. Furthermore, daily backups are encrypted and stored offsite in a secured facility. 

Local clients also depended on ChoicePay to process their payment functions, particularly those utility clients who were literally knee-deep in service issues due to the outage. Utilities worked around the clock to bring service back to their customers, so they couldn’t afford to devote valuable resources to handle payment-related issues. They also had their own redundancy and even attendance problems, with one company reporting only 25% employee attendance the day after the storm.

“The December 2007 ice storm caused the largest blackout in state history, bringing many challenges to our customers,” said Dave Arnold, Vice President, Customer Service Distribution Companies, Oklahoma Natural Gas Company. “Many relied on natural gas, and our natural gas delivery system was fully operational during the crisis. ChoicePay was also available uninterrupted, delivering payment services as promised.”

This time, the ChoicePay data center redundancy and business continuity procedures kept ChoicePay up and running. There was no need to fail over to a standby disaster recovery site. However, ChoicePay recommends providers should carefully review their e-Payment provider’s ability to fail over to a second location if they want to ensure the ability to weather any situation.

Basic questions to ask a provider to evaluate their business continuity strategy
When developing the ChoicePay business continuity plan, the company set out to build a best-in-class strategy by incorporating the best practices for disaster recovery and business continuity. The following questions and best practice responses are a good guide to utilize when analyzing a provider’s ability to continue providing services in the event of an emergency.

At a minimum, payment providers should be able to provide FIs the Best Practice (BP) answers to the following questions regarding their redundancy plans for each payment channel they provide.

REGARDING DOCUMENTATION
FI: Does the organization have documented plans for business continuity and IT disaster recovery?
BP:  Vendors should have redundancy plans that are formally documented, and each member of the disaster recovery team should maintain copies at their work station and offsite.

REGARDING ALTERNATIVE POWER SOURCES
FI:  Does the organization have redundant power?
BP: Vendors should utilize multiple power grids/providers so that they do not rely on a single company or transformer substation for power.

FI: Are all of the organization’s systems connected to an Uninterrupted Power Source (UPS) or is UPS limited to only certain systems?
BP: Vendors should employ data center-wide UPS including all servers, networking systems, security systems, etc.
 
FI:  Does the organization have a back-up generator and will it power all systems, not just core systems?
BP:  Vendors should have generator power that provides 100% power to all systems and should indicate how long the generator will run without refueling.

REGARDING CRISIS COMMUNICATION SYSTEMS
FI:  Does the organization have redundant communication facilities?
BP:  Vendors should utilize multiple telephony paths so that service is provided by multiple service providers as well as multiple data paths, so that data connection is provided by more than one provider.

FI:  Do the organization’s services reside on multiple servers?
BP:  Vendors applications should reside in a load-balanced clustered environment so that one server can be out of service and not affect the application.

REGARDING A DISASTER RECOVERY SITE
FI:  Does the organization have a disaster recovery (failover) location, an alternate site location for data center recovery purposes?  If yes, is it a hot, warm or cold site?”
BP:  Ideally, vendors should have a hot site that is at least 50 miles away from the primary site.

FI should assess how much downtime and lost data they can afford and choose accordingly based on the following choices:
 

• A hot site is a site that is always on and shares traffic. It is an alternative site to processing transactions. No downtime is required to activate in the event of an emergency, so no data is lost.
• A warm site is a site that is always on, but only handles services during a primary site outage.
• A cold site is a site that is off, where data must be manually restored before services can be offered, which can take days or weeks.

FI:  Does the organization’s Disaster Recovery (DR) site provide 100% processing capacity or can only a subset of services be performed? 
BP:  Vendors should utilize a DR site that is 100% self sufficient and provides mirrored services of the primary site.

REGARDING SCHEDULED BACK UP PROCEDURES
FI:  How often does the organization perform system data backups – hourly, daily, weekly, monthly?
BP:  Vendors should back up customer data hourly, with tape back up performed nightly, encrypted and stored off-site in a PCI compliant facility.

REGARDING TESTING OF DR PLANS
FI:  What components of the organization’s disaster recovery systems and infrastructure are tested?
BP:  Vendors should practice a structured DR test plan that includes testing all systems on a documented schedule. Testing should be performed on the generator on a weekly basis and on the UPS system quarterly. Data center failover for warm or cold site should be tested at least annually. The FI should be able to participate or audit the annual DR test.

REGARDING EXTERNAL AUDITS OF THE DR PLAN
FI:  Does the organization receive external audits to verify the DR plan?
BP:  Vendors should receive annual SAS 70 Type II audits with unqualified results.
(Note:  While a SAS70 Type II audit is not required by regulatory entities, a vendor who engages a qualified third party to perform an SAS 70 Type II audit is obtaining an independent assessment of its internal control environment.

An SAS 70 Type II audit is an American Institute of Certified Public Accountants standard. The following tasks are normally performed during an SAS 70 Type II audit:  identification and documentation of key controls, evaluation of the effectiveness or the controls via walkthroughs and testing activities, and the delivery of a SAS70 Type II Service Auditors Report, which contains an Independent Auditor’s Opinion. Selecting a vendor who has been audited against SAS 70 standards will provide management with an additional level of confidence that the vendor has implemented an adequate control environment and that the controls in place are operating effectively.)

REGARDING THE DR TEAM
FI:  Does the organization have a dedicated team of professionals focused on business continuity and/or IT disaster recovery?
BP:  Vendors should have a team of individuals dedicated to business continuity and/or IT disaster recovery who have designated responsibilities in the event an emergency situation. Knowledge must be dispersed with known protocols, so that any team member can trigger the plan to ensure business as usual, never losing sight of serving clients’ needs.

ChoicePay has recommended some of the basic questions that FI should ask when analyzing a payment provider to determine their ability to conduct business in the event of a disaster which could last days or weeks. However it is also vital that the provider has a sound security plan for day-to-day operations. FI who want to know more about firewall zones, encryption and file transmission of financial information, authentication, vulnerability scans, security patch reviews, virus scanning, the physical security of the building and all of the areas of security can contact ChoicePay directly at (866) 559-2455 or request to be contacted at www.choicepaycorp.com, on the Contact Us page.

“Oklahoma weathercasters recommend that homeowners keep a disaster supplies kit including a hand crank radio, water, and a flashlight, among other things,” says Williamson. “For businesses, the list is longer, but the goal is very similar – to keep as safe as possible in the face of extreme conditions.”