Measuring Scalability and Performance

It is crucial that any software acquisition does not just review requirements and functionality, but also guarantees the vendor’s system will grow with the business and remain valuable over a period of many years. Insurers should understand the scalability and performance of any system they purchase, confident that the software will scale with their company’s growth and perform as well in 10 years as it does upon implementation.  

Most insurers understand that before any major technology acquisition , they must complete a thorough process of requirements validation. This typically involves months of requirements gathering sessions, vendor RFIs and shortlists, product demonstrations, and possibly even an extended proof of concept period. When it comes to the nonfunctional requirements, however, there is much less common understanding and therefore much less rigor.  

Nonfunctional requirements

As the name implies, this applies to the requirements of a system that do not have to do with function, features, or behavior : they are less tangible and harder to quantify, but just as important to a system’s long-term value for a company. A list of some important nonfunctional requirements is as follows:  

Availability: How much uptime will the system provide, and what will the vendor guarantee in a service-level agreement?

Customizability/ extensibility: How easily can a customer add its own features and keep these features during a software upgrade process?

Usability: How well designed is the system, and how easy will it be to train new users?

Scalability and performance: How will a system grow with an insurer’s business, and will it continue to respond well as it comes into full usage?  

This is by no means an exhaustive list, as any criterion a customer uses to judge a system that does not involve explicit features and functionality falls into the nonfunctional category. Carriers tend to have less experience evaluating these nonfunctional requirements, though some are so inherent they are the focus of the first questions asked (price) or so culturally ingrained that there are governance bodies established to perform the review (security).  

Major scalability and performance metrics

In order to gauge how well a system will scale and perform in production, there are a number of metrics that an insurer can check before licensing a vendor’s software. Not all vendors will have test results for all of these metrics and those who do have results may have interpreted the tests in different ways , such that it is difficult to make direct comparisons between vendors. But if an insurer understands the different areas for testing and – perhaps more importantly – how these different areas will impact their particular business, it will be much easier to determine if a vendor’s solution will effectively serve their needs now and in the years to come. 

Load scalability/ number of transactions per hour

One of the most important factors in measuring a system’s ability to perform and scale has to do with its ability to handle large transaction loads. When running in production, a system will need to handle a certain number of transactions per hour.  

It’s just as important for the insurer to determine what kind of usage they expect in production, both upon going live and in the future. To determine how a system will be used, an insurer should ask some questions internally before judging a vendor’s scalability and performance .

Service scalability/ number of users

A similar metric to transactions is the number of simultaneous users that can be supported by a system. When running in production there may be five users on the system, 50, or 5,000. If an insurer expects its distributed network of agents to be logging onto the system every day, it’s important that the vendor has verified that many remote users can utilize the system at the same time without performance slowdowns or crashes.

Data scalability/ number of policies

Over time, any major insurance application will grow to hold an enormous amount of data. For policy administration systems, the obvious data growth area will be the number of policies, though for other systems it might be claims or bills.

For any major new system, an insurer should determine the largest areas of data growth and be prepared to discuss them with a vendor. This will help an insurer predict how large the database will need to be upon initial production and in future years. It’s also important to watch out for other data growth areas, especially with systems that have built-in auditing functionality. If a vendor boasts that every click by every agent is stored in the system for reporting and compliance purposes, it is important to ask them where those “clicks” are stored.

Page/ screen response time

The page/screen response time is a measure of how quickly the end-user interface of an application responds. This metric is somewhat redundant with transaction testing, because a page response time will be closely tied to a transaction response time. The page response time, however, helps provide a real world number that will directly impact all users of a system.  

Like most metrics in scalability and performance testing, the page/screen response time is not entirely straightforward. There are many usability studies that attempt to define an acceptable screen response time, though there is no generally accepted answer. Despite this, everyone can agree that faster is better and that users will be more productive if an application responds quickly or immediately.

In addition to these metrics, there are many areas that impact a system’s ability to scale and perform over time. There are also important ways that a vendor can provide comfort to the insurer that a system will be manageable in production..

As the load increases in production, a desirable system will be able to scale with additional hardware resources, either vertically or horizontally. Vertical scaling is when, in order to support an increase in load, an individual server is given increased memory or processing power . Most vendor solutions support this straightforward vertical scaling, though poorly designed software may not be able to handle an increased load even with bigger servers ; Horizontal scaling is when increased load is handled by adding servers to a distributed system. The most typical example of horizontal scaling is the addition of application servers to handle increased business logic processing . Supporting horizontal scaling can be significantly more complex for a vendor, especially at the database level. It is typical to see production environments where the application servers are scaled horizontally but there is only one database server that needs to be scaled vertically.  

Some vendors provide special tools for monitoring and adjusting the performance of their system. Others expect that the insurer will utilize existing tools built into the operating system or application server. These tools will help a system administrator see where there system is facing bottlenecks , or utilizing a significant percentage of the processor.

If they do not provide a tool and cannot recommend monitoring tools, it is a sign that they may not be doing much performance monitoring.

All of t hese metrics will be used in different ways by different vendors, and an insurer must be sure to understand exactly what a vendor has tested. A vendor should not be rejected just because it does not have test results in all of these areas, but the vendor should be ready to answer relevant questions or be prepared to run additional tests at the client’s request.

Perhaps more importantly, an insurer must ask questions about its own systems and user behaviors to properly plan out current and future scalability and performance needs.