Event-Driven Architectures (EDA): How nimble is your financial enterprise?

Necessity has often been deemed as the mother of invention. In the IT world, necessity is probably the mother of technology adoption, too; any vendor invention is rather useless unless it appeals to the intellect and pain of an end-user. We are at the crossroads of such an invention-adoption cycle with Event-Driven Architectures (EDA), especially in capital markets. A better understanding of EDA concepts can be gained by investigating relevant market forces that are driving the need for such innovations. With the onslaught of electronic trading, hundreds of thousands of data points are created globally every second as orders are executed, currencies are traded, prices are determined, and financial products are bought and sold. Furthermore, the relevance of these data points diminishes dramatically with time. In certain cases, information has a shelf life of just a few seconds before most likely being relegated as historical data! Add competition, regulatory compliance and market volatility to the mix, and you have a perfect ecosystem that embodies contemporary Darwinism - Survival of the fastest.

To cope with today’s frenzied market, financial enterprises need systems in place to quickly sift through this data deluge and identify meaningful nuggets of information that drive trading and hedging strategies. Bad decisions or slow decisions often result in unimaginable monetary loss and loss of customers to a more agile competitor. The vast majority of business programming in most financial IT organizations is based on how data has traditionally been cataloged, distributed and stored with systems. Dealing with fast changing data, or ‘data in motion,’ is a difficult programming and architectural challenge, and one that can't be solved with existing infrastructure components such as databases and messaging systems. In certain cases, custom applications that sense pre-defined conditions or continuously poll for new events have been deployed. But, these do not provide the flexibility and scalability required to handle new and more complex financial instruments that necessitate constant changes to the market patterns being monitored. The ultimate goal would be to have a system that handles large volumes of market data, computes various aggregates based on this real-time information, correlates these streams to historical data and other sources such as reference data, and drives trade executions automatically with limited human interference. EDA adoption is a much-needed first step in that direction.

A simple overview of EDA
At the core of an EDA-based solution is an event processing engine (as seen in Figure 1). This engine serves as a conduit for real-time information to applications and clients that are impacted by changes in an ecosystem. Real-time data streams (for e.g., market data streams) from external and internal sources serve as inputs into this analysis engine. More quiescent data sources such as historical databases and reference databases also serve as information producers. Client applications act as information consumers by defining or registering patterns or queries of interest and receive relevant notifications when these registered queries are impacted by an incoming event. Providing users with an API or a user-interface based in a well-known language like SQL to model their queries goes a long way in helping organizations migrate to EDA with little effort.

Figure 1: Functional Overview of Event-Driven Architectures

The event-processing engine performs two critical functions:
Analysis: First, thousands of incoming event streams are processed with almost no latency to identify the most meaningful ones and their impact on queries or patterns registered by a user. Depending on the nature of user-specified queries, this analysis may involve correlation of real-time streams with data from static sources. Events are analyzed individually rather than in batch and simple events may even be combined to form more complex, derived events.

Distribution: Once the meaningful events and their impact have been identified, the event-processing engine propagates the necessary updates or changes to the relevant clients. This dissemination of information is more intelligent than simple broadcast mechanisms of the past, as it sends only updates to clients that are actually impacted by a particular state change. Such a selective event delivery model greatly reduces network bandwidth requirements. Sophisticated EDA implementations also support a model to conflate events, wherein the client applications can specify the time interval between successive updates as they are distributed. For instance, a trader may be interested in monitoring bond index movements every 500ms as opposed to monitoring every tick. Such controls are especially useful when there are network and hardware/CPU constraints at the client, which may not be able to process information at the rate at which the event-processing engine dispatches updates.

Applying EDA in your enterprise
The volume of real-time information being generated in today’s capital markets and the latency requirements therein make EDA relevant across several lines of business and technology units within a financial services firm. Some key areas of application include:

Electronic/Algorithmic trading: An EDA-based solution can be deployed to support trading strategies that are governed by specific patterns (e.g., index movements), or time-based metrics (e.g., VWAP) or correlated movements across multiple asset-classes (e.g., variance in equity and option prices of the same underlying security). Sensing these market movements and immediately driving relevant trade executions often translates to significant profitability.

Compliance and Risk Management: Nowadays financial institutions have to be cognizant of an increasing amount of regulations, both internal and external, that impact their operations such as trading, clearing, settlement and risk management. The ability to immediately track any violation (e.g., holding excess of a particular security) or potentially fraudulent activity (e.g., pre-arranged or wash-trades) can save firms a significant amount of money, reduce risk and avoid any legal headaches. Deploying EDA can also help organizations keep a real-time check on positions and minimize market risk and exposure.

Data cleansing and normalization: Erroneous records in areas such as reference data can cause significant operational inefficiencies in trading and related activities. In such situations, EDA-based approaches can help in continuously identifying data inconsistencies and faults as reference data is published into the system. This is accomplished by monitoring incoming data and checking for relationships and expected patterns. Such approaches to cleanse the data prior to storage in reference databases can prevent incorrect data from permeating multiple systems and consequently avoid any errant decisions.

While EDA is still in its infancy when it comes to adoption, certain progressive buy-side and sell-side organizations have production deployments that leverage its benefits. One large asset management firm has deployed a new event-driven order management system using GemFire? from GemStone? Systems. The order management system lets the organization effectively analyze real-time market data and order transactions to drive worldwide trading strategies, identify profitable price/execution points, flag trades that might be fraudulent and perform transaction cost analysis/research (TCA/TCR) in real-time. Similarly, one of the largest investment banks and brokerage firms is utilizing EDA (deployed with GemFire) to capture, analyze and store trade and market quote streams for use in real-time foreign exchange position keeping, and monitoring of events by internal trading clients.

Even outside of capital markets, EDA can be applied to such diverse areas as Radio Frequency Identification (RFID) data analysis in retail, fraud detection in credit-card processing, and intrusion prevention in network security gateways.

EDA vs. Request/Reply mechanisms – A Comparison
A vast majority of IT applications, be it client-server topologies or service oriented architectures (SOA), are based on a request/reply paradigm, wherein a client request is matched with a server reply. This model though relevant even in today’s IT climate is not appropriate for the kind of applications being discussed in the previous sections. The world is becoming more event-driven and architectures need to move in that direction too. In simple terms, the move to EDA-based technologies from a request/reply paradigm is comparable to the transition to a GPS based navigation system for driving directions. In the good old days, a car driver had to pull out a map every once in a while or make a pit stop at a gas station to ensure that he or she was on course. With GPS-based navigation, the system guides the user (the driver) with appropriate notifications at relevant times, without the user having to request information at frequent intervals. Further, event notification happens without any unnecessary disruptions - the driver no longer needs to stop the car to seek directions.

Broadly speaking, EDA and request/reply architectures differ from each other along the following four key dimensions:

Communication protocol: EDA works on a predominantly ‘push’ model, with information being sent to relevant users. Request/Reply is based on a ‘pull’ model, with clients requesting information. EDA has the flexibility to support a ‘pull’ model too if required.

Latency: Time is a far more significant attribute in EDA as the significance of any individual event - in terms of its business importance - quickly depreciates. Latency is a less stringent requirement in request/reply mechanisms. A relevant event that is not caught by an EDA-based system could very well be replaced by a more uninteresting event within a matter of milliseconds.

Querying: Request/Reply mechanisms support an ad-hoc query model, wherein data is mostly static and query parameters and predicates are not defined a priori. In addition to ad-hoc querying, EDA-based systems support a continuous querying model as well. In this model, pre-registered queries are constantly evaluated on underlying data that is rapidly changing.

Data Types and Volumes: As both real-time data streams and static data sources have to be managed within EDA, the volumes of data supported in such systems are much larger than those in request/reply architectures, where data sources are mostly static in nature.

Deploying EDA – Data infrastructure requirements
While most of the preceding discussions have focussed on the definition, need and application of EDA, the data infrastructure needed to support an EDA is a topic of equal importance that merits discussion. Given the key functions of EDA highlighted in an earlier section (‘A simple overview of an EDA’), the need for a sophisticated data platform cannot be overstated. Such a platform needs to incorporate the following key capabilities to enable successful EDA deployments:

Handle high data/event processing rates with low-latency: The quote, trade and price data volumes being generated in financial markets every second pose a stiff challenge even for the most robust infrastructures. To support EDA amidst this data deluge, there is an obvious need for sophisticated in-memory data processing techniques. Such techniques guarantee extremely high throughput in read, write and query operations.

Support correlation with multiple data sources: In most scenarios, be it electronic trading or real-time compliance, real-time information streams need to be analyzed in the context of other relevant static data entities like reference data or historical data. Only then can these time-sensitive signals be converted to actionable business information. Hence, a data infrastructure supporting EDA must include a framework that can store and process fast changing data as well as static data. The ability to join and query across such disparate data entities is often essential.

Provide scalable persistence and distribution capabilities: In addition to analyzing events with low latency, there is a need to store and distribute such raw events as well as the information derived from them - for instance, a raw quote may be transformed into a price that needs to be distributed. To accomplish this, a scalable persistence and data distribution layer is absolutely essential. Scalable persistence in this context translates to the ability to store increasing volumes of events for any historical analysis or regulatory compliance (for e.g., depth of book mandate in RegNMS). Scalable distribution, on the other hand, refers to the ability to propagate raw events or derived information to hundreds or even thousands of clients in a guaranteed manner. As is the case in many complex environments, unencumbered data movement is a significant roadblock that must be accounted for during the planning phase. With data coming from dozens of internal and external sources, the risk of latency in disseminating analyzed information is a near certainty without consideration.

A data infrastructure that can bring together these key requirements is a much-needed operational data layer that is being described as an ‘enterprise data fabric (EDF)’ or an ‘information fabric’ by progressive thinkers in the IT community.

Summary
Real-time data analysis is a key component to business profitability in an electronic economy. Classic forms of business intelligence that collect and organize historical information to support offline decision-making are no longer sufficient. In many business applications, thousands of events must be monitored, analyzed and used as triggers to generate immediate actions by the organization. This real-time version of decision support relies upon a combination of underlying events into useful information through complex event processing. With EDA, data becomes a shared, constantly updated resource for people and systems throughout the business. EDA helps by intelligently distributing useful information to all parts of the enterprise that need it, correlating data from multiple online and archived sources to form low-latency, high-availability data pools. While reengineering business processes and architectures for a changing business environment is difficult, it's not impossible. Through incremental changes in underlying architectures, including event-driven architectures and enterprise data fabrics, organizations can indeed leverage data for maximum benefits.