Ask the experts: Information overload

As technology develops at an ever faster rate, so does the volume of digital information available to the enterprise. Accessing this data is easier than ever before – if you know what you’re looking for. Bradley Allen, CTO and founder of Siderean Software, discusses the challenge of ensuring that digital information works for your business.

FST. Information is usually treated as a precious commodity – how has the growth in availability of digital information caused problems for the enterprise?

BA. The total amount of information produced in today’s world is growing dramatically, on the order of several exabytes per year. This information is increasingly produced automatically using many different sources, and the boundary between information produced and managed directly within an enterprise and that available on the rest of the web is beginning to disappear. In particular, the growth of the web-based media and customer communications, especially that which is embodied in the blogosphere, is beginning to have a significant impact on how information flows between enterprises and their customers.

This impact presents two problems to enterprises: first, enterprises no longer can assume that they completely control either the flow or content of information about their products and services to their customers and partners. Secondly, those businesses whose business is information provisioning, for example media and publishing, must understand how to avoid being separated from their customers and partners through disintermediation – whereby others add context, relationships and participation to improve the ability to search and discover actionable information within the business’ own content.

FST. What are the limitations of traditional keyword search technologies to navigate this mountain of info?

BA. Keyword search involves free-text indexing. Free-text indexing takes the contextual cues (or to use the technical term, ‘metadata’) out of a document and throws it all away, leaving you with just a ‘bag of words’. When this happens, much of the value in the information that allows people to understand were it came from, how recent or reliable it is, how it relates to other, perhaps critical, related pieces of information is lost forever.

This creates a situation where the user has to make up for the loss of information by being clever about how to craft keyword queries. When this works, users get what they want; but when it doesn’t, as is frequently the case when users are novices or people unfamiliar with the vocabulary used in the indexed documents, the limitations of keyword search becomes painfully obvious.

FST. So how can alternative ‘dynamic’ search techniques help enterprises hone in on the valuable information they need?

BA. Context is key. By using metadata to provide context, users can orient themselves and find their way reliably through vast information spaces, without having to be initially aware of the vocabulary being used in the documents. We can move the focus of effort from having to think of what keywords to type to one of selecting from a set of alternatives presented by the system based on the current context of the user’s interaction with the system. This contextual framework gives users the ability to approach the way in can hone in on information from many different directions, increasing their chances of success in finding a specific piece of actionable information, as well as giving them the ability to discover relationships between different pieces of information that may have been hidden in the mass of results obtained through keyword search. Instead of simply searching for documents that match keywords, the user can explore and comprehend the relationships between people, organizations, events, topics, and the documents that bind them together. Leading to deeper insight and better decision-making.

Additionally, by giving users the ability to participate in adding to the value of underlying information by allowing them to contribute their own metadata (e.g., as user tags or saved searches) they can expose relationships between different pieces of information, further enhancing the information resource and making it even more relevant and current to a shared community of interest.

FST. Which particular areas in the financial services sphere can dynamic search add value?

BA. Success in financial affairs is heavily dependent on both a comprehensive awareness of current market conditions and the ability to quickly discover non-obvious information about the market that can be exploited to good advantage. Systems that support the monitoring and discovering nuggets of insight in vast amounts of market-relevant information by users, whether they are investors, traders, brokers or clients, increase the chances of being successful in the market. Using dynamic search, particularly as embodied in the notion of relational navigation, is key to making this happen, and to increasing the strength of the relationship between the user and the enterprise.

While searching and discovering information related to the market is paramount in financial services, it is also important to note that, as enterprises go, those in the financial services arena are particularly complex and dynamic as organizations themselves. The effort of understanding and managing the flow and content about these organizations (for example the skills and capabilities of the people and divisions within the organization, the sales and marketing information involved in explaining and selling services to customers, and the IT systems and investments involved in supporting the organization itself) is itself significant, and dynamic search and navigation has a role to play in making decision making within these enterprises more informed and effective.

Bradley P. Allen founded Siderean Software in 2001 and continues to serve as its Chief Technology Officer.