Archiving solutions come of age

Mailbox Management to eDiscovery

The first commercial use of email occurred in 1985, and within ten years, reliance on the technology was so pervasive in the corporate world that IT groups were struggling to deal with the volume of information being sent, received, and (even worse) stored on email servers. This led to the imposition of mailbox quotas and the proliferation of local personal archives, which proved to be severely problematic in their own right. Email archiving solutions introduced early in the new millennium automatically off-loaded email servers according to configurable corporate policies, and moved the information into a long-term repository. These solutions solved the immediate and urgent problems of email server volume without creating the inefficiencies and risks of the local personal archive approach. The best of these solutions applied techniques such as de-duplication and compression to dramatically reduce the overall storage requirements and generate compelling cost savings. By creating pointers or “stubs” on the server to represent the archived messages, the archive was essentially transparent to end-users, who could continue to access and work with their information. 

It soon became apparent that email archiving could also offer substantial value to organizations that were subject to regulations that governed their treatment of sensitive information. For example, financial services companies were subject to regulations such as the Securities and Exchange Commission (SEC) 17 (a)-4 and the National Association of Securities Dealers (NASD) 3010 and 3110. For such companies, email archiving offered a means of gaining centralized control over the email information to ensure that their regulatory obligations could be met. This resulted in the emergence of a number of hosted data centers that enabled organizations to outsource the long-term retention and management of their email information in accordance with the regulations. 

Meanwhile, the legal industry was in the midst of a fundamental shift from paper to electronically stored information, and from discovery to electronic discovery, or eDiscovery, processes. As corporate information was exchanged primarily through email by this time, email archiving solutions soon began to play a vital role in the eDiscovery process as well.

Moving Beyond Email

Most organizations have heterogeneous technology and information infrastructures, with numerous different repositories, applications and formats.  As they continued to deploy archiving solutions to stabilize their email environments, enable regulatory compliance and support eDiscovery processes, these organizations began to explore the potential to apply the same concepts on other types of content servers in the enterprise. In recognition of this growing requirement, archiving vendors began to introduce new modules to their products to include additional content sources such as file systems, MS SharePoint servers and Instant Messaging servers. Additionally, customers began to demand integrations between archiving solutions and other complementary technologies, such as Enterprise Content Management (ECM) systems and Records Management (RM) solutions. While some vendors have advocated that all content (including email) be ultimately migrated to a single repository (usually based on their existing ECM product), the reality is that the sheer number of messages soon overwhelms any repository that isn’t specifically optimized for email. For this reason, most organizations favor a “records in place” approach to integration in which multiple repositories are maintained and united under an enterprise-wide management layer to provide end-to-end Information Governance.

Parallel Systems: Live and Archived Information

The corporate deployment of these expanded archiving solutions has been constrained by the growing realization that the separation of the operational or “live” information systems and the archive solutions is problematic and inefficient.  Archive solutions, by definition, are only focused on information that has been archived – typically after the completion of its active, collaborative phase. Other solutions exist to unify the live information systems, but deploying and maintaining two parallel sets of connectors into every different corporate repository is a daunting, expensive, inefficient and risky proposition. Autonomy, with its Intelligent Data Operating Layer (IDOL), is the industry’s leading vendor in the unification of the live systems. IDOL is an information processing platform that is capable of accessing information from virtually any corporate repository.  With built-in connectors to over 400 repositories, Autonomy was already plumbed into the heterogeneous information infrastructures of many of the world’s largest corporations. 

Recognizing the opportunity to leverage this connectivity to simplify archiving solutions, the company made the strategic decision to acquire ZANTAZ, a leading archiving and eDiscovery vendor, in mid 2007. By porting all of the ZANTAZ products onto the IDOL platform, the company has created the industry’s first unified information management solution that spans both live and archived information. This unification is particularly significant for pan-enterprise business processes such as eDiscovery, information governance and regulatory auditing. For such processes, the ability to categorize, search, navigate and analyze information across the entire enterprise is critical. Furthermore, if a regulatory agency or court challenges the validity of the information provided for a given matter, the organization must be able to defend the completeness and accuracy of their information processes and prove the validity of the chain of custody of each information object. This is greatly facilitated when a unified information processing layer is overlaid across the entire heterogeneous enterprise, including both live and archived components. 

The Continuing Problem of Volume

Having continuously moved information from the original content servers into the archive for several years now, archiving customers are realizing that the problem of information volume remains. The de-duplication and compression of the archive certainly results in a significant reduction in overall storage requirements, but the enormous and growing volume of electronically stored information cannot be addressed through those measures alone. Even without duplicates, the volume of information that is being stored over long periods of time within many corporate archives has become staggeringly high. This has resulted in rising costs for infrastructure, storage and backup, as well as productivity reductions related to the efficiency of managing such a large volume of information.  Additionally, as the volume increases, the risk of mistakes in the retention, deletion, location and production of information increases, which could lead to hefty, multi-million dollars fines if required for litigation purposes. 

To effectively deal with this voluminous body of information, more sophisticated automatic analysis and management capabilities to apply to live and archived content alike is required. One area of strong interest is “intelligent archiving,” which involves applying automatic categorization and classification to eliminate the long-term storage of inconsequential information, and also to enable effective disposition policies to be enacted. Another emerging requirement involves advanced search and navigation techniques such as concept searching, information clustering, dynamic hyperlinking and automatic query creation, all of which greatly increase a user’s ability to confidently find the specific information that is needed, be it for operational, regulatory or legal purposes. Business users who are working with the entire body of information also require analytics and pattern detection algorithms that facilitate trend analysis and high-level visualization. All of these requirements for advanced information processing are precisely what IDOL was engineered to accomplish. With the IDOL platform supporting the entire Autonomy ZANTAZ Proactive Information Risk Management solution suite, the company is uniquely positioned to address the challenges of relentless volume growth both within the archives and throughout the organization on content servers and end-user machines. 

Onsite or On-Demand… or Both?

The original hosted services in the archiving market were specifically targeted at compliance requirements, while the original software solutions were targeted at mailbox management and storage optimization. These distinctions are blurring as organizations on both ends of the spectrum demand more capabilities. The Software as a Service (SaaS) model of application delivery is gaining visibility and momentum within the IT space in general. Companies are seeking ever greater flexibility in the deployment of their archiving solutions, as well as greater levels of service and operational assistance in their ongoing information management. In response, vendors of archiving software solutions are scrambling to provide hosted options for their customers, often by striking partnership agreements with data centers. Autonomy ZANTAZ, a recognized leader in both the onsite and on-demand archiving markets, has responded to these shifting requirements by combining their premised-based and hosted products into the unique and powerful EAS On-Demand product that combines the strengths of both products into a hybrid offering with unparalleled flexibility and versatility.

Shifting Priorities

Despite all of the growth and change within the archiving markets, the fundamental drivers have remained unchanged, although their relative priorities have shifted. Mailbox management, once the primary business problem that drove companies to seek archiving solutions, remains an issue for many organizations, and continues to prompt the deployment of archiving solutions. But mailbox management issues are rarely considered in isolation now, as both regulatory compliance and litigation readiness have steadily risen in prominence over several years. 

The primary change in compliance is one of breadth. Where it was originally a concern only for those companies operating in regulated industries such as financial services and healthcare, it is now virtually universal. In the wake of the high-profile corporate scandals of the ’90s, regulatory scrutiny on corporate behavior increased across all industry sectors, resulting in more cross-industry regulation. Companies are increasingly aware of the need for “Information Governance” to establish and proactively enforce clear policies that limit potential corporate exposure and vulnerability. These information governance policies must be enforced in real-time, and they span the full lifecycle of the information, including creation, retention, access management and ultimately disposition. 

The corporate world’s focus on litigation readiness came sharply into focus with the amendments to the U.S. Federal Rules of Civil Procedure (FRCP) that came into affect late in 2006. These amendments reflect a global trend toward the formalization of the legal treatment of electronically stored information, which has become the predominant form of corporate information. Organizations that fail to comply with these amended rules risk significant fines as well as adverse legal judgments which can cause lasting brand and valuation damage as well as further financial impacts. These amended rules apply to any company that does business in the U.S., and the result has been that legal concerns have emerged as the strongest business driver in the archiving market today.

Conclusions

The result of all of these changes is that archiving has grown from an urgent but tactical solution to a specific IT challenge into an integral component of a strategic solution to proactively manage information risk. As the focus shifts from email to all corporate content, and as the use case grows from mailbox management to enterprise-wide legal and regulatory search, it no longer makes sense to consider archiving as a stand-alone technology. It must be considered within the full context of pan-enterprise information governance, encompassing the full information lifecycle. 

The sheer volume of information consumed and produced by today’s organizations necessitates the existence of an archive for efficient long-term storage, management, retention, access and ultimately disposition of information. But the movement of information from live systems into that archive should not generate a whole new set of policies and procedures. Nor should it require the creation and maintenance of another set of indexes. In the event of an eDiscovery or compliance request, it should be possible to undertake a single cohesive project encompassing both live and archived information rather than having to manage multiple parallel projects with different subsets of the corporate information corpus. With a full spectrum of Proactive Information Risk Management products, each of which is built on the industry-leading IDOL platform, Autonomy ZANTAZ is ideally positioned to help organizations rise to this challenge.