Increase Operational Efficiency by Modernizing Batch Processing

By Tidal Software

Enterprises have long since institutionalized job scheduling tools as a way to automate and control batch processing. What has happened, though, is that the modern enterprise environment has evolved faster than legacy job scheduling tools. With the advent of SOA, composite applications, and enterprise wide ERP; the proliferation of business data and the business intelligence initiatives to make sense of that data; consolidated and virtualized data centers; it is painfully clear that those legacy job scheduling tools have not kept up.

In fact, enterprises that are using legacy tools from vendors such as IBM, CA, & BMC have significantly diminished their operational efficiency over time. With 60% of data processing in the average enterprise being batch driven, the repercussions of this loss of efficiency are large, glaring, and, ultimately, very costly in terms of dollars and competitive advantage being bled away.

The important question IT management needs to be asking about this issue is: what should a ‘modern’ job scheduler or workload automation engine deliver in order to do improve operational efficiency in the modern data center?

Four aspects jump out as capabilities which can improve batch processing efficiency, lower TCO, and increase employee productivity.

Coverage
Modern enterprise data centers typically have enterprise-class applications such as ERP applications (SAP, Oracle, PeopleSoft, JD Edwards, Lawson) and Business Intelligence Applications (SAP BW, Informatica, Business Objects, Cognos, DataStage, SAS); Databases such as Oracle and MSSQL; File & Storage services such as FTP/SFTP/FTPS, Veritas, Tivoli Storage Manager, and Symantec Backup Exec; SOA-based Web Services, JMX, JMS; all within a server environment that is being increasingly virtualized using VMWare, for instance. Legacy schedulers typically do not support all of the components of this modern data center, or force you to stitch together multiple products. This not only creates added complexity and decreases efficiency; it also creates a fragmented view of your batch operations, and limits the flexibility you have in managing cross-application triggers and job streams.

Look for a scheduling solution that covers as much of your batch processing as possible across applications, platforms, and computing environments. This will help you get a view of your entire batch operations from a single pane of glass, and optimize across all of these. While deciding what kind of coverage you need, also consider what applications, platforms, and environments you do not currently have in your data center, but may have in the future.

Architecture
Legacy schedulers carry the baggage of having started as parts of larger management frameworks, extensions of the mainframe era, and then being extended to fill gaps in functionality. This has led to their having complex architectures that are not only difficult to set up and maintain, but also result in a higher probability of brownouts and outages. For instance, a popular legacy scheduler uses a proprietary database, an application server to provide controlled access to information in the database, a scheduler which interprets and processes all events in the database, as well as components from the vendor’s management framework including access management and a portal. In addition, many functions such as SLA management require installation of third-party tools. As a result of all this, not only do installation and upgrades take days, customers must also pay for expensive professional services to complete installation and upgrades, as well as large maintenance bills.

Compare this to a modern scheduler which has been designed from the ground up to manage all the enterprise’s batch operations from a single pane of glass, and integrates with popular management frameworks. Such a scheduler has a much more straightforward architecture, fewer components to install and maintain, and does not use any proprietary database, nor does it require third-party tools for complete functionality. This leads to a much shorter startup time, greater reliability, less hardware and software capital outlay and operations cost, lower cost of maintenance, and lower TCO.

Ease of Use
How do you define ‘ease of use’? Some people say ease of use is like a home run swing– you know it when you see it, way before you see the ball go out of the park. While technology geeks often pride themselves on being able to work with arcane and often difficult-to-use applications, the reality is that ease of use does have real financial benefits. These include lower training costs, higher productivity with a smaller learning curve, and ability to operate the application using less expensive resources. In addition, good ease of use also results in lower number of operator errors.

In the context of scheduling, a key part of ease of use is avoiding the need to create custom scripts to define jobs and job streams, job dependencies, alerts, events and other scheduling tasks. A modern scheduling and batch business process automation solution should be able to accomplish these tasks through drop-down menus, point-and-click, drag-and-drop and other graphical user techniques. The screens, click flow, and metaphors used should be intuitive and not even require reading the manual. Avoiding custom scripting and employing a single highly usable graphical interface to accomplish all scheduling tasks can significantly improve the manageability and quality of service delivered by a job scheduler.

Ease of Migration
With batch processing comprising an average of 60% of data processing in the enterprise, and almost all mission-critical applications having some element of batch processing, migration of the job scheduling application can appear to be an intimidating task.  To minimize risk of failure of critical jobs during migration, it is important to look for a scheduler that has the tools to automate as much of the migration task as possible, from a vendor with demonstrated expertise in the field of job scheduling. A well-managed migration to a modern scheduler with a clean architecture can actually be far easier than a messy upgrade of a legacy scheduler with a complex architecture and poor support.

Barriers to Modernization
With all the obvious benefits to modernization, why has this been such a blind spot for most enterprises? The answer is that there are several historical reasons; some of which no longer exist but continue to perpetuate in the minds of IT practitioners at all levels.

The first reason is linked to the mission-critical nature of job scheduling operations. Many operations people do not want to risk the chances of an outage during a migration and therefore avoid it. What they often don’t realize is that not migrating is actually creating a higher risk of outage than migrating.

The second reason is the belief that scheduling needs to be a part of the larger management framework that is employed in the data center. In fact, scheduling is often a poorly thought out add-on to a larger deal for some vendors. With modern job schedulers providing the ability to integrate with popular management frameworks, enterprises can continue to see batch scheduling operations within the entire operations framework, while reaping the benefits of a modern, easy-to-use job scheduler with broad coverage and a clean architecture.

Finally, the amount of attention that the batch processing environment receives, as compared to the real-time environment at the CIO level, is very low. In general, batch processing and job scheduling is considered a ‘solved problem’ and few actually take the time to examine the problems that arise from the evolution of the IT environment. It’s kind of like the plumbing in a 60-year old home, you seldom think about it until something breaks; although when that happens, you probably realize that it was outdated and inefficient in retrospect. In fact, scheduling operation outages or brownouts can cause many millions of dollars of revenue recognition issues, or SLA penalties. The question that remains is – can you afford to wait for something to break in your scheduling environment, or would it serve you better to proactively think about modernizing?

This article is brought to you by Tidal Software. Learn more about what to look for in a modern job scheduler or read our ROI case study with Staples authored by Mainstay Partners, an independent consulting firm that has performed over 200 studies for leading information technology providers including Oracle, SAP, Microsoft, Dell, HP, BearingPoint, EMC, Network Appliance, EDS, and Tidal Software.

If you would like a free estimate of the ROI you could get from modernizing your job scheduling environment, please send email to info@tidalsoftware.com and mention priority code USFST in the subject of your email.