Professional Documents
Culture Documents
Beyond Job Scheduling: The Road To Enterprise Process Automation
Beyond Job Scheduling: The Road To Enterprise Process Automation
Beyond Job Scheduling: The Road To Enterprise Process Automation
Introduction ............................................................................................................................. 3
Job Scheduling Defined.......................................................................................................... 4
The Evolution of Job Scheduling ............................................................................................ 4
UNIX CRONtab....................................................................................................................... 5
Distributed Scheduling Products ............................................................................................ 6
Distributed RDBMS-based Scheduling................................................................................... 7
Application API’s..................................................................................................................... 7
Enterprise process automation............................................................................................... 8
Dynamic, Data-Driven Automation ......................................................................................... 8
Conditional Business Rules.................................................................................................. 10
Workload Balancing.............................................................................................................. 11
End-user involvement........................................................................................................... 12
Fast Flexible Implementations.............................................................................................. 13
Accelerated Information Delivery.......................................................................................... 14
Java-Enabled Client.............................................................................................................. 15
Interactive Graphical Management....................................................................................... 15
Security and Auditability ....................................................................................................... 15
Conclusion ............................................................................................................................ 16
About UC4 Software............................................................................................................. 16
Introduction
The relentless pace of today’s business environment and its enormous pressures on IT staff
has made it critical for corporate business systems to become more responsive and
reliable. End-user expectations for reliability and speed grow dramatically as more and
more users have come in contact with a wider array of software applications. Expectations
now approach that of the telephone, with users demanding business systems to be
available instantly and operate flawlessly.
Coincident with these rising user expectations are the increasing complications of
enterprise computing. IT personnel have been forced to come to grips with the accelerating
rates of change, and the demands of corporations everywhere to quickly respond to the
competitive threats and opportunities brought about by e-business.
The rise of the web has proven to be the ultimate nemesis for the so-called ‘batch window’ -
that nightly opportunity to run large groups of transactions offline. Instead we have seen the
rise of the ‘real-time organization’ that needs to respond to changing business conditions
quickly and effortlessly in a 24 / 7 fashion. This creates significant challenges for
corporations as they try to respond to the demands of near real-time processing with their
traditional batch-oriented back end systems.
In traditional enterprise computing, the availability of the batch window meant that a
significant period of time could be set aside to launch, monitor and manage a variety of
batch/sequential activities that were necessary for the proper running of the business. The
new global business environment, and in particular the Internet with its implied on-line, real-
time, 24x7 operation has dramatically altered this rigid notion of scheduling. The idea of
having batch processes with a multitude of similar transactions that are processed at night
is still relevant in many cases, but there are many other examples where traditional time-
based, operations centric ‘scheduling’ is wholly inadequate. Instead, companies require
near, or real-time processing.
Traditional batch scheduling held that business processes could be mapped out and
executed in predictable ways that followed one or more business calendars. The dominant
concept was that certain business activities and the software systems that supported those
activities operated in sequential, predictable ways. Scheduling tools were only required to
support notions of day, date, time and multiple business calendars to support corporate
and/or departmental needs.
For example, accounting might be scheduled on a calendar year basis and the emphasis
would be on months, quarters, and fiscal year end. Alternatively, manufacturing might place
all of its emphasis on weekly periods for purposes of planning, inventory analysis, and
production. Manufacturing would also be forced to deal with plant shutdowns and holiday
periods that impact production cycles, but may have little relevance for accounting
purposes.
Today’s supercharged environment requires companies to respond much more swiftly to
changing business conditions. Referring to the discussion above, it might still be viable for a
company’s fiscal operations to run on a fairly routine basis. However, the realities of
increased global competitiveness dictate a manufacturer to be linked electronically with its
business partners in an extended ‘just in time’ enterprise. To exploit the extended
enterprise, the manufacturer must replace or automate its manufacturing, ordering and
inventory processing systems to respond dynamically to changing business requirements
and competitive pressures.
Improving business responsiveness through automation requires a comprehensive
approach build in from the start. The selected solution must automate processing across a
wide spectrum of software, computing platforms, and business processes.
Implementing a solution that moves beyond mere scheduling and fulfills the promise of
enterprise process automation can mean the difference between corporate success and
failure. Traditional batch job scheduling products cannot meet today’s enterprise automation
needs, leading to the creation of a new breed of software to handle this increasingly
complex and demanding role.
Implied in this accelerated business climate is the requirement that IT staffs must build
computing infrastructures that create responsive platforms for change. Job scheduling tools
with their rigid adherence to the limitations of automating tasks based on day, date and time
don’t meet this requirement. New solutions are required that can automate and accelerate
existing business processes and adapt to new processes and systems as they arise.
Simple scheduling
For instance, all businesses rely on one or more business calendars. Calendars govern
such things as when payroll is run, when corporate reports are generated, and when fiscal
months and quarters are closed. The key issues with calendars and other important
scheduling features are flexibility and reliability: Can the scheduler adapt to the changing
needs of the business, and deliver predictable results time after time?
Additionally, programs and applications need to run on a variety of platforms, at specified
times, in specified orders, and with varying levels of resource demands and prioritization.
Schedulers need to be flexible enough to accommodate these varying technology, business
and resource demands. If the job scheduler is able to sequence processes and manage
resource contentions, applications will execute faster and more predictably, and scheduling
throughput will increase. Unfortunately, most scheduling products only focus on sequential
activities and ignore other opportunities to balance and prioritize workload.
UNIX CRONtab
If the manual approach is not the answer for distributed scheduling, the next logical step is
to look to UNIX or other distributed platforms to supply some of the necessary tools. In the
world of UNIX the tool is CRONtab. CRONtab is a positional, character-based tool designed
to facilitate repetitive ‘scheduling’. To use CRONtab, a user enters the required positional
characters in a text file that in turn is read by CRONtab. An example of CRONtab is:
10 5 * * 1-6 /usr/lib/myprog
Translated, this statement runs a job at 10 minutes after 5:00 A.M. six days a week (the
week starts on Sunday, which is day zero).
CRONtab is in very wide use despite a number of serious limitations including:
• No business calendars.
• No dependency checking between tasks.
• No centralized management functions from which to control and monitor the
overall workload across the network of CPUs.
• No cross platform dependencies are allowed. CRONtab only manages workload
on the specific machine where it resides and it cannot manage dependencies on
other machines.
• No audit trail of jobs that have executed.
• No automated restart/recovery of scheduled tasks.
• No recovery is possible from machine failure. If the machine where CRONtab is
executing crashes, there is no mechanism to move processing to a backup
machine to continue the scheduling.
• The scheduling rules are rigid and not easily changed.
• No dependencies are possible on the arrival of a file.
If we extend our CRONtab example to other environments, similar parallels exist between
MVS/JES and DOS/Windows BAT files.
Distributed Scheduling Products
When mainframe-centric computing began to give way to distributed computing, job
schedulers were created for this new environment. Conceptually, the requirements for
distributed job scheduling were thought to be no different than they were for the mainframe:
time-based sequencing and dependencies. With mainframe scheduling tools providing the
conceptual model for how job scheduling worked, the implementation challenges in the
distributed environment were mainly architectural. Specifically, how do you implement
secure job scheduling cross a potentially large number of diverse platforms?
When the batch workload is widely distributed, the scheduling becomes increasingly
complicated. Specific distributed scheduling issues include:
In the event of a failure, how will the machines resynchronize with one another when the
network is restored?
How can I manage and monitor the workload from these broadly distributed systems?
• How can I keep my activities secure from unauthorized users?
• High-end distributed scheduling solutions address these issues. Moreover,
improvements in the reliability of networking, security and hardware have made the
underlying environments stable and secure.
In recent years, hardware and operating system vendors have concentrated on building fail-
over mechanisms into their systems, which have dramatically improved reliability of the
underlying IT infrastructure. This has the additional benefit of allowing a single fail-over
approach to be implemented across the enterprise instead of needing to create and
document dissimilar fault tolerance and recovery routines for each application or tool in the
environment.
Despite these architectural advances, most schedulers are actually updated versions of
CRONtab: they simply schedule a script written around the core program that needs to
execute.
With traditional job schedulers, scheduling involves not only the creation of scheduling rules
within the scheduler, but the creation of a script for each program to be executed.
Application API’s
The rise of ERP (Enterprise Resource Planning) and other packaged applications in the late
1990’s created another force for change in distributed job scheduling. Historically, job
scheduling and systems management emphasized the management of the various
components in the IT infrastructure. Networks, CPU’s, databases, controllers, programs,
disks, and more require management and monitoring. Specific tools appeared for each of
these areas to monitor each component in the infrastructure.
Over time, IT personnel realized that focusing strictly on a given component without regard
to the applications that these components supported was missing the big picture:
monitoring and managing the enterprise. Therefore the notion of application management
became popular and large systems management frameworks became available.
One-size-fits-all frameworks have been discredited as costly, difficult to implement, and
lacking ‘best of breed’ functionality. But the goal of managing applications and not just the
underlying components has prevailed. For many IT organizations, managing and monitoring
core business applications, and maintaining service level agreements based on those
applications, is the essential service they provide to the rest of the organization.
Interestingly, most packaged applications have some type of rudimentary scheduling and
management capability. This built-in scheduling capability creates its own challenge as
each application becomes its own ‘island of automation’, isolated from other applications in
the enterprise. This creates significant integration problems when trying to synchronize
processing across the multitude of enterprise applications. Enterprise application vendors
responded to this challenge and created the application program interfaces (APIs).
APIs, designed for use by programmers and other technical personnel, allow the scheduler
to communicate directly with the packaged application to launch and monitor jobs.
Sometimes these API’s work in conjunction with the application’s own
scheduling/automation tool. In other implementations the application’s scheduling tool is
completely bypassed with the external scheduler controlling all batch activity. The
scheduling vendor writes an interface to the application’s API, which then allows the
scheduler to perform some minimum subset of commands necessary to launch and monitor
the application’s batch jobs. As discussed later, this is an important, yet still incomplete,
approach to automating packaged application processing.
Enterprise process automation
The previously discussed features are the minimum core selection criteria for any job-
scheduling product. However, the market requirements for automating system and business
processes have moved beyond job scheduling and its rigid paradigm of scheduling ‘jobs’
based on day, date, and time. Today’s enterprise process automation solutions use day,
date, and time as pieces of the automation puzzle, but also incorporate additional features
designed to address the dynamics of the business and improve the responsiveness of
enterprise computing.
Enterprise process automation goes beyond job scheduling in the following areas:
The following section describes how the UC4 platform implements the above requirements
to provide unprecedented levels of automation, ease-of-use, reliability, and security.
The entire business process ‘X’ runs from Step 1 to Step ‘n’, but is broken up by a series of
manual reviews and job submissions. Eliminating the manual steps of gathering input data,
reviewing output, and launching tasks changes the nature of the business process as
shown below:
Eliminating manual process accomplishes two very important goals. First, the processing
latency inherent in manual efforts is removed and processing is accelerated. Second, errors
are reduced or eliminated by removing or reducing end-users involvement in the process.
Error reduction is very important. Recovering from the effects of bad data or incorrectly
managed processes can be costly and time consuming. Additionally, in today’s accelerated
business environment, there is very little time available to recover from processing errors.
The UC4 platform uses application data to provide superior automation, and in so doing,
improves business responsiveness, eliminates errors, and frees up staff time.
Traditional job scheduling products don’t provide the advanced functionality required to
model conditional business logic.
Workload Balancing
Application processing today cannot be separated from the notion of workload balancing.
Workload balancing ensures that scarce computing resources are allocated appropriately
among competing tasks and applications. Ignoring the effects of these resource contentions
can severely impact processing, with the result that service levels can suffer. Efficient
automation products need to provide technology for managing these resource contentions.
The UC4 platform uses queues to balance the workload on application machines.
Workload balancing provides not only the ability to set priorities for processes, but also to
create multiple queues so that jobs of similar priorities can be grouped together and run
according to a predetermined algorithm. Additionally, priorities and queues must be easily
changed to accommodate the changing needs of the business.
Creating and managing queues, changing queue priorities, and assigning priorities to
processes and schedules are all features provided by UC4. These capabilities optimize
application throughput and balance the load across systems and applications.
End-user involvement
Many of today’s enterprise applications are designed to empower the end user. Empowered
users submit their own requests, generate their own reports, and are free to make demands
on the system round the clock. As organizations have implemented these large applications
and their companion reporting products, they have realized some additional challenges.
The users, enticed by the promise of more control and better data access, want to submit
requests and reports 24/7. This poses a problem for the operations staff as these user
requests create fluctuations in system load that impact overall system performance. These
fluctuations are often severe enough that operations personnel can no longer meet their
service level objectives. Some user requests also collide with other mission critical business
processes. The challenge is to continue to allow end-users the freedom to maximize their
use of the system while maintaining a reasonable system load and managing resource
contentions.
The answer to this problem lies with the unique end-user capability available from UC4. By
combining the automation of end-user interaction with the workload balancing and security
capabilities, both operations and end-users receive the control and responsiveness they
need.
Like the enterprise applications themselves, an enterprise process automation product must
be flexible and easy to use so that an end-user can exploit the product’s features. The end-
users often have the best grasp of the business and know how their business processes
can be accelerated or streamlined. The UC4 platform gives them that chance.
A Request window provides a mechanism to safely manage conflicting user requests and
gives end-users the ability to request jobs on an ad-hoc basis. Additionally, authorized
users can submit chains (groups of jobs linked together with one or more dependencies)
and monitor and manage them to completion. Upon completion, users have the ability to
view output online and redirect output to a desired printer.
The key to making this all work in the enterprise is being able to set the correct processing
and prioritization policies for managing user requested jobs. Operations personnel can
create and maintain process queues, priorities and end-user profiles. These parameters
can be tuned to eliminate undue system load and resource contention. Queues can be
established that only handle certain classes of requests and/or operate only during certain
times of the day, which in turn allows users the flexibility to submit jobs for execution at any
time without IT staff needing to worry about unmanageable processing loads.
From an end-user’s perspective, he or she has gained an unprecedented level of control
and visibility. Launching, monitoring, and canceling requests at will, no longer requires
intervention from IT staff. This eliminates an unnecessary burden from IT staff in terms of
handling ad hoc user requests via phone or email. Moreover, since end-users control the
submission, the chance for costly errors to be introduced when the user passes the request
to the IT group is eliminated. Additionally, because input data sources can be validated at
the time of the user’s request, the simple but potentially costly errors associated with
running ad hoc submissions with incorrect data can be avoided.
In some cases, IT departments have found it desirable to have the UC4 platform in the
background, intercepting user requests after they have been submitted through some other
means. With this approach, it is not necessary for users to learn a new interface to submit
their requests. They’re often not even aware that automation software is involved in
processing the request. The UC4 platform can still be configured to control the requested
tasks and operations personnel can capitalize on queue management and prioritization
schemes to meet their service objectives.
However, involving the end-users in the managing and monitoring of their requests
increases user satisfaction and cuts down on help-desk calls. Users feel empowered and
more involved when they are able to actively participate in managing the application. The
UC4 platform provides the necessary facilities to allow this type of user participation.
The UC4 platform provides its own built-in FTP capability. This built-in function allows
programmers, systems personnel, and end-users to transfer files without the need for
coding. Files may be moved between servers and applications or even sent to the desktop.
Conditions can be designed to check for the arrival of an FTP file before executing a
business process. This function is extremely convenient when consolidating data
transferred from different locations.
The easy-to-use FTP feature improves developer, administrator, and end-user efficiency by
saving time during setup and maintenance that would otherwise have been spent
developing scripts.
The UC4 platform provides integrated online-viewing for standard job output (stdout), and
report output. Operations personnel, as well as end-users, can use this viewing function.
On-line viewing results in significant cost savings over printed output since it allows users to
either skip the printing process entirely, validate report contents prior to printing, and/or print
only selected parts of desired reports.
Java-Enabled Client
To take advantage of the widespread use of the Internet in corporate computing, the UC4
platform offers a fully enabled Java client. It can be accessed from any standard browser,
and it automatically installed when the user enters the correct URL. Maintenance demands
on IT staff are significantly reduced as updating to the latest release of the Java client
happens the next time the users log in.
Conclusion
The rapid pace of today’s business environment has created demand for a new breed of
tool to automate business processes. This new breed of tool must go beyond the world of
scripting and simple time-based events, venturing into a new realm where applications and
events can be driven dynamically in response to constantly changing business conditions.
To achieve these new levels of automation, the tool selected needs to reach out across a
wide array of platforms and applications to offer maximum functionality and flexibility.
Additionally the tool must be easy to use and responsive to change. Users need to be able
to interact with the new breed of tools so that the corporation can leverage their skills and
free IT resources that can be better utilized on profitable initiatives like e-business. Finally,
this new breed of tool needs to be much more application and data aware, with facilities to
access the underlying application’s data and use that data to drive the dynamics of the
business.
The UC4 platform provides all these capabilities and more in a straightforward and easy to
implement solution. The product has proven itself in hundreds of corporations worldwide as
the right solution for enterprises seeking a state of the art solution to their automation
needs. By adopting the UC4 platform early in a development cycle, organizations can
maximize their ROI, and build in a comprehensive automation philosophy from the ground
up.