DataStage – Job Sequence Invocation & Control

Job Sequence Invocation & Control

 Options
 Run through the DS client Director menu
 Command line interface DSJob Commands
 Used directly or within a OS shell or batch script
 DSJob available with client installation
 DataStage API
 callable through a C/C++NotProgram
covered as part of this course
 Distribute DLLs and header files to enable remote execution without a DS Client
 Through other DataStage Executable Components - which have to be in turn invoked
through any of the listed means
 DataStage BASIC Job Control
 Written in DataStage Basic Script
 Embedded as a Job Control script within the job definition OR
Not covered as part of this course
 Called as a Server Routine OR
 In Parallel Jobs, wrapped into a Basis Transform Stage
 Invoked as an activity within a Sequence Job
 which has to be in turn invoked through any of the listed means

August 7, 2021 2
Command Line Interface
 Use dsjob for controlling DataStage jobs
 Options available are:
 Logon
 Starting a job
 Stopping a job
 Listing projects, jobs, stages, links and parameters
 Setting an alias for job
 Retrieving information
 Accessing log files
 Importing job executables
 Generating a report
 CLI commands returns status code to OS
 Use dsjob – run to start, stop, validate or reset DataStage jobs

August 7, 2021 3
Job Sequences
Job Sequences …
 Specifies the sequence of Server Jobs to run (control flow)
 Contains control information, like specifies different courses of action depending on
Job’s status (success or failure)
 Consists of Activity Stages and Triggers
 Supports parameters for Activity Stages and also for Job Sequences
 Restartable with the help of Checkpoint information (maintained by DS)
 Supports automatic Exception Handling

Activit
y

Trigger

August 7, 2021 4
Creating Job Sequences
Creating Job Sequences …
 Select File  New

August 7, 2021 5
Job Sequence Development
Developing a Job Sequence involves …
 Adding Activity Stages
 Linking Stages
 Setting up Properties for each Activity
 Setting up Trigger Information for each
Activity
 Compiling the Job Sequence
 Running the Job Sequence

Repository

Palette Design Pane

August 7, 2021 6
Activity Stages
Activity Stages …

Job – Specifies a DataStage Server Job

Routine – Specifies a routine from the Repository (but not transforms)

ExecCommand – Specifies OS command to execute

Email Notification – Specifies the email notification to be sent using SMTP

Wait-For-File – Waits for a specified file to appear or disappear

Nested Conditions – Allows branch the execution of sequence based on condition

August 7, 2021 7
Activity Stages
Activity Stages …

Sequencer – Allows to synchronize the control flow of multiple activities

Start Loop and End Loop – Together these two stages allows to implement For…Next
or For…Each loop

Terminator – Allows to ensure the sequence is stopped cleanly for certain situations

User Variable – Allows to define variables within a sequence

Exception Handler – Executed if a job in the sequence fails or aborts and

“Automatically handle activities that fail” option is set. There can be only one Exception
Handler in a Job Sequence.

August 7, 2021 8
Triggers
 Triggers …
 Provides control information to Stage Activities
 Specifies different courses of action depending on Job’s status (success or failure)
 Trigger names must be unique for each Activity

 Three types of Trigger:

 Conditional – fires the Target
Activity if the Source Activity fulfills
the specified condition
 Unconditional – fires the Target
Activity once the Source Activity
completes
 Otherwise – used as default where
Source Activity has multiple triggers,
but none of the conditional ones
have fired

August 7, 2021 9
Scenario for a Job Sequence 1
Scenario …
 3 Dimension Load Jobs and 2 Fact Load Jobs
 Fact Load Jobs to start after all Dimension Load Jobs completes successfully
 If any Dimension Load Jobs fail, terminate all running Jobs

Job Sequence for the above scenario involves …

Job Activity – Execute the Dimension and Fact Load Jobs

Sequencer Activity – Synchronize the control flow of Jobs

Terminator Activity – Stop the Job Sequence for failures

August 7, 2021 10
Sample Job Sequence 1
Sample Job Sequence 1 …

Executes
Dimension_Load_3 Waits for the
Job successful
completion and
triggers other Jobs
Terminates the
Sequence if
Dimension_Load_
3 fails

August 7, 2021 11
Job Activity
Job Activity …
 Executes a DataStage Job

Job
selection

Other options:
Reset then run,
Validate only,
Reset only

Trigger
types

Supply
parameters
to Job

August 7, 2021 12
Sequencer and Terminator Activities

Sequencer Activity … Terminator Activity …

 Synchronizes control flow of multiple  Allows Job Sequence to stop
activities

All mode: All inputs

TRUE  trigger
Any mode: Any
inputs TRUE 
trigger
Message
outputted in
Director Log

August 7, 2021 13
Scenario for a Job Sequence 2
Scenario …
 5 input files are available in a folder with the same layout
 Single Server Job available to sort a input file
 Wait for a trigger to start the Job
 Send a message to a computer after Job completion (success or failure)
 Handle exception

Job Sequence for the above scenario involves …

Job Activity – Execute the Sort Job

Wait-For-File Activity – Wait for the trigger file before executing the Job

Start and End Loop Activity – Create For…Next loop to process 5 files

ExecCommand – Send a message to a computer using OS command

Exception Handler – To handle exception when a failure occurs

August 7, 2021 14
Sample Job Sequence 2
Sample Job Sequence 2 …
For…Next Loop to
execute Sort Job
for 5 input files

Waits for trigger Executes OS

file to appear command to
send
message

Executes Sort
Job
When any failure
occurs, control is
transferred here

August 7, 2021 15
Wait-For-File & Start and End Loop Activities
Wait-For-File Activity …
 Waits for a specified file to appear or
disappear
 Appear option – does not delete the file
after finding it
Start and End Loop Activity …
 Implements For…Next or For…Each
loop
 Current value of counter stored in
stage_label.$Counter

Numeric
Loop
List Loop

Waits for the file

forever …

August 7, 2021 16
ExecCommand and Exception Handler Activities
ExecCommand Activity …
 Specifies OS command to execute
 For command failures, enabling Log
warnings or Automatically handle failed
activities in Job Sequence properties
records warnings in Director Log
Exception Handler Activity …
 No input triggers and sends one
unconditional output trigger
 Variables $ErrSource, $ErrNumber,
$ErrMessage can be passed to
subsequent activities
 Enable Automatically handle failed
activities in Job Sequence properties to
invoke exception handler
Message
outputted
in Director
Log
August 7, 2021 17
Other Activity stages

Other Activity Stages not covered in the scenarios…

Routine
 Specifies a routine from the Repository (but not transforms)
 Routine arguments can be accessed by subsequent activities
 Email Notification
– Specifies the email notification to be sent using SMTP
 Email template file dssendmail_template.txt under Projects folder allows to create
different email templates for different projects
 Nested Conditions
– Allows branch the execution of sequence based on condition
 Example: If today is weekday execute weekday_Job else Weekend_Job
 User Variable
– Allows to define global variables within a sequence
 For example, the activity can be used to set Job parameters

August 7, 2021 18
Job Sequence Properties
Job Sequence Properties…
 Select Edit  Job Properties
Specifies Display
parameter s code
s
Enables
Director to
run multiple
instances

Records
checkpoint
information
for DataStage
to recover
the Sequence

• Error handling for

activities that does
not handle itself
• Control passed to
Exception Handler (if
any), else aborts

August 7, 2021 19