Professional Documents
Culture Documents
ISUG Melbourne Operators
ISUG Melbourne Operators
Audience
! Who has DataStage experience? ! Who understands difference between Parallel and Server
jobs?
! Who knows what an Operator is? ! Who knows what OSH is?
Introduction
! What happens when you click Run? ! Whats an operator? ! Whats OSH? ! Creating your own operators ! Summary
Introduction
! Well cover Parallel jobs ! Operators are a Parallel concept
! Script runs in the Orchestrate Shell OSH ! Each Job becomes one OSH script ! Each Stage becomes one or more executable Operators ! Operators are (sort of) equivalent to commands in Unix shell script, but ! ! have multiple inputs and outputs, ! and hence a slightly different syntax to Unix shell scripts
2012 Data Migrators Pty. Ltd.
End
An OSH Script
InputOperator! {parameters}! 0> [] FirstLink.v! ;! ! ProcessOperator! {parameters}! 0< [] FirstLink.v! 0> [] SecondLink.v! ;! ! OutputOperator! {parameters}! 0< [] SecondLink.v! ;! ! !2012 Data Migrators Pty. Ltd. !
Visualise OSH
! Writing stand-alone OSH, or
OSH At Runtime
! A Node Configuration file tells DataStage ! ! How to execute multiple parallel instances of your job ! How to map operators to O/S processes ! DataStage may combine operators ! Good for performance, bad for debugging ! Can disable this with $APT_DISABLE_COMBINATION ! DataStage may add additional operators to your job ! E.g. Sort or Partition to ensure correct operation of Join Operators ! Can disable this with $APT_NO_SORT_INSERTION
The Orchestra
!
2012 Data Migrators Pty. Ltd.
! Custom Stages ! Built in C/C++ ! Fewer restrictions than a Build Stage. E.g.
! Can create data sources and data targets ! Can create combinable operators
! Both of these !
! ! ! !
Creates a native OSH operator Offer high performance Custom Icon DataStage native interface
Example
! Experian QAS Batch ! Postal address cleaning solution ! A bespoke database ! A C/C++ API which provides !
! Start(), Open(), Clean(), Close(), Shutdown()
! Thats it!
! We integrated QAS Batch so it runs as an operator ! Scales performance of QAS Batch linearly ! QAS Batch is now grid-enabled ! [demo]
2012 Data Migrators Pty. Ltd.
Summary
! Dont fear the OSH! ! It represents your real DataStage job ! It tells you whats really happening under the hood ! Understanding them can help performance diagnosis ! OSH scripts can be auto-generated ! Build an operator ! Theyre fast ! Theyre reusable ! They can be used to integrate virtually anything, seamlessly ! If you can do it in C/C++, then you can build an operator for it ! They open new possibilities
2012 Data Migrators Pty. Ltd.
Fin