Professional Documents
Culture Documents
View Based ETL
View Based ETL
View Based ETL
a
Ver
g
sio
View Based ETL n
e
1
1.1
of
10
1. Document History.................................................................................................................... 1
Scope..............................................................................................................................................2
Purpose.......................................................................................................................................... 2
Background.....................................................................................................................................2
Data Flow........................................................................................................................................3
UTL_DML.LARGE_MERGE (Oracle).............................................................................................4
CDC Managed Interfaces............................................................................................................... 6
1. Document
History
Date Revision Author Reason/Description
05/23/2019 1.0 Ron Behrendt Initial Draft
1 of 10
75214-TMP, v3.0
P
a
Ver
g
sio
View Based ETL n
e
2
1.1
of
10
Scope
This scope of this document is to describe an ETL method for utilizing views as an ETL source to
target tables. The intended audience is ETL developers.
Purpose
This purpose of this document is to set a standard method for refreshing tables based on views
in the same database platform. This is an alternative Informatica workflows and materialized
views.
Background
This method is designed for small and medium sized data sets (< 10 million records) when daily
refresh method is full and complete. Traditionally, such tables might be refreshed by Informatica
or be created in Oracle as materialized views.
When the ETL logic is light and within Oracle, Informatica only adds overhead. All the data
needs to be transferred to and from Informatica servers. A refresh done via PL/SQL keeps the
data within Oracle.
Oracle, materialized views keep the data within Oracle, but are not very efficient when daily
refresh method is “complete”. Complete refreshes delete and re-insert the entire data set. It is
not possible to have CDC or accurate timestamps on target table when all records are
processed. Also, excessive writes on the target database are incurred even if no data changed.
Fast refreshes for materialized views require materialized view logs on all source tables. This is
not always feasible.
This method can be used to refresh tables within the same database or over DB links.
2 of 10
75214-TMP, v3.0
P
a
Ver
g
sio
View Based ETL n
e
3
1.1
of
10
Data Flow
Source View: The source view needs to have the same column names as the target table. The
view should be as close to source tables as possible.
Create the view in the database containing the source tables.
The view needs to be created such that primary keys are guaranteed not generate duplicate
values.
Target Table: Create the target table normally. The minimum requirement is that it have a
primary key. Optionally, it can have a standard update timestamp that will be maintained based
on changes to target table. Timestamp maintained columns need a default value of SYSDATE
and needs to be one of these standard column names:
INSERT_TIMESTAMP
UPDATE_TIMESTAMP
INSERT_TS
UPDATE_TS
INS_TS
UPD_TS
3 of 10
75214-TMP, v3.0
P
a
Ver
g
sio
View Based ETL n
e
4
1.1
of
10
4 of 10
75214-TMP, v3.0
P
a
Ver
g
sio
View Based ETL n
e
5
1.1
of
10
UTL_DML.LARGE_MERGE (Oracle)
LARGE_MERGE is used to merge from source view to base table. Only the differences between
source and target are processed. Differences are determined by running MINUS queries.
Reference:
procedure large_merge(
source_table varchar2, -- Source table
target_table varchar2, -- Target table
dml_mode integer -- Combination of INSERT,UPDATE,DELETE
);
-- MERGE procedures.
-- These procedures will merge all records in the source table
-- to the target table.
--
-- Source_table and target_table:
-- The source table and target table should have
-- common column names. They need to have at least the PK columns in common.
--
-- DML_MODE parameter: (1-7)
-- 1 - Insert
-- 2 - Update
-- 3 - Insert and Update
-- 4 - Delete
-- 5 - Insert and Delete
-- 6 - Update and Delete
-- 7 - Insert, Update, Delete
--
-- The operations done on the target are controlled by the DML_MODE
-- parameter.
-- Insert: Any record in the source not in the target is inserted.
-- Update: Any record in the source and target is updated.
-- Delete: Any record in the target but not the source is deleted.
--
5 of 10
75214-TMP, v3.0
P
a
Ver
g
sio
View Based ETL n
e
6
1.1
of
10
Example
begin
utl_dml.large_merge('XDS_MAIN.ETL_TIME@DIH','GLBL_MAIN.B_TIME',7);
end;
If the target table is empty, LARGE_MERGE procedures will automatically run standard pre/post
load procedures.
UTL_DDL.DROP_INDEXES
UTL_DDL.SET_TRIGGER (Disable)
UTL_DDL.SET_TRIGGER(Enable)
UTL_DDL.ANALYZE_TABLE
UTL_DDL.BUILD_INDEXES
If you want to force a truncate and re-load, then truncate the table before running.
Example
Begin
utl_ddl.truncate_table('GLBL_MAIN', 'B_TIME');
utl_dml.large_merge('XDS_MAIN.ETL_TIME@DIH','GLBL_MAIN.B_TIME',7);
end;
6 of 10
75214-TMP, v3.0
P
a
Ver
g
sio
View Based ETL n
e
7
1.1
of
10
Add to CDC_TABLES
insert into cdc_tables (TABLE_ID,NODE_ID, TYPE_CODE, TABLE_NAME, TABLE_OWNER)
values ('ETL_TIME','DIH','DATA','ETL_TIME','XDS_MAIN');
7 of 10
75214-TMP, v3.0
P
a
Ver
g
sio
View Based ETL n
e
8
1.1
of
10
8 of 10
75214-TMP, v3.0
P
a
Ver
g
sio
View Based ETL n
e
9
1.1
of
10
9 of 10
75214-TMP, v3.0
P
a
Ver
g
sio
View Based ETL n
e
10
1.1
of
10
Utl_ddl.truncate_table(‘GLBL_MAIN’,’B_TIME’;
cdc.run_view_intf('B_TIME');
end;
/
10 of 10
75214-TMP, v3.0