<<Feed Name>> Inbound Interface

Design
Project
<<Project Name>>

Project No:
<<Project Name>>

Source System:
<<Source System Name>>

Document Version 0.03

Prepared by:
<<Author>>
Teradata Professional Services

Enterprise Data Warehouse

REVISION LIST
Ver. #
0.01

Date
19-11-2010

Author

Description
Initial Draft

Document Version: 0.01

Page 2 of 13
<<Feed Name>> Inbound Interface Design

Enterprise Data Warehouse

Table of Contents
DOCUMENT VERSION............................................................................................................................. 1
REVISION LIST......................................................................................................................................... 2
1

INTRODUCTION................................................................................................................................. 4
1.1
1.2
1.3

INTERFACE REQUIREMENTS..........................................................................................................5
2.1

IN SCOPE EXTRACTS........................................................................................................................................ 5

HIGH LEVEL INTERFACE DESIGN...................................................................................................6

3.1
3.2
3.3
3.4
3.5

PURPOSE OF THIS DOCUMENT.......................................................................................................................... 4

SCOPE OF THIS DOCUMENT.............................................................................................................................. 4
INTENDED AUDIENCE........................................................................................................................................ 4

OVERVIEW....................................................................................................................................................... 6
GENERAL CONTACT INFORMATION..................................................................................................................... 6
FILE LOCATION INFORMATION............................................................................................................................ 6
FILE INTERFACE INFORMATION.......................................................................................................................... 7
EXTRACT DESIGN SCHEDULE............................................................................................................................ 7

DETAILED INTERFACE DESIGN.......................................................................................................8

4.1 MONITORING AND CONTROL INFORMATION........................................................................................................ 8
4.1.1
Detail..................................................................................................................................................... 8
4.1.2
Control file............................................................................................................................................. 8
4.2 EXTRACT FILE SPECIFICATIONS....................................................................................................................... 10
EC_ACCOUNT Extract...................................................................................................................................... 10
4.3 GENERAL FILE INFORMATION........................................................................................................................... 11
4.3.1
File Format.......................................................................................................................................... 11
4.3.2
Character set....................................................................................................................................... 11
4.3.3
Record Terminator............................................................................................................................... 11
4.3.4
Column Separator............................................................................................................................... 11
4.3.5
Null Fields............................................................................................................................................ 11
4.3.6
Bad and Missing Data......................................................................................................................... 11
4.3.7
Data Type Formats.............................................................................................................................. 11
4.3.8
Empty Files/Record Types................................................................................................................... 11
4.3.9
Automation of Extract.......................................................................................................................... 11
4.3.10 Duplicate Rows................................................................................................................................... 12
4.3.11 Data Masking...................................................................................................................................... 12
4.3.12 History................................................................................................................................................. 12
4.3.13 Delta Mechanism................................................................................................................................. 12
4.3.14 Data Consolidation.............................................................................................................................. 12
4.3.15 Data Type Determination..................................................................................................................... 12
4.3.16 Load Dependency Summary............................................................................................................... 12

APPENDICES................................................................................................................................... 13
5.1
5.2

QUESTION LOG.............................................................................................................................................. 13
SOURCE RCIS EXTRACT RUN BOOK............................................................................................................... 13

Document Version: 0.01

Page 3 of 13
<<Feed Name>> Inbound Interface Design

Enterprise Data Warehouse

INTRODUCTION

1.1

Purpose of this Document

The Enterprise Data Warehouse (EDW) will ultimately provide <<Client Name>> with a single integrated database
to support both enterprise and business unit analytics and reporting across all lines of business, while phasing out
or consolidating areas of redundancy.
EDWs Data Acquisition Layer refers to the area that resides between Source and Data Integration Layers, and is
used as a collection area for source data. This layer is off-limits to EDWs Information Consumers as source data is
not yet integrated, and the underlying supporting structures are not designed for query performance.
The objective of the Data Acquisition Activity is to design, build, and implement the data acquisition process to
transfer data from source feeds and applications to the EDW Data Acquisition Layer.
The purpose of this IID is to document the detailed description for the transport of RCIS data to the EDW
Acquisition Layers file system.

1.2

Scope of this Document

The purpose of the IID is to define, with the highest level of detail possible and all information available, the extract
files that will be provided to the DI stream developer.

1.3

Intended Audience

This document should be read by the following:

Name
DI Mapping Lead
DI ETL Off-Shore
Development Lead

Title
Mapping team
Development
CUT team

Signature

Document Version: 0.01

Date

Page 4 of 13
<<Feed Name>> Inbound Interface Design

Enterprise Data Warehouse

2
2.1

INTERFACE REQUIREMENTS
In Scope Extracts

Source File/Table Name

EC_ACCOUNT
EC_ACCT_ROLE
EC_BILL_CYCLE

Extract Criteria
DailyChanges
Full Snapshot
Full Snapshot

Document Version: 0.01

Frequency
Daily
Daily
Daily

Volume(MB)
4,356
<1
<1

Page 5 of 13
<<Feed Name>> Inbound Interface Design

# of
Attributes
37
4
3

Enterprise Data Warehouse

HIGH LEVEL INTERFACE DESIGN

3.1

Overview

3.2

General Contact Information

Contact
Source Owner
IT BI/DW SME
Source System Business Contact Name & Telephone #
Source System Name
SSDA Development Representative

3.3

Information
Data Governance
Open Ticket with IBM

File Location Information

Location
Production or Development Files
Internal or External Source
Source ID

Information
Development
Internal
Data Governance To Supply

Pickup Location (e.g. IP address, directory,

database)

DEV:
QA:
PROD:

Flat Files

edw_rcis_ec_account_yyyymmddhhmiss.dat
edw_rcis_ec_account_yyyymmddhhmiss.ctl
edw_rcis_ec_acct_role_yyyymmddhhmiss.dat
edw_rcis_ec_acct_role_yyyymmddhhmiss.ctl
edw_rcis_ec_bill_cycle_yyyymmddhhmiss.dat
edw_rcis_ec_bill_cycle_yyyymmddhhmiss.ctl

Profile Data Location

\\Rsoesndeveci\IDE_FTP_Shared\DataProfiling\RCIS

Document Version: 0.01

Page 6 of 13
<<Feed Name>> Inbound Interface Design

Enterprise Data Warehouse

3.4

File Interface Information

Interface
Source Type (e.g. DB, File)
Type of Transmission (e.g. FTP, OBDC)
Character Code (ASCII, Binary, Hex)
File format (fixed, variable, delimited)
Extract file archiving
Extract file encrypting

3.5

Information
File
Delivered via Control M job running UNIX shell scripts
ASCII
Pipe Delimited
Yes
No

Extract Design Schedule

Definition around when the extract will arrive in the development area.
Note: Profile and Unit Test Dates apply to all files below
Source File/Table Name
Profile date
EC_ACCOUNT
22 Mar 2011
EC_ACCT_ROLE
22 Mar 2011
EC_BILL_CYCLE
22 Mar 2011

Masking Date

Document Version: 0.01

UNIT test date

Page 7 of 13
<<Feed Name>> Inbound Interface Design

Enterprise Data Warehouse

4
4.1

DETAILED INTERFACE DESIGN

Monitoring and Control Information

Detail

The detail records will contain the actual data that is to be loaded. This data will be in a pre-defined format as
shown in this document.

Control file

ETM data files must have an accompanying control file with the information to perform audit and reconciliation of
the data file. The control file name should match the name of the data file with a ctl file extension and it is piped
delimited. Contents of the required control file should contain the following:
Field
SRC_FL

SRC_FL_GEN_DTTM

Format
File name convention:
edw_<system>_<table_name>_yyyymmddhhmiss.dat
Sample file name:
edw_rcis_ec_cust_name_20110218114257.dat
DATETIME: MMDDYYYY:HH:MI:SS

SRC_SYS_ID
SRC_COL_SUM
NAME_ID
SRC_COL_POS
2
SRC_REC_CNT
NUMBER
SRC_COL_SUM_VAL
NUMBER(30,2)

Description
Name of source file

Source file generation date

in landing layer
Source system from which
the source file was
extracted
Source file column name to
calculate sum (if applicable)
The column number of the
field used for the sum (if
applicable)
Total records present in the
source file
Expected value of the sum
of the summed file (if
applicable)

Example
For example edw_rcis_ec_account.ctl contains
edw_rcis_ec_account.dat|04302010:14:04:05|RCIS|PRIMARY_ACCT_ID|6|1413415|220147867957.00
The list of control files and what is expected in each field in each file are listed on the next two pages.

Document Version: 0.01

Page 8 of 13
<<Feed Name>> Inbound Interface Design

Enterprise Data Warehouse

Table

SRC_FL

EC_ACCOUNT

edw_rcis_ec_account_yyyymmddhhmiss.dat

EC_ACCT_ROLE

edw_rcis_ec_acct_role_yyyymmddhhmiss.dat

EC_BILL_CYCLE

edw_rcis_ec_bill_cycle_yyyymmddhhmiss.dat

Document Version: 0.01

SRC_FL_GE
N_DTTM
MMDDYYYY
:HH:MI:SS
MMDDYYYY
:HH:MI:SS
MMDDYYYY
:HH:MI:SS

SRC_
SYS_
ID

SRC_COL_SUM

SRC _
COL_
POS

SRC_REC_CN
T

SRC_COL_SUM_
VAL
<checksum total>

RCIS

ACCT_ID

<record count>

RCIS

NONE

<record count>

RCIS

BILL_CYCLE

<record count>

Page 9 of 13
<<Feed Name>> Inbound Interface Design

0
<checksum total>

Enterprise Data Warehouse

4.2

Extract File Specifications

EC_ACCOUNT Extract

Record level information is listed in the table below.

Characteristic
Extract Load Dependency
Fields Masked in DEV
Source table(s)/database/schema
Control-M Job Name and
Scheduling Details
File Name

Value
RCIS.06.04
RCIS_CICSK
TBD
edw_rcis_ec_account_yyyymmddhhmiss.dat

Details of the layout of this record type are in the table below.
Field Name
Field Type
ACCT_ID
NUMBER(TBD,TBD)
NAME_ID
NUMBER(TBD,TBD)
SRC_CD
VARCHAR2(20)
SYSTEM_SRC_CD
VARCHAR2(20)
SRC_ACCT_KEY
VARCHAR2(60)
PRIMARY_ACCT_ID
NUMBER(TBD,TBD)
CONSOL_ACCT_ID
NUMBER(TBD,TBD)
SRC_ACCT_EFFECTIVE_DT
DATE
SRC_ACCT_EXPIRY_DT
DATE
BILL_CYCLE
NUMBER(3,0)
BILL_LANGUAGE
VARCHAR2(3)
SRC_ACCT_TYPE
VARCHAR2(5)
SRC_ACCT_SUB_TYPE
VARCHAR2(5)
SRC_ACCT_BUS_TYPE
VARCHAR2(5)
PARTY_TYPE
CHAR(1)
SRC_ACCT_STATUS
CHAR(1)
ACCT_STATUS
VARCHAR2(10)
CONVERSION_TYP
VARCHAR2(8)
CREATE_DT
DATE
UPDATE_DT
DATE
UPDATE_USER_ID
VARCHAR2(20)
SRC_BILLING_ENTITY
BT_ACCT_NUMBER
BT_CONV_STATUS
BT_PLAN_CONV_DT
BT_ACTUAL_CONV_DT
MIG_DEFER_REASON_DESC
SRC_CBP_NUMBER

VARCHAR2(4)

NUMBER(TBD,TBD)
CHAR(1)
DATE
DATE
VARCHAR2(80)

NUMBER(TBD,TBD)

PK
Y
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N

Unique
Y
N
N
Y
Y
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N

Nulls
N
N
N
Y
Y
Y
Y
N
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
N
N

N
N
N
N
N
N
N
N

N
N
N
N
N
N
N
N

N
Y
Y
Y
Y
Y
Y
Y

Document Version: 0.01

Page 10 of 13
<<Feed Name>> Inbound Interface Design

Enterprise Data Warehouse

4.3

General File Information

File Format

This section outlines the characteristics of the extract file that are operating system dependant.

Character set

The file will be in ASCII format.

Record Terminator

Records will be terminated with the UNIX terminator is x0A.

Column Separator

The column separator used for the project will be the pipe character (|). This is ASCII character x7C. This
character will be removed from the body of any field.

Null Fields

Fields that are null values in the source will be treated the same as fields that are empty in the source. Any such
fields will be sent as column separators with no value between. To demonstrate, the fifth field in the example below
will be treated as null in the target.
001|12345233|HP|2007-12-14 08:34:32||YES|The quick brown fox jumps over the lazy dog.

Bad and Missing Data

Key fields that have invalid or null values will result in the record being rejected by the load program. Other fields
that have invalid values will be sent to the target replaced with null by default. The mapping document will describe
any actions other than the default behaviour. The extract should prevent these values from appearing.

Data Type Formats

See the table below for a description of data types and the expected format.
Data Type
Numeric
Date
Timestamp
Text fields

Format
Scientific notation is not to be sent. Expand the data to decimal format.
Negative numbers are to be proceeded with a - sign, not enclosed in brackets.
Numbers may be zero left padded, but this is not required.
MM/DD/YYYY
MM/DD/YYYY HH:MI:SS
ASCII characters x20 through x7E.
Fields are not to be enclosed by quotation marks.
Leading, trailing and multiple embedded blanks may be trimmed by the load
program.
No embedded record termination characters or column separator characters.

Empty Files/Record Types

If a particular file has no data to contribute at extract time, the file will still be produced with a control file indication 0
record counts. The load programs are quite happy to process with no data.

Document Version: 0.01

Page 11 of 13
<<Feed Name>> Inbound Interface Design

Enterprise Data Warehouse

Automation of Extract

Extracts will be automated.

Duplicate Rows
Source system extracts are requested to remove all primary key duplicates.

Data Masking
Data masking to remove any personally identifiable information will be performed by the extract programs. Masking
will have sufficient intelligence to maintain any relationships between tables for which the masked field is the foreign
key.

History
TBD

Delta Mechanism
Extracts will be date driven using some control mechanism. Each record type should be individually controlled in
terms of the time period of rows to be extracted. This mechanism will allow the extract to reach back further in time
than the normal processing period if required when in production. Interim changes to the rows will not be captured,
only the state of the row at extract time will be sent.

Data Consolidation
As much as possible, the data in the extract will be combined data from multiple tables in the source to the benefit
of the load programs. If it makes sense to combine columns in a where clause in the source, perform the join. If a
filter will reduce the number of rows, use the filter. If data combination results in additional logic necessary in the
load, leave the data in the format of the source. These principals will be reflected in the mapping document.
Each record type will hold the all the data it requires to load the target table. No look ups between data in different
record types will be performed by the load programs.

Data Type Determination

Data types will be determined by following three steps:
1. Examine the SSDA Data Dictionaries
2. Examine any profiling information and modify data types determined in step 1 as necessary
3. Examine data loaded from extracts during development. If any further discoveries are made, data types will
be further modified as necessary.

Load Dependency Summary

This table contains the summary of extract load dependencies and load order.
Record
Type

Source

Source Table(s) and Application

Load Order

Record Type
Dependency

To get Foreign Keys information along with Source System Tables DDLs.

Document Version: 0.01

Page 12 of 13
<<Feed Name>> Inbound Interface Design

Enterprise Data Warehouse

5
5.1

APPENDICES
Question Log

Question

5.2

Whom?

Date

Source RCIS Extract Run Book

The link below is to the Run book for the sources that exist as production feeds as of 3/16/11.
<<Link to External ops / scheduling documents>>

Document Version: 0.01

Page 13 of 13
<<Feed Name>> Inbound Interface Design

Closed