Idmc Iics 101 Labs v10

You might also like

Download as pdf or txt
Download as pdf or txt
You are on page 1of 120

Informatica Intelligent Cloud Services - IDMC/IICS

Data Integration - Boot Camp for Developers Lab


Exercises—Table of Contents
Connections Setup ...................................................................................................................................................................3
Pre-Requisite: .........................................................................................................................................................................3
Exercise #1 Download, Install and Configure the Secure Agent ...........................................................................................4
Business case.......................................................................................................................................................................4
The Logic ..............................................................................................................................................................................4
Secure Agent Installation ......................................................................................................................................................4
Secure Agent Configuration ..................................................................................................................................................4
Step by Step Solution ...........................................................................................................................................................5
Exercise #2 Create Connections. .......................................................................................................................................11
Business Case ....................................................................................................................................................................11
Step by Step Solution .........................................................................................................................................................15
Exercise #3 Source and Target Definitions, Simple Mapping .............................................................................................15
Business case.....................................................................................................................................................................15
The Logic ............................................................................................................................................................................15
The Informatica Cloud solution ...........................................................................................................................................15
Step-by-step solution ..........................................................................................................................................................15
Exercise #4 Expression and Aggregator Transformations ..................................................................................................22
Business case.....................................................................................................................................................................22
Background.........................................................................................................................................................................22
The logic .............................................................................................................................................................................23
The Informatica Cloud Solution ...........................................................................................................................................24
Step-by-step solution ..........................................................................................................................................................25
Exercise Adding data from another source ....................................................................................................................35
Exercise #5 Flat File Source, Joins ....................................................................................................................................37
Business case.....................................................................................................................................................................37
The logic .............................................................................................................................................................................37
The Informatica Cloud solution ...........................................................................................................................................37
Hints ...................................................................................................................................................................................38
Exercise #6 Lookup and Rank Transformations .................................................................................................................53
Business case.....................................................................................................................................................................53
The logic .............................................................................................................................................................................53
The Informatica Cloud solution ...........................................................................................................................................53
Hints ...................................................................................................................................................................................54
Exercise #6.2 Unconnected Lookup ......................................................................................................................................57
Exercise #7 Complex mapping to load Salesforce Data, Router Transformation ...............................................................61
Business Case ....................................................................................................................................................................61
The logic .............................................................................................................................................................................61
Informatica Cloud Solution ..................................................................................................................................................61
Step by Step Solution .........................................................................................................................................................61
Exercise #8 Local variables, Sorter Transformation ...........................................................................................................69
Business case.....................................................................................................................................................................69
The logic .............................................................................................................................................................................69
The Informatica Cloud Solution ...........................................................................................................................................69
Hints ...................................................................................................................................................................................70
Exercise #9 Using Sorted Input Optimization .....................................................................................................................80
Business case.....................................................................................................................................................................80
The logic .............................................................................................................................................................................80
The Informatica Cloud solution ...........................................................................................................................................80
Hints ...................................................................................................................................................................................81
Exercise #10 Slowly Changing Dimension Type I (MappingTask) ........................................................................................85
Business case.....................................................................................................................................................................85
The logic .............................................................................................................................................................................85
The Informatica Cloud solution ...........................................................................................................................................85
Hints ...................................................................................................................................................................................87
Exercise #11 Simple Taskflow and Error Handling ...............................................................................................................97
Business Case ....................................................................................................................................................................97
The Informatica Cloud solution ...........................................................................................................................................97
Hints ...................................................................................................................................................................................98
Exercise #12 Complex Taskflow......................................................................................................................................... 101
Business case................................................................................................................................................................... 101

Copyright Infocrest 2024 1


The logic ........................................................................................................................................................................... 101
The Informatica Cloud solution ......................................................................................................................................... 101
Hints ................................................................................................................................................................................. 101
Exercise #13 Mapping Parameters..................................................................................................................................... 110
Business case................................................................................................................................................................... 110
The logic ........................................................................................................................................................................... 110
The Informatica Cloud Solution ......................................................................................................................................... 110
Hints ................................................................................................................................................................................. 111

Copyright Infocrest 2024 2


Pre-Requisite:
1. Create an informatica Cloud Account:

a. Go to https://www.informatica.com/trials/api-and-application-integration.html
b. create an account with your personal email
c. Enter the requested information to start your Free Cloud trial.
d. You will receive an email with your login credentials. Click to confirm new trial account
e. Create a new password for your new Informatica cloud account.

Copyright Infocrest 2024 3


Exercise #1 Using a Secure Agent to run Integration Jobs
Business case

In Order to access data and perform Integration Tasks, you need a secure agent to do the job.

1. Informatica Hosted Secure Agent is in the cloud and can do the work

2. However, you need a local secure agent on premise (behind your firewall) to make sure your
sensitive local data remains behind your firewall

3. For the purpose of this week’s boot camp use Informatica’s Hosted Secure Agent.

4. If you feel adventurous and want to create your own secure agent on your local machine
(Windows 10/11 or Windows Server or unix/Linux only! no Apple Mac is supported)

The Logic

• The secure agent setup is common for all users, so the Administrator is highly advised to
download and distribute the secure agent setup for all users.

• Install and configure the secure agent on your machine.

• After you correctly configured the secure agent, access the Informatica cloud web interface and
check the status of ICS Secure agent.

Secure Agent Installation

• Download the secure Agent for Windows or Unix Operating System.

• Run the Secure Agent Executable file.

• Choose the appropriate installation directory or leave it as default.

• Pre-Installation summary.

• Installation Process.

• Check the Installation status.

Secure Agent Configuration

• Register the secure agent by entering your Informatica Cloud login Username and Password.

• Check the secure agent status and optionally configure the proxy server information if it Exist.

Copyright Infocrest 2024 4


Step by Step Solution

1) Go to the URL sent to you by Informatica when you registered for the free trial, and login to
Informatica cloud (https://dm-us.informaticacloud.com/ma/home)

2) Go to the Administrator service. In the Runtime Environments tab download Secure Agent (top
right).

3) Double click the executable file to run it.

Copyright Infocrest 2024 5


4) Determine the directory file for installation.

5) Check the Pre-Installation summary.

Copyright Infocrest 2024 6


6) After selecting the Directory file location and checking the requirement, start the installation.

7) Configure your secure agent by entering your Informatica cloud Username and Install token.
Follow Steps a & b to get the install token:

a. Go to Administrator page > Runtime Environment tab > Click on Generate Install token.

Copyright Infocrest 2024 7


b. Copy the token:

Fill the user name and install token information then click register.

Copyright Infocrest 2024 8


8) Check the Agent status if is running or not. If not check your credential username and install
token.

Copyright Infocrest 2024 9


9) In the runtime environment tab on Informatica cloud you can see the installed secure agent and
check its status.

Copyright Infocrest 2024 10


Exercise #2 Create Connections.
Business Case

To perform integration tasks we need to define the source and target data Objects. In this lab we will
create all the required connections from different data types Flat file, SQL Server and Salesforce .

1. Login to Informatica Cloud using your credentials


2. Go to the Administrator Service

3. In the Administrator create the following connections:

A. Connection for SQL Server: sourcedb


Note: Runtime Environment is your Secure Agent Name or Informatica Cloud Agent
Id/pwd is : source/source

Copyright Infocrest 2024 11


B. Connection for Snowflake: SOURCEDB
Id/pwd is: SOURCE/Source111

C. Connection for SNOWFLAKE: TARGETDB


Id/pwd is: TARGET/Target111

Copyright Infocrest 2024 12


D. Flat file connection.
• Enter the connection name: SourceFiles
• Select the data type as flat file
• Select the appropriate secure agent.
• Enter the directory where the lab files are located on your localhost machine
• When you click on Browse you may get the following error: ‘The runtime environment
cannot get the requested file system information. The file system information might not be available because none
of the agents in the runtime environment are running or because the Secure Agent service for Windows is not
running with administrator privileges.’
• Follow the steps below to resolve the issue:
o In the services console on windows go to the informatica cloud secure
agent service, then right click on it and choose properties.

o In the logon tab choose this account and enter the username and
password of a user that is in the windows Administrator user group.

o Then stop and restart your service from the service console.

Copyright Infocrest 2024 13


• Choose the Date Format and Code page shown below

• Test the connection

Copyright Infocrest 2024 14


Exercise #3 Source and Target Definitions, Simple Mapping
Business case

You need to transfer some columns of data from a production table into your own schema in
development. Your database administrator is too busy, so you decide to do the data transfer yourself,
using Informatica cloud and a simple pass-through mapping.

The Logic

• What data do we need, where is it located?


o The table we want is called Orders and is located in our labs source schema
o The columns we need are OrderID, CustomerID, EmployeeID and OrderDate
• What are the requirements for the target table?
o The target will be a brand new table, containing the fields mentioned above
• What is the transformation logic?
o None, we will just copy the data straight from the source table into our target table

The Informatica Cloud solution

• Create a new mapping asset. Call the mapping ‘m_TransferOrdersDataFromProd’.


• Add a source definition object to the mapping. Connect to the Orders table in the labs source
schema.
• Use the preview data function to take a look at the source data
• Add a target definition object to the mapping. Connect to the ‘T_OrdersDev_x’, where x is the
student number. So every student will load data to a separate target.
• Save the mapping
• Run the mapping.
• Preview the data for the target, in the mapping designer view.

Step-by-step solution

• Naming conventions
Informatica recommends the following conventions to name various objects.
o Target definitions: T_TargetName
o Transformations: <abbreviation>_TransformationName as in
▪ EXP_xxx, for an expression
▪ AGG_xxx, for an aggregator
▪ FIL_xxx, for a filter
o Mappings: m_MappingName
o Mapping task: mtsk_MappingTask

Copyright Infocrest 2024 15


The labs will follow these conventions.

Note: This is a good start, but you need to use meaningful names for these objects to
help in the “self-documentation” of your work.

1. Launch Informatica Intelligent Cloud Sevices:


a. Using the web-browser go to ‘https://dm-us.informaticacloud.com/ma/home’
b. Enter username & password
c. Go to the data integration cloud service
d. Go to Explore and open the FoundationLab folder.

2. Create a mapping
a. Click on new icon go to the mappings window and select Mapping.

b. Name the mapping ‘m_TransferOrdersDataFromProd’ in the mapping properties tab.

Copyright Infocrest 2024 16


3. The Source and target transformation are automatically added to your mapping.
a.

b. Click on the source transformation to Connect to the Orders table in the database. In the
General tab specify the name ‘Source_Oders’.In the orders tab Specify the MSSQL_Source
connection and specify the Orders Object.

Copyright Infocrest 2024 17


c. Preview the source data by clicking on the Preview Data button.

4. Connect to the T_OrdersDev_x (where x is the student id) table in the database. Specify the
MSSQL_Target connection and specify the T_OrdersDev_x Object.

Copyright Infocrest 2024 18


a. In the target tab specify the MSSQL_Target connection. Click on select to add the
‘T_OrdersDev_x’ object.

b. Go to field mapping in the target to map the output fields to the corresponding input.

5. Save and run the mapping. Choose the Lab_env as the runtime environment.

Copyright Infocrest 2024 19


6. Go to my jobs to monitor the status of the job. The job must process 830 rows.

7. Then download the log file of the job and in the session load summary you can identify the
number of records read from the source and the number of records inserted into the target.

8. To see the end result go back to the mapping and Preview the data of the target table. Click on
the target table -> go to target in the properties pane -> Preview data. The Result is as follows.

Copyright Infocrest 2024 20


Copyright Infocrest 2024 21
Exercise #4 Expression and Aggregator Transformations
Business case

The TradeWind Company wants to improve its fulfillment systems. It is worried that too many orders
might be late getting to customers. Customer satisfaction is a top priority for the TradeWind Company. It
is their competitive advantage.
Create a mapping to solve the following
1. How long, on average, does it take for an order to be shipped to a customer?
2. If orders are late, what is the average number of days late, per customer?
3. What is the maximum number of days an order has ever been late, per customer.

Background

A simplified schema of the TradeWind (source) database

Copyright Infocrest 2024 22


The logic

1. Data sources, what should they be and where are they?


a. CustomerID, orderDate, requiredDate and shippedDate
b. All fields from the Orders table
2. What are the requirements for the target table?
a. A customer ID field that uniquely identifies a customer. This key can be used to join to the
customer table as well
b. A field for the average number of days to shipping
c. A field for the average number of days for late orders. This can be zero if all orders are on
time for this customer
d. A field for the maximum number of days late. This can also be zero if orders are always on
time for this customer
e. A batch insert date field. This will ease the pain if we ever have to rollback the table
3. What is the transformation logic?
a. Weed out rows with disqualifying data, such as NULL shipping date (orders not shipped yet)
and NULL order date (DB defect)
b. For each row, compute
i. Time to shipment (shipped date – order date)
ii. If the order is shipping late, the number of days late (shipped date –
requested date)
iii. Or, if the order is on time or there is no required date, set the number of
days late to zero
c. Group data by customer and compute
i. Average time to shipment
ii. Average number of days late
iii. Maximum number of days late
d. Add a batch timestamp to each row loaded in the target (Systimestamp)

NOTE: The solutions for the labs have been provided with SQL Server as the database
for the source & target. Please make the appropriate changes by selecting the
appropriate database type (ORACLE, DB2, etc.) for either Source or Target.

Copyright Infocrest 2024 23


The Informatica Cloud Solution

1. Create a mapping called ‘m_OrdersByCustTimeMetrics’.


2. Add a source transformation to connect to the source orders table.
3. Add A filter transformation to remove disqualifying rows (NULL shipping and order dates)
4. Add An expression transformation, to compute the number of days to ship and number of days
late, row by row
5. Add an aggregator transformation, to group the data by customer and perform the aggregate
functions (AVG days to ship, AVG days late and MAX days late).
6. Add an expression transformation, to insert the session start time to identify the rows created
during a single batch.
7. Add a target transformation to connect to the target table ‘T_OrdersByCustTimeMetrics_x’ where
X is your student number.
8. Save and run the mapping. Monitor the rows processed in My Jobs window.

Note: you can also verify the target data using a SQL tool to query the table or Preview
Data from the mapping view by clicking on preview data in the target tab of Target
transformation properties window.

Copyright Infocrest 2024 24


Step-by-step solution

1. Create a mapping to implement the transformation logic


a. Click on new icon go to the mappings window and select Mapping.

b. Proceed to create the following mapping

c. Enter the name of mapping in the mapping properties tab: ‘m_OrdersByCustTimeMetrics’

Copyright Infocrest 2024 25


IMPORTANT: Add a short description for the mapping and each transformation object. This is
a good step towards self-documenting mappings.
2. Configure the source transformation to connect to the orders table.
a. Add the connection information in the source properties tab.

b. Go to fields tab and delete the unwanted fields. Select unwanted fields and click on delete
icon. Keep the fields shown below since they will only be used in the mapping.

Copyright Infocrest 2024 26


3. Click on the icon to get a list of suggested transformation. Add a filter transformation and name
it ‘FIL_RejectNULLDates’.

Copyright Infocrest 2024 27


4. Go to filter Tab in the filter properties pane. Change the filter condition to advanced and click on
define the filter condition and add the condition as shown below. Then validate the Expression.

Hint: To input function names and port names, select them in the expression editor’s
navigator and click the add button next to the field. This reduces the chances of
typos.

5. Add an expression transformation and name it ‘Exp_ComputeToShipAndLateDays’.


a. Select the Expression tab in the properties pane and add an output port named DaysToShip.
Make sure this port is an integer.

b. Click on configure next to DaysToShip to enter the expression editor and enter the
expression shown below.

Copyright Infocrest 2024 28


Hint: For details on how DATE_DIFF and other functions behave, click the ? button
and type the function name in the search box.

c. Validate the expression and create another output port named DaysLate, as shown below.

d. Enter the expression editor for the DaysLate port and add the expression shown below.

Copyright Infocrest 2024 29


6. Add an Aggregator transformation and name it ‘AGG_GroupByCust’.
a. Select a Group by Port (CustomerID) from the group by tab in the properties pane of the
aggregator.

b. Go to the incoming fields tab to exclude the fields that are not further required in the mapping.
This is optional but makes your mappings cleaner. In the include field rule change field
selection criteria to named field and configure it to include the required fields as shown below.

c. Press on configure under the field rule detail and select the following fields to include.

Copyright Infocrest 2024 30


d. Go to the aggregate tab and Create three output ports AvgDaysToShip, AvgDaysLate, and
MaxDaysLate of numeric datatypes (averages should be decimal (9,2); maximum should be
integer). Configure the Expression on the new output ports to add an aggregate function as
follows.

7. Add another expression transformation and name it ‘EXP_AddInsertDate’.


a. In the Expression tab of the properties pane add the ‘InsertDate’ field of type date/time.

Copyright Infocrest 2024 31


b. Configure the InsertDate field to add the function ‘Systimestamp()’ that is found under Built-in
Functions -> date.

8. Click on the target transformation and name it ‘T_OrdersByCustTimeMetrics’.


a. Add the connection of the target table in the target tab and select the object
‘T_OrdersByCustTimeMetrics_x’, where x is the student number.

Copyright Infocrest 2024 32


9. Save the mapping and check if it is valid. Go to the validation panel that will show a message
‘Mapping is valid’ if it is valid or it will show error messages if the mapping is invalid.

Validation Panel

10. You can test the developed logic before running the mapping using the run preview. For example
click on the expression ‘Exp_ComputeToShipAndLateDays’ to validate the expressions created.
a. In the Preview tab click on run preview.

b. Choose the runtime environment Lab_env and the number of rows you want to read from the
source. Click run preview.

Copyright Infocrest 2024 33


c. You will get the following results.

11. Run the Mapping and choose the runtime environment as ‘Lab_Env’. Then go to my jobs to check
the status of job. It must process 89 rows. Then you can preview the data in the target table from
the mapping and the result should be as follows.

Copyright Infocrest 2024 34


Exercise 4.2 Adding data from another source
Modify the mapping to include the Customer’s company name & phone number in the target table. There
are 3 ways to do this…

Copyright Infocrest 2024 35


Copyright Infocrest 2024 36
Exercise #5 Flat File Source, Joins
Business case

Tradewind’s marketing department wants to know about sales made to customers outside the USA. You
will build a new target table to hold a summary of sales made to these customers using a flat file
customer list as a source and joining it to the relational tables Orders and Order_Details

The logic

1. What data do we need, where is it located?


a. This time, we’ll use a flat file holding the most current customer list.
b. We’ll extract the fields CustomerID, CompanyName, and Country from this file.
c. On the relational side, we will need two tables: Orders and Order_Details, the table that holds
price, quantity and discount columns for products sold (refer to the schema above for table
relationships)
2. What are the requirements for the target table?
a. We’ll need the following fields: CustomerID, CompanyName, Country and
CustomerSalesAmount
3. What is the transformation logic?
a. We’ll join Orders and Order_Details based on the OrderID field then join the results to the
customer flat file list
b. However, before we join the flat file to the relational sources, we will remove customers with a
country field equal to ‘USA’ from the flow
c. After the join, we will sum up the order sales for each non-US customer. The sales amount
formula for one detail line is UNITPRICE * QUANTITY * (1 – DISCOUNT)

The Informatica Cloud solution

1. The customers list flat file is of fixed width, so you must create a new Fixed-Width File Format
component called ‘CustomersFlatFile_Format ’.
2. Create a mapping called ‘m_ForeignCustomersSales’. The mapping will contain the following:
a. Source transformation that will read from CUSTOMERS_FLAT.DAT file.
b. A Filter transformation, to remove customers from the US. Note that, since our flat file source
is a fixed width file, we’ll need to TRIM the customer’s country before we compare it to ‘USA’.
c. Source transformation with multiple objects type containing the Orders and Order_Details
tables. Your database has a primary/foreign key relationship enforced between the Orders
and Order_Details tables, meaning you won’t have to do anything special in defining the
relationship. The cloud integration service will handle everything automatically.
d. A Joiner transformation to join flat file and relational streams, based on customer ID
e. An Aggregator transformation, to sum up the sales amounts for each customer.
f. A target transformation to connect to ‘T_ForeignCustSales_x’ where x is your student ID.
3. Save the mapping and make sure it is valid
4. Run the mapping.

Copyright Infocrest 2024 37


Hints

1. An iconic view of the ‘m_ForeignCustomersSales’ mapping

2. Create a fixed width File Format.


a. Go to new -> Components -> Fixed_Width File Format

b. Name the Fixed_Width File Format ‘CustomersFlatFile_Format’ and specify the connection
details to import the file CUSTOMERS_FLAT.DAT. Then add the lines to the data preview
window to specify the column boundaries.

Copyright Infocrest 2024 38


c. Edit the Columns name in the source file to show as follows:

d. Click save.

3. Create a mapping named ‘m_ForeignCustomersSales’.

Copyright Infocrest 2024 39


4. Click on source transformation to connect to the Customers_Flat file.
a. Specify the connection details.

b. Go to formatting options to specify the file type as fixed_width and add the file format that you
have created previously.

5. Add a filter transformation after the ‘Customers_Flat’ source to remove customers from the US.

a. Choose the filter condition to be advanced and then use an RTRIM function to get rid of
trailing spaces before you compare the country field to the string ‘USA’.
Click on dropdown list to navigate between functions and fields to add to the expression.

Copyright Infocrest 2024 40


b. Include only the needed fields shown below (i:e modify the include rule with named fields as
the field selection criteria).

6. Add a source transformation to connect to order and Order_Details tables by joining on OrderID
field.

a. Choose the MSSQL_Source connection and specify the source type to Multiple objects. Add
the Orders object by clicking on the following icon , and then choosing Add source object.
To add the related object Order_Details click on ,and then click on Add Related objects.

Copyright Infocrest 2024 41


The Relationship is directly configured since the tables have the same Primary key. If you
want to edit the relationship you can double click on it.

b. Delete all unwanted fields thus keeping the fields below that will be used in the mapping.

7. Create a joiner to join the flat filtered source to the relational sources. This join is a normal (equi)
join, based on CustomerID. The flat file source is the smallest stream based on the number of
rows and should constitute the master side of the joiner’s relationship.
a. Click on joiner + icon to see the Master and Detail inputs.

b. Choose the join condition based on CustomerID

Copyright Infocrest 2024 42


c. Include only the needed fields shown below (i:e modify the include rule with named fields as
the field selection criteria).

8. Add the aggregator transformation. You need to group by Custo and sum up order sales
based on the formula UNITPRICE * QUANTITY * (1 – DISCOUNT) (discount is a
percentage):

Copyright Infocrest 2024 43


a. In Group by section add the group by field

b. In aggregate sectio, create a new output field.

c. Configure the CustSalesAmount expression

Copyright Infocrest 2024 44


9. Connect the aggregator to the target transformation. Configure the target to connect to the object
T_ForeignCustSales_x, where x is student ID.
a. Add the connection details.

b. Include only the needed fields

c. Map the target fields to the input fields. Click on Automap -> ExactFieldName

10. Save and run the mapping.

Copyright Infocrest 2024 45


Validate the Target Data
The resulting data set should start with the following rows:

Copyright Infocrest 2024 46


Exercise #5.2: modify the mapping to get the “OrderDate” year for each and every Customer.

Copyright Infocrest 2024 47


Iconic view of the mapping

1. Create a User Defined Function to extract year from date. In this way the function will be
developed once and used in all your mappings.
a. Create a new User-Defined function component.

b. In the general tab Name the function ‘UDF_ExtractYear’, define the return type as string.

Copyright Infocrest 2024 48


c. In the arguments tab Create a new argument ‘InputDate’ of type date, which will be the input
for the function.

d. In the expression tab Define the expression that will extract the year of the InputDate
argument.

e. Click on Save.
2. Go to the mapping and Add an expression transformation called ‘Exp_Year’ after the joiner and
Create an output port called Year (String (4)) .

Copyright Infocrest 2024 49


3. Open the Editor tab of the expression field and write the following function to extract the year from
the OrderDate:

Copyright Infocrest 2024 50


4. After Connecting the year to the target output port, check the truncate table option to recreate the
table at runtime with the new added field.
a. In the incoming field of the target add Year in the list of included fields.

b. In field mapping tab connect year to target year field.

Copyright Infocrest 2024 51


5. Save and rerun the mapping. Then preview the target table to check the result.

Copyright Infocrest 2024 52


Exercise #6 Lookup and Rank Transformations
Business case

Tradewind wants to maintain a roster of its most productive employees, based on the number of orders
they generate.
Your job is to create a relational table listing all employees, with their names and titles, along with their
ranking.

The logic

1. What data do we need, where is it located?


e. Our main data source is the Orders table. It contains the basic information: OrderID and
EmployeeID
f. Employee information will be looked up from the Employees table. We’ll need: FirstName,
LastName and Title
2. What are the requirements for the target table?
a. We need to connect to a new target ‘T_TopEmployeesByOrders_x’. Its fields are:
EmployeeID, RankIndex, NumOrders, EmployeeFirstName, EmployeeLastName and
EmployeeTitle
3. What is the transformation logic?
a. Fetch OrderID and EmployeeID from the Orders table.
b. Group data by employee, counting the number of orders for each employee
c. Rank the employees by number of orders
d. Retrieve additional employee information (name & title) from the Employees table

The Informatica Cloud solution

1. Create a mapping called ‘m_TopEmployeesByOrders’ with the following components


a. Data source definition, connected to the Orders source table.
b. An Aggregator transformation to group by employee and count the number of orders.
c. A Rank transformation to rank each employee according to the number of orders they
generate. The rank index port will be used in the target as well. Use an arbitrarily high
number of ranks since we want all employees to show up.
d. A Lookup transformation, to fetch employee’s name, last name and title from the
employees table. The lookup join condition is on EmployeeID
e. A target definition to connect to the target table ‘T_TopEmployeesByOrders_x’ where x is the
student number.
2. Save the mapping
3. Run the mapping and validate the target data.

Copyright Infocrest 2024 53


Hints

An iconic view of the mapping

m_TopEmployeesByOrders

The aggregator transformation


Nothing fancy here, just a group by EmployeeID and a COUNT() function:

The rank transformation


We have 9 employees at the moment. Specify a top 100 to rank all employees and leave some room to
grow. Make sure you rank by the number of orders generated by each employee:

Copyright Infocrest 2024 54


The lookup transformation
Create a new lookup transformation and select the Employees table from the ‘MSSQL_Source’
connection in the Lookup Object tab:

In the lookup condition tab you will get an error with respect to name conflicts so press on Resolve field
name Conflicts and prefix the incoming fields from the rank transformation by ‘in_’.

Set the Lookup join condition:

Copyright Infocrest 2024 55


Connect your target to the target table ‘T_TopEmployeesByOrders_x’, where x is the student ID.

That’s it. The only thing left to do is to connect the appropriate rank and lookup output ports to your target

Save and run the mapping.

Copyright Infocrest 2024 56


Validate the Target Data by previewing the target table
The T_TopEmployeesByOrders_x target table should contain 9 records:

Exercise #6.2 Unconnected Lookup


Use an unconnected lookup to add the last name of each employee’s supervisor to the target table. The
supervisor’s employee id is given in the field ‘ReportsTo’ in the employee table. If ‘ReportsTo’ is NULL,
then use ‘Self’ as the supervisor name in your target. In this exercise, the secondary lookup is needed
because the relationship between ‘Reports To’ and ‘EmployeeID’ is recursive.

An iconic view of the mapping

m_TopEmployeesByOrders_Supervisor

Copyright Infocrest 2024 57


Create the unconnected Lookup

After adding the lookup transformation, you have to check the ‘unconnected lookup’ button.

Then add an incoming field named ‘Reports to’.

Specify the lookup object connection and the lookup condition.

Copyright Infocrest 2024 58


Then specify the lookup return field by turning on the radio button of the Last name field. Delete all
unused fields.

Calling the unconnected lookup from the Expression ‘Exp_GetSupervisorName’

Connect SupervisorName to Target field.

Copyright Infocrest 2024 59


Copyright Infocrest 2024 60
Exercise #7 Complex mapping to load SnowFlake Data,
Router Transformation
Business Case

Your company needs to combine account information from Salesforce with order data from SQL server.
The data will be loaded into two different tables depending on the total price of the order. The tables are
used in the order fulfillment system.

The logic

• Create a new mapping that will join the salesforce account object with the orders_prem table.
• We need to lookup the Products file to get the detailed information for each product.
• Then we have to load the data into the Accelerated Shipping target if the total price of the order is
greater than or equal to 1500, otherwise we have to load them into the Normal Shipping target.
• We are going to use Salesforce, flat file, and MSSQL connections.

Informatica Cloud Solution

• Create a mapping called ‘m_LoadShippingDetails’.


• Create a source transformation that connects to the Account object in Salesforce.
• Create a second source transformation that connects to the orders_prem table in the MSSQL
source Database.
• Create a joiner transformation to join the account object and the orders table
• Create a lookup transformation to get the product information from the Product file.
• Create an expression transformation to calculate the total price of the order.
• Create a router transformation with two groups. One group filters rows where the total price of the
order >= 1500. Then this group will connect to the target ‘T_ AcceleratedShipping’. The second
group is the default group which will connect to the target ‘T_NormalShipping’.
• Save and Run the process.

Step by Step Solution

1. An overview of the mapping is as follows:

2. Create a new mapping and name it ‘m_LoadShippingDetails’

Copyright Infocrest 2024 61


3. Create a source transformation to connect to the Account object.
a. The connection type is SFDC.

b. Delete the fields that you will not use in the mapping. Only Keep the fields shown below.

4. Create another Source that connects to the Orders_Prem table.


a. The connection is MSSQL_Source

Copyright Infocrest 2024 62


b. Delete the fields that you will not use in the mapping. Only Keep the fields shown below.

5. Create a joiner transformation that joins the account object and Orders_Prem table based on the
condition Account_External_ID_c = CustomerID

6. Create a lookup on the flat file ‘Products_Prem.csv’ to get the Product Description and
Price_Per_Unit fields.
a. Connect to the Products_Prem.csv file.

b. Specify the lookup condition

7. Create an expression to calculate the total price of the order.


a. Create a new variable field named PPU to convert Price_Per_Unit to integer. (In the
properties of the expression transformation go to the expression tab)

Copyright Infocrest 2024 63


b. Create a new variable field named QTY to convert Quantity to integer.

c. Configure the variable as follows:

d. Create an output port named OrderTotalPrice

c. Configure Order total price as follows: PPU*QTY

Copyright Infocrest 2024 64


8. Create a router transformation with an AcceleratedShipping group with condition
OrderTotalPrice>=1500. All other rows will go to the default group.

9. Create two target transformations one that will create ‘T_AcceleratedShipping’ and the other
‘T_NormalShipping’ at the MSSQL target database. Connect the router to the targets as follows.

10. Configure the ‘T_AcceleratedShipping_x’ target.


a. Specify the connection as MSSQL_Target and choose to create the target object at runtime.

Copyright Infocrest 2024 65


b. In the incoming fields modify the incoming field rule to include only the fields that you want to
insert into the target as follows.

11. Configure ‘T_NormalShipping_x’ target by repeating the same steps provided in step 9.

Copyright Infocrest 2024 66


12. Save and run the mapping. Choose the runtime environment ‘Lab_Env’.

13. Validate the result by previewing the target tables in MSSQL server.

Copyright Infocrest 2024 67


Copyright Infocrest 2024 68
Exercise #8 Local variables, Sorter Transformation
Business case

Errors have popped up in the daily production extracts. Duplicate orders with conflicting primary keys
have found their way into the system. Your job is to filter out duplicate orders from the extract file and
send them to a separate table for cleanup.
This process will leave the first row in a series of duplicate going to the target table while the other
duplicate rows will be sent to the error table.

The logic

1. What data do we need, and where is it located?


The daily extract file from production system.
We’ll extract all data from this file.
2. What are the requirements for the target tables?
Non-duplicate orders go to the data warehouse table T_Orders
Duplicate orders go to a copy of this table, called T_DupeOrders.
3. What is the transformation logic?
Read the daily extract flat file.
Order the data stream by its primary key (order ID)
Detect duplicate primary keys before inserting any rows in the data warehouse target
Direct good orders to the warehouse and duplicate orders to a reject table.

The Informatica Cloud Solution

1. Create a mapping with the following components:


-Data source transformation, connected to ‘Orders.dat’ flat file definition. This is a tab delimited
flat file.
-Create a reusable mapplet that detects duplicate records. The mapplet contains the following:
• An input transformation with the inputs ‘OrderID’, ‘CustomerID’, ‘OrderDate’, and
‘EmployeeID’
• A sorter transformation, to order the data by ‘OrderID’
• An expression transformation, to hold the values of ‘OrderID’ for 2 consecutive rows,
using local variables and variable forward reference techniques.
• An output transformation with the fields ‘OrderID’, ‘CustomerID’, ‘OrderDate’,
‘EmployeeID’, and PrevOrderID.
-A router transformation, to channel duplicate and non-duplicate rows to their respective targets
-Two target transformations one for the good orders ‘T_Orders_x’ and another for the duplicate
orders ‘T_DupeOrders_x’
2. Save and run the mapping. Check the results by previewing the target tables from the mapping.

Copyright Infocrest 2024 69


Hints

An iconic view of the m_RejectDuplicateOrders mapping

Create a new mapplet

In the properties tab of the mapplet name it ‘mplt_DetectDuplicates’.

Copyright Infocrest 2024 70


The input and output transformations are directly created.

Click on the input to add the ‘OrderID’, ‘CustomerID’, ‘OrderDate’, and ‘EmployeeID’ fields.

Copyright Infocrest 2024 71


Add the sorter transformation that sorts the data based on order ID. If there are duplicate
OrderIDs, they will follow each other in sequence.
Set the sorter condition:

And the sorter properties to look like this:

Add the expression transformation to catch duplicate rows before they hit the warehouse, we need to
compare the value of the ‘OrderID’ primary key in the current row with its value in the previous row and
output both values to the router, which will make the dupe/no-dupe decision.
You will implement this logic using local variables. The logic depends on 2 properties:
1. Variables are evaluated from top to bottom, in the order they appear in the port’s list
2. A variable can be set to the value of another variable located later it in the order of evaluation,
this is called a forward reference
To remember a value from a previous row, you need 2 variables, following each other in the port’s list:

Copyright Infocrest 2024 72


1. The first variable is set to the value of the second variable, preserving the value set during the
previous row
2. The second variable is set to the current row value, taken from an input port
You also need a port to output the value of the previous row, the one that is saved in the first variable.
Create the three ports shown below of type integer. Two variable: v_PrevOrderID & v_OrderId. Output
port PrevOrderID. Create the ports in the order they appear.

Local variable forward reference diagram

Select the output transformation to add the output ports OrderID’, ‘CustomerID’, ‘OrderDate’, and
‘PrevOrderID’ as seen below.

Copyright Infocrest 2024 73


Click on the field mapping tab to map input and output field.

Save the mapplet.

Create a new mapping


Configure the Source transformation to connect to the file ‘Orders.dat’ (directory to be assigned by your
instructor).

This is a tab delimited flat file so you have to specify the formatting options as shown below.

Copyright Infocrest 2024 74


Note that both OrderID and EmployeeID fields are integers and that OrderDate is a date. You’ll have to
change the data types in the fields tab of the Source transformation as shown below. Click on options and
check edit metadata to be able to change the data types of the fields.

Copyright Infocrest 2024 75


The dates in the source file are using this format: ‘1996-04-23 00:00:00’. So we have to specify the date
format in the Flat file connection initially (i:e The connection you created to the SourceFiles directory in
the Administrator service view) or else we will get an error while reading the file.

Add the mapplet transformation


Select the mapplet you have created.

Copyright Infocrest 2024 76


Connect the source to the mapplet. In the field mapping tab map the mapplet inputs.

The router transformation


Create a router group for detecting a dupe condition (Create an advanced filter condition with expression
current row’s OrderID equals the previous row’s OrderID) and channel these rows to the dupe table. Use
the default group to channel the regular rows to the regular target.

Copyright Infocrest 2024 77


Create two targets one for non-duplicates orders ‘T_Orders’ and one for duplicate orders ‘T_DupeOrders’.

Connect to both targets (T_Orders and T_DupeOrders) and Map the target fields

Copyright Infocrest 2024 78


Save and run the mapping, then Validate the Target Data by Previewing the target tables.

Copyright Infocrest 2024 79


Exercise #9 Using Sorted Input Optimization
Business case

Marketing would like to make sure the products they advertise in their marketing campaigns all have an
active status and are not discontinued. They want an aggregated semi-annual Total Sales for both active
and discontinued products. To optimize the mapping you can turn on the Aggregator’s sorted input
property.

The logic

1. What data do we need, where is it located?


a. We need 3 data sources
i. The Orders table
ii. The OrderDetails table
iii. The Products table
b. We’ll extract the orderID, orderDate from the Orders table
c. We’ll extract the orderID, productID, unitPrice, quantity and discount from the details
d. We’ll extract the productID, productName and discontinued flag from the Products table.
2. What are the requirements for the target table?
a. ProductID , ProductName, ProductStatus (active, discontinued)
b. Year, FirstSemesterSales, SecondSemesterSales
c. Batch insert date
3. What is the transformation logic?
a. Join the sources on orderID and productid. Sort the input on ProductID and OrderDate.
b. For each row, extract the year and semester from the order date. Semester 1 includes
months from Jan to Jun, semester 2 includes Jul to Dec.
c. For each row, compute the sale amount (UnitPrice * quantity * (1 - discount)) and categorize
the sale depending on the semester the order was placed
d. Grouping by product and year, take the sum of the first and second semester sales
e. For each aggregated row, replace the discontinued flag by a literal value, ‘Active’ or
‘Discontinued’. The original values are ‘T’ (for discontinued) and ‘F’ (for active)
f. Add a batch insert date.

The Informatica Cloud solution

1. Create a mapping that implements the logic above


a. Add a source transformation that joins and sorts the three tables Orders, OrderDetails,
and Products table.
b. Add an expression transformation that specifies the semester and year of each order and
calculates the total sale.
c. Add an aggregator transformation to sum the sales of each semester.

Copyright Infocrest 2024 80


d. Add an expression transformation that specifies the status of each row and adds the
insert date time
e. Add a target transformation to connect to the target ‘T_SemiAnnualSalesByProduct_x’
where x is your student number.

Hints

Iconic view of the mapping

m_SemiAnnualSales

Source transformation

Add the three sources and the relationship will directly be established in between since the keys between
the tables have the same name. To add the Products table you have to highlight Order_Details and then
add the related object.

To be able to use the sorted input property of the aggregator we must sort the data by the group keys
ProductID and OrderDate. In the source properties under query options configure the sort criteria as
follows.

Copyright Infocrest 2024 81


Expression transformation
Extract the year and semester from the order date using a GET_DATE_PART function. Semester 1
covers the months from January to June, Semester 2 from July to December.
Compute an output port for the amount of each order (UnitPrice * Quantity * (1- discount))

Aggregator transformation
Use a SUM with a conditional clause to compute both semester sales amounts.

Check the sorted input property of the aggregator.

Copyright Infocrest 2024 82


Expression transformation
If no order was placed for that product in a given semester, the sum will be NULL. Take care of that use a
default value. Evaluate the status of each order and add a timestamp for each row.

Connect to the target table ‘T_SemiAnnualSalesByProduct’ and map the target fields.

Save and Run the mapping, then Validate the data


The target table will look like this:

Copyright Infocrest 2024 83


Copyright Infocrest 2024 84
Exercise #10 Slowly Changing Dimension Type I (MappingTask)
Business case

TradeWind is testing a new e-commerce system. The system takes orders on the web and generates a
daily extract of order details as a flat file. The system collects information about new customers and
allows existing customers to modify their profile. Your job is to take the daily extract and create a
customer dimension table and update it day by day. New customers get added, and modifications made
online to existing customers are reflected in the dimension.
Note: to simplify the exercise, only a few of the Customers dimension fields will appear in the daily
extract.

The logic

1. What data do we need, where is it located?


a. Daily extract flat file.
b. For the first run, we’ll use ‘DWOrdersExtract.dat’. For the second run, this will be
‘DWOrdersExtractDay2.dat’.
c. We’ll select the available customer fields: company name, address, country and phone
number.
d. We’ll need to lookup against our target to determine if a customer is new or needs an update.
2. What are the requirements for the target table?
a. The target is a new customer dimension table, including an insert date for new rows and an
update date for updated rows.
3. What is the transformation logic?
a. Extract data from flat file
b. Lookup the data in the file against the Customers dimension
c. We may have several rows in the extract file that belong to the same new customer.
However, we need new customers to be inserted just once in the dimension. We will
compromise by aggregating the data to select only the latest customer information.
d. Determine if the customer is new (no customer PK matching that company ID) or in need of
an update (fields in the extract file don’t match existing fields in the Customers dimension) or
can be left alone.
e. If the customer is new, generate a new customer PK and insert it in the dimension
f. If some of the customer information has changed, update the existing row with the new data.
g. Add the appropriate insert or update dates.

The Informatica Cloud solution

1. Create a mapping with the following components


a. Data source transformation connected to ‘DWOrdersExtract.dat’ a tab delimited file.

Copyright Infocrest 2024 85


b. An Aggregator to group rows by CompanyID. Be sure to group by all the attributes we want to
compare for change. This will give us a distinct data flow (of the last record from the file)
before we hit the Lookup.
c. A lookup transformation that queries your T_Customers_DIM_x table, looking for a given
match based on the company id.
d. An expression, to flag rows as insert (new customers) or update (changed customer) by
comparing attributes from the source to attributes from the lookup
e. A router transformation, to examine the flag computed in the expression and channel
customer data depending on the results:
i. New customer
ii. Updated customer
iii. Unchanged customers
f. A sequence generator, to create a customer PK for each new customer.
g. Two target transformations connected to the T_Customers_Dim_x target, one for updates,
the other for inserts.
2. Save the mapping
3. Create a mapping task that includes the mapping, so the current value of the sequence generator
will update for each run.
4. Run the mapping with the ‘DWOrdersExtract.dat’ file as a source
5. Run it a second time but after modifying the source in the mapping to read from
‘DWOrdersExtractDay2.dat’ file.

Copyright Infocrest 2024 86


Hints

Iconic view of the slowing changing dimension type 1 mapping:

m_UpdateCustDimType1

Source Transformation
Define the formatting options of the source as follows.
Delimiter: Tab
Field Labels: Auto-generate

Copyright Infocrest 2024 87


Aggregator transformation

Include only the needed fields in the mapping and rename them as follows. This is done by configuring
the include rule in the incoming field tab of the aggregator.

Group by the fields that we will need to compare.

Copyright Infocrest 2024 88


Lookup transformation
Look for a match on company id. Return custPK, address, phone and country if successful. Add prefix
‘in_’ to the incoming fields to avoid name conflicts in the lookup transformation.

Copyright Infocrest 2024 89


Compare Attributes expression
Create an output port named Flag of type integer. If the lookup returns a null value, the customer in the
source file is new. Otherwise, compare all attributes from both the lookup and the source (i:e values from
source have prefix in_). If some are different, you will flag the row as an update.

Copyright Infocrest 2024 90


Add an output port ‘Now’ to get the system date.

Add a Sequence generator

Add a router with two groups one for new customers and another for updated customers

Copyright Infocrest 2024 91


Add a target transformation to connect to T_Customers_DIM_x with Insert operation

Note that NEXTVAL is the field coming from the sequence generator.

Add a target transformation to connect to T_Customers_DIM_x with update Operation

Copyright Infocrest 2024 92


Create a Mapping task that will run the mapping m_UpdateCustDimType1
Click on the three dots on the right of the top bar and click on New Mapping Task.

Copyright Infocrest 2024 93


Click on next then finish.

Run the mapping task

Validate the Target Data


During the first run, with ‘DWOrdersExtract.dat’, you should have inserted 86 rows. No updates since your
customer dimension table was empty.
From my jobs tab click on the mapping instance to go to the session logs, you can see that all records are
routed as insert.

Copyright Infocrest 2024 94


Preview target table to check the results.

Then go to the mapping and change the source object so it can read from ‘DWOrdersExtractDay2.dat’.
During the second run, with ‘DWOrdersExtractDay2.dat’, your mapping should have inserted the following

Copyright Infocrest 2024 95


records: ‘France Restauration’, ‘La corne d’abondance’, ‘Specialites du Monde’ and updated the following
record: ‘Bon appetit’.

Copyright Infocrest 2024 96


Exercise #11 Simple Taskflow and Error Handling
Business Case

Your job is to put together a simple Taskflow containing 2 tasks linked in series and implement error
handling.
You will have to make sure that:
a. Tasks fail at the first mapping error
b. Tasks do not run when the previous Task fails
c. Taskflow fails if anything goes wrong
The ultimate goal is to run the taskflow to completion, without any errors.
You can pick any two valid tasks in your folder.

The Informatica Cloud solution

1. In your student folder create a new Linear Taskflow


2. Call it ‘tkf_ErrorHandling_x’ where x is your student number
3. The TaskFlow will contain
a. Two tasks to provide a run-time environment for 2 mapping tasks
b. Tasks will be linked in series
4. Make sure to check the Stop on error button to prevent a later task from running if a previous task
fails
5. Save and Run the taskflow
6. Monitor your job in My jobs window
7. Make sure the tasks and taskflow are successful. Don’t forget to use the log files to look for errors

Copyright Infocrest 2024 97


Hints

A view of the tkf_ErrorHandling taskflow

Creating the taskflow


1. From the data integration service, go to your folder click on new > taskflows > linear taskflow.

Copyright Infocrest 2024 98


2. We created earlier a slowly changing dimension task you can add it. Create another mapping
task from the valid mappings to use it in the taskflow.
The sequence number show the order in which the tasks run.

3. Then save and run the taskflow.


4. View the job in My jobs window.

5. Then click on the 2 tasks link to see the run-time details of each task.

Copyright Infocrest 2024 99


Copyright Infocrest 2024 100
Exercise #12 Complex Taskflow
Business case

Your data warehouse daily load dimension tables based on a specific logic.

The logic

1. Sources: 4 dimension loading mappings


2. Dependencies
a. The mappings read from source files that are transferred every day to a local directory.
b. Dimension table 1, 2 and 3 can be loaded at the same time
c. Dimension table 4 must be loaded only if dimension table 3 succeeds or else we have to run
a cleanup script after 5 mins.
d. Send an email if the command script fails to execute.
3. Error behavior
a. A failure in any of these tasks must stop the entire process except for dimension table 3 and
the command script.

The Informatica Cloud solution

1. Create a taskflow ‘tkflw_LoadDimensions’


2. Add any valid mapping task to represent each dimension table in the taskflow.
3. Create a file listener component that watches the arrived files at the local directory.
4. Add a file watcher task to the taskflow that uses the file listener component.
5. Add a commad task that executes the cleanup script.
6. Add a notification task that sends an email of the cleanup command status.

Hints

This exercise is about taskflow logic, not actually loading data. You will be using any valid mapping task
each time you need to add a data task.

Dimension tables loading taskflow

Copyright Infocrest 2024 101


Create a file listener component to monitor the arrival of the source files

Copyright Infocrest 2024 102


You can run the file listener on a schedule

Create the taskflow

Copyright Infocrest 2024 103


Then add the file watcher task and assign to it the created file listener component. The mappings to load
dimension tables will be executed once the file arrives to the file listener directory.

Copyright Infocrest 2024 104


Then add a parallel path object to add three mapping tasks that will load Dimension table 1, 2 and 3. Click
on the parallel path object and in the parallel paths properties tab add new paths.

Copyright Infocrest 2024 105


Add a Data Task for each path and assign the task you want to run in the data task tab.

In the path that loads dimension table 3 add a decision object as follows.

Copyright Infocrest 2024 106


The properties above depend on the task status of LoadDim3. If LoadDim3 succeeds we will load
dimension table 4 or else, we will wait for 5 minutes before executing the cleanup command.

Wait Task Configuration:

Define the script path in the Command Task input field tab:

Copyright Infocrest 2024 107


Use any simple command similar to the one in the screenshot below.

If LoadDim3 task fails, the taskflow must not stop since further analysis is done. So, change the ‘on error’
property in the Error handling properties tab of the data task to ‘Ignore’.

Add a notification task after the command to send an email for the command execution status.

Notification task configuration

Copyright Infocrest 2024 108


If CMD_Cleanup task fails, the taskflow must not stop since an email must be sent for the cleanup status.
So, change the ‘on error’ property in the Error handling properties tab of the command task to ‘Ignore’.

Copyright Infocrest 2024 109


Exercise #13 Mapping Parameters
Business case

You need to pass Order records to a staging table in increments. Each incremental load will truncate the
stating table and load 500 new orders. In addition, you will keep a history table with statistics for each
incremental load. You will use mapping in-out parameters to handle the incremental load automatically.

The logic

1. What data do we need, and where is it located?


We will use both Orders and Order_Details tables.
We’ll extract the OrderID, CustomerID and OrderDate fields from the Orders table
We’ll extract the OrderID, UnitPrice, Discount and Quantity fields from the Order_Details table.
2. What are the requirements for the target tables?
The staging table has the following fields: OrderID, CustomerID, OrderDate and OrderAmount
[(UnitPrice – Discount) * Quantity].
The load history table will have the following columns: FirstOrderID, LastOrderID, NumOrders,
NumDetailLines, TotalAmount, FirstOrderDate, LastOrderDate and a timestamp field.
There will be only one row written to the load history table for each incremental load.
3. What is the transformation logic?
Declare starting values and increment values as mapping in-out parameters.
Join the sources on OrderID, with a custom filter based on your mapping in-out parameters. We
want to restrict the load to just a slice of the data, containing at most 500 orders.
Aggregate the data flow by OrderID to compute the total amount of each order. These rows will
go to the staging table
Aggregate load statistics for all rows and send the resulting row to the load history table, with a
timestamp

The Informatica Cloud Solution

1. Declaring mapping in-out parameters


You must create a new mapping first!
Then open the Parameters Panel,
Create a parameter called $$StartOrderID as an integer with an aggregation method of ‘Max’ and
set the default value to 10000. This will be the starting point for your first load.
Create a parameter called $$OrdersToLoad as an integer with a default value of 500. This is the
increment value, the maximum number of orders that will be written to the staging table
during each load.
2. Using the in-out parameters
In your Source Qualifier, add a custom source filter to pull orders with an OrderID between
$$StartOrderID (not inclusive) and $$StartOrderID + $$OrdersToLoad (inclusive).
You will also need an Expression transformation to set the value of $$StartOrderID to the current
value of the OrderID field, using the SETMAXVARIABLE function. When your mapping task

Copyright Infocrest 2024 110


finishes successfully, the cloud integration service saves this value to the repository and uses
it to start the next load.
3. Improving performance
Make sure to use the sorted input property for the first aggregation. Make sure to sort your data
flow by the correct key first.
4. Creating a mapping task so the ‘$$StartOrderID’ parameter value will update for each run.

Hints

Iconic view of the mapping

m_IncrementalLoad

Target definitions
The following target definitions shows the fields that must be loaded when you create a new target at
runtime.
T_OrdersStaging_x, where x is your student number.

The load history table ‘T_OrdersStats_x’ looks like this:

Copyright Infocrest 2024 111


Open the Parameters Panel

Add two in-out parameters as follows:

Copyright Infocrest 2024 112


Connect to Order and Order Details tables

Delete unwanted fields. Keep fields shown below.

Copyright Infocrest 2024 113


Source filter

Copyright Infocrest 2024 114


Aggregator to compute the total amount of each order

Expression to set the variable value


Create a variable field named v_SetOrderID and configure the expression to set the
$$StartOrderID variable.

Copyright Infocrest 2024 115


Aggregator to calculate load statistics

Target tables fields

Don’t forget to check the truncate target table option of T_OrdersStaging_x. Create a new runtime target
object for both targets and just include the incoming fields that you need to load into the target.

Copyright Infocrest 2024 116


Copyright Infocrest 2024 117
Creating a mapping task

Run the mapping task when you finish

Validating the data for the first run


Staging table:

Copyright Infocrest 2024 118


Load history table:

Verifying the post-session value of the $$StartOrderId in-out parameter

The value could be checked in the mapping task. Click on edit go to the in-out parameters tab and you
will see the updated value of $$StartOrderID.

If you select the StartOrderID in-out parameter and hit the ‘Reset In-out parameter’ button , the in-out
parameter is removed from the repository. The next time you run, the default value set in your mapping
will be used, unless you are using a parameter file.

Copyright Infocrest 2024 119


Validating data for the other runs
3 runs will exhaust the data in the Orders table. At the end of the 3rd run, your load history table should
look like this:

Copyright Infocrest 2024 120

You might also like