Module 2 Edcmicdiforcdw 21588742042605

April 2020
Cloud Data Warehouse and

Data Lake Modernization
`
Today’s Agenda
1 Welcome
2 Accelerate Cloud Modernization and Migration with an Enterprise Data Catalog
3 Efficiently Ingest Streaming Data, Files, and Databases to Cloud Data Warehouses & Data Lakes

4 Integrate & Cleanse Data into Cloud Data Warehouse & Data Lakes
5 Close
2 © Informatica. Proprietary and Confidential.

Accelerate Cloud Data
Warehouse Modernization &
Migration with Enterprise
Data Catalog
`
Architecture for Cloud Data Warehouse & Data Lakes
Streaming 2
Stream Real-time
6
Processing
Stream Storage Analytics Business
User
IoT Machine Apps
Data
5
Cloud Data Lake Enterprise
Data Provisioning
Data Integration
Log files Social Mobile Analytics Data
Analyst
3 Data Integration & Quality Cloud Data
Data Ingestion
On-Premises Warehouse Line of Business /

Self-Service
Line of
Mainframe Application Databases Analytics Business
Servers Landing Data Enterprise
Zone Enrichment Zone 4
Data Provisioning
Documents Data
Warehouse
Data
Engineer
SaaS Cloud Storage Data Science/AI
Spark Processing Data

ERP DRM Scientist
Data Catalog &

1
Data Governance Discovery Lineage Glossary

Streaming 2
Stream Real-time
6
Processing
User
IoT Machine Apps
Data
5
Data Provisioning
Data Integration
Analyst
Data Ingestion

Self-Service
Line of
Data Provisioning
Documents Data
Warehouse
Data
Engineer

ERP DRM Scientist
Data Catalog &

1

Growing Complexity of Data Landscape
Explosion in New New Data Data in Machine

Data Volume Users Types the Cloud Learning/AI
(mobile, social, IoT)
20.6 zettabytes 500 million 20 billion Over 94% of data 1 billion workers
per year business data connected center traffic will be assisted by
in global data users and growing devices will come from machine learning
center traffic the Cloud or AI
Data Cataloging is the First Step
Data Governance Advanced Analytics Cloud Modernization Customer Experience
Worldwide Online Large Regional Health Care

Automotive Manufacturer
Payments Shipping Company Company
Discovery of critical enterprise Flipped the 80-20% rule for Enterprise-scale metadata Opening up data visibility is
data provided foundation for finding data to analyzing effort understanding allows them to fueling development of new
data governance program for data analysts simplify cloud migration services and improving quality
projects of patient care

First, What is a Catalog?

Enterprise Data Catalog
• Find the data you need with simple, powerful

semantic search
• Understand your enterprise data with a holistic
view
• AI-powered automatic discovery, classification
and business context
• Comprehensive metadata connectivity for all
enterprise data
• Big Data scale and flexible deployment options
• Open framework for custom extensions and
integrations

The Catalog of Catalogs
Self-Service Data Master Data Cloud Metadata Data
Data Quality
Analytics Governance Management Modernization Intelligence Integration
Open APIs, Full Access
• • • •
•
Discovery
Profiling •
Semantic Search
Domain Discovery
Enterprise Data Catalog •
Relationships
PK-FK Discovery •
Reviews/Ratings
Questions/Answers
• Lineage • Similarity Clustering • Business Context • Data Certifications
Metadata System of Record
• Impact Analysis • Business Term Association • Custom Annotations • Change Notifications
for The Enterprise
Knowledge Graph + AI/ML
Breadth of Active Metadata
AWS Glue Azure Data Catalog Google Data Catalog
BigQuery Cloud
HANA ADLS Storage
On-prem File BI On-prem/ ETL
Databases Systems Tools SaaS Apps
Enterprise | Unified Metadata | Intelligence
Relationship Discovery Natural Language Translation
Data Similarity Schema Inference
Data Tagging Anomaly Detection
Entity Discovery Recommendations

A Catalog-First Strategy for Cloud
What data assets should be migrated? Are there duplicate data sets?
Which data consumers

How is sensitive data identified? are affected?
Enterprise
Data Catalog
Is there a need to modify data

before moving to cloud? What is the quality of data?

What Data Assets Should Be Migrated?
Goal: Identify data assets and reports to

be migrated
• Data Lineage describes the flow of data

• Impact Analysis identifies dependencies of
the data asset which may be affected when
migrated to cloud

How is Sensitive Data Identified?
Goal: Identify and understand the

meaning of source data considered for
cloud migration
• Data domain discovery identifies the

functional meaning of data in data sources
based on semantics of data.
• Ex: SSN belongs to PII data domain

What is the Quality of the Data?
Goal: Assess data quality for content and

structure
• Column profiles determine characteristics of

columns in a data source, (i.e. value frequency,
percentages, and patterns).
• Understand quality levels of key data assets from
within EDC
• View data quality rules associated with data
assets
• View the quality Score Cards & Metric Groups

Which User Groups Need to be Targeted First?
Goal: Understand the relationship between

the asset tagged for cloud migration
• Discover and understand related data sets,

technical, business, semantic and usage based
meaningful and holistic view of data
relationships.

Is There a Need to Modify before Migrating?
Goal: Understand schema structures

identified for cloud migration
• Drill-down lineage view expands any lineage

path to show columns and lineage diagram
metrics
• Post migration lineage diagrams can be used to
validate that structure changes have been
performed correctly

What Duplicate or Related Data Be Rationalized?
Goal: Identify data sources that contain

duplicate or similar data
• Leverage column similarity that identifies

Like photo identification but for Columns duplicate and redundant data sets across
data sources.
• Column similarity uses unsupervised
clustering to identify the similar columns

How Can a Catalog Help Accelerate Development?
Goal: Help developer accelerate data

integration by leveraging a catalog
• Search, view and select assets for data

integration directly in Cloud Data Integration
• Automatically populate connection values from
searched data objects
• Easy-to use wizards to guide non-technical
users with quick integrations

Demo
Next Steps
Download this eBook to learn more about how to drive your

business forward with an AI-Powered Catalog of Catalogs for the
enterprise
Leverage this workbook, “

7 Best Practices to Drive Data Catalog Adoption”, to help advance
data catalog adoption in your organization

Efficiently Ingest Streaming Data,
Files, and Databases to Cloud Data
Warehouses & Data Lakes
Streaming 2
Stream Real-time
6
Processing
User
IoT Machine Apps
Data
5
Data Provisioning
Data Integration
Analyst
Data Ingestion

Self-Service
Line of
Data Provisioning
Documents Data
Warehouse
Data
Engineer

ERP DRM Scientist
Data Catalog &

1

Streaming 2
Stream Real-time
6
Processing
User
IoT Machine Apps
Data
5
Data Provisioning
Data Integration
Analyst
Data Ingestion

Self-Service
Line of
Data Provisioning
Documents Data
Warehouse
Data
Engineer

ERP DRM Scientist
Data Catalog &

1

Informatica Intelligent Cloud Services
ANY TECHNOLOGY ANY LATENCY ANY DATA ANY USER MULTI-CLOUD
DATA DATA METADATA

INTEGRATION QUALITY MANAGEMENT
CLOUD DATA INTEGRATION CLOUD DATA QUALITY ENTERPRISE DATA CATALOG
MONITOR AND MANAGE

DATA ENGINE
CONNECTIVITY
INFORMATICA INTELLIGENT CLOUD SERVICES (CLOUD NATIVE )
STRUCTURED
STREAMING

Cloud Mass Ingestion – Use Case Patterns
Data Warehouse Accelerate Kafka Journey for

Cloud Data Lake Ingestion
Modernization/Migration Real-time Analytics
• Mass ingestion of files onto • Mass & CDC Ingest on- • Logs/Clickstream ingestion
cloud & on-prem data lakes prem database content onto • IoT data ingestion
• Streaming & IoT data Cloud DWH (Snowflake) • CDC Ingestion onto Kafka
ingestion onto data lake • Mass & CDC ingest
• Mass Ingest on-prem mainframe content onto
database content onto Cloud DWH
cloud/om-prem data lake

Informatica Cloud Mass Ingestion
Files and Databases Cloud Data Lakes
Mass Ingestion
AI/ML &
Cloud Storage Data Science
File
Informatica Mass Ingestion
Data Integration and Quality

Files Databases
Advanced
Analytics
Google Cloud
Informatica
Storage
Streaming
Cloud Data
ADLS Gen2
Ingestion
Database Reports &
Warehouse Dashboards
IoT Machine Logs
Data
Messaging
Self-Service
Analytics
Messaging
Streaming
Ingestion
Event Store Streaming

Analytics
Kafka Amazon Azure
Kinesis EventHub
27
Informatica Cloud Mass
Ingestion Service Overview
• Cloud native service for all ingestion uses cases
• File
• Database (initial and incremental - CDC)
• Streaming & IoT
• Unified user experience for ingestion

• Common wizard experience for designing
• Deployment & scheduling
• Real time monitoring experience
• Connectivity to various sources and targets

• Example Sources: File, Database, Streaming & IoT
• Example Targets: Cloud Data Warehouses and Data lakes,
Messaging Hub

Streaming Ingestion
Disruption With Streaming Analytics
Real-time offer Predictive Maintenance Clinical Research Real-time Fraud

management and Smart Factory Optimization Detection and Alerting
Combine web searches Identify stress signals Collect and process bedside Run real time fraud
& camera feeds to coming from devices and monitor data for clinical detection machine-
identify the customer act on them before its researchers to more effectively learning model on the
and roll out real-time too late understand and transaction data
offers detect disease

Common Streaming Ingestion Use Cases
Cloud Data Lake Streaming Analytics and ML

• Real-time Data Ingestion and Data Integration • Real time dashboarding
• Ingestion of streaming data into messaging hub or • Operationalizing actions
onto cloud data lakes
• Real time analytics
• Real time integration of streaming data
• Operationalize Actions on IoT devices
• ML models and Rules processing

Streaming Ingestion - Overview
Data Lake
AI/ML &
Data Science
Informatica Mass Ingestion

Sensor/ Machine
IoT Data
Streaming Ingestion
Advanced
Analytics
Google Cloud
Weblogs Social Storage
ADLS Gen2 Reports &

Messaging Dashboards
Self-Service
Analytics
Amazon Azure Messaging
Kinesis EventHub
Streaming
Analytics
Amazon Azure
Kinesis EventHub
Ingest streaming data: Logs, clickstream, social media, Kafka Real-time monitoring of ingestion jobs with lifecycle management
Kinesis, S3, ADLS, Firehose, etc. and alerting in case of issues
Apply simple transformation rules to ensure the data is ready for Orchestrate streaming data ingestion in hybrid/cloud as managed
analytics and secure service

Streaming Ingestion
Key Capabilities
Authoring Wizard Intent Driven Ingestion +

•Creating data flows
address Schema drift
•Ease of deployment
Connectivity
Edge Transformation
• Streaming sources
• Filter, Segregator, Combiner
• Messaging and data lakes
Real Time Monitoring

• Visualization driven
• Lifecycle management
• Alerting in case of failures

Benefits
1 Single ingestion solution 2 Wizard driven 3 Enable business to ingest

for all patterns experience for ingestion streaming data for their
1 2 3 usage
Save time and money Increase business agility Faster decision making
4 Edge transformations 5 Connectivity to 6 Real time monitoring

for cleansing data streaming sources & and alerting
targets
Increased trust in data assets No need to hand code Faster troubleshooting

Demo
Streaming Ingestion
Database Ingestion
Targets
Amazon S3
Apache Kafka
Microsoft ADLS
Microsoft SQL-DW
IICS Snowflake
Communication
Secured
Cloud ata F l
ow
red D
Secu
On Prem
Database Databas
Secure
e
Source Agent Secured Data Flow
Ingestio
n
Database Ingestion
• Provides Database ingestion capabilities as part of IICS • Real-time monitoring of ingestion jobs with lifecycle
Mass Ingestion service management and alerting
• Ingest relational database data from Oracle, SQL-Server & • Orchestrate database data ingestion in hybrid/cloud as
MySQL. Also supporting Schema Drift on CDC supported managed and secure service
Databases
39
Database Ingestion
Benefits of new service
1. Intuitive for use with broad connectivity stack
2. Easy to build mapping less specifications (wizard driven)
3. Minimal impact to any production systems,
4. Scale to 1000’s quickly and easily (point & acquire)
5. Runs autonomously with minimal intervention (24*7*365)
6. Auditability; needs to detect/trace changes, and ensure data has been applied where it should be
7. Aiming to provide strong monitoring capabilities with consistent look and feel

Demo
Database Ingestion
File Ingestion
Common Use Cases
Azure DW, Data Lake
• Lake Ingestion FTP/sFTP/FTPs Server
- Ingest data that arrive as files to Data lakes and

repositories
Redshift
- No data manipulation is needed. Focus on File Ingestion
ingestion. Connectivity
S3
- Transfer files from remote FTP/SFTP/FTPS Tracking & Monitoring
Connectivity
servers
- Any file size, any file types
Google storage
Azure Blob

Mass Ingestion Files
Overview
Cloud Google storage
Data
MI Metadata Integration
Cloud Redshift S3
1 MI Task
4 Update Job
log
Azure DW, Blob, Data Lake
Secure
2 Agent
3
File Mass
Advanced FTP/SFTP/FTPS Ingestion service
connector
Provides file transfer capabilities for exchanging files between on-

Job and file level tracking and monitoring
premises and cloud repositories
Orchestrate File transfer and ingestion in hybrid/cloud as

Transfer any file type with a high performance and scalability
managed and secure service
44
Demo
File Ingestion
Summary
CLICK TO INSERT IMAGE CLICK TO INSERT IMAGE
Cloud native ingestion Connectivity Wizard Driven Design Real-time Monitoring

• IICS service for mass • On-prem Database & CDC • Simple easy to use • Pictorial view of the
ingestion • On-prem & cloud files wizard ingestion job
• Orchestration for • IoT & Streaming • Edge transformations • Real time flow
• Intent driven ingestion visualization
ingestion from variety • Cloud data lakes,
of patterns Datawarehouse and • Lifecycle management
messaging hub

FREE TRIAL – Cloud Mass Ingestion Service!
• Register TODAY here: https://www.informatica.com/trials.html

Integrate & Cleanse Data into Cloud Data
Warehouse & Data Lakes
Streaming 2
Stream Real-time
6
Processing
User
IoT Machine Apps
Data
5
Data Provisioning
Data Integration
Analyst
Data Ingestion

Self-Service
Line of
Data Provisioning
Documents Data
Warehouse
Data
Engineer

ERP DRM Scientist
Data Catalog &

1

Streaming 2
Stream Real-time
6
Processing
User
IoT Machine Apps
Data
5
Data Provisioning
Data Integration
Analyst
Data Ingestion

Self-Service
Line of
Data Provisioning
Documents Data
Warehouse
Data
Engineer

ERP DRM Scientist
Data Catalog &

1

ANY TECHNOLOGY ANY LATENCY ANY DATA ANY USER MULTI-CLOUD
DATA DATA METADATA

INTEGRATION QUALITY MANAGEMENT
CLOUD DATA INTEGRATION CLOUD DATA QUALITY ENTERPRISE DATA CATALOG
MONITOR AND MANAGE

DATA ENGINE
CONNECTIVITY
INFORMATICA INTELLIGENT CLOUD SERVICES (CLOUD NATIVE )
STRUCTURED
STREAMING

Cloud Data
Integration
Cloud Data Integration
Key Data Integration Capabilities to support Cloud Lakehouse Data Management
Data Ingestion Data Integration Data Integration Processing
• Easy to use 4-step data ingestion wizard • Broad connectivity to on-premise & cloud • Pushdown integration logic
end-points to cloud data warehouse end points
• Mass ingest hundreds automatically
of data sources from files, databases, • Out-of-the-box transformations &
streams into Cloud Data Warehouse & Data templates for common data warehousing • Intelligently auto-scale, provision, and
Lakes patterns and advanced DI, DQ auto-tune data pipelines using Spark-
based serverless cloud clusters with built-
• Replicate on-premises data to Cloud Data • Built-in transformational capabilities to in HA and Recovery.
Warehouse handle multiple data formats (XML,
JSON, Avro, Parquet)
• Automatically detect schema drift.
• Integrated experience with Enterprise Data
Catalog to search and discover data objects

Connectivity: Breadth, Depth & Intelligent
Highly-performant connectivity to any

end-point, on-premises or cloud, for any
data type: structured, semi-structured, or
streaming data
Support advanced connectivity capabilities
i.e. pushdown optimization, partitioning,
change data capture
Pattern aware: Leverage ODBC, JDBC
and proprietary interfaces for databases;
REST, SOAP & proprietary interfaces for
applications

Capabilities and features
• Out-of-the-box purpose-built
transformations
• Built-in transformational capabilities to
handle multiple data formats
• Rich-set of functions for data
conversion/manipulation
• No-Hand coding required for data
manipulation or transformation

Complex File Processing with Intelligent
Structure Discovery
• CLAIRE based AI-driven technology to
automate complex file processing
• AI / machine-learning to discover and parse
complex files with easy to use interface
• High performance loading with advanced
partitioning and push-down optimizations
• Overcome Data fluctuation, Data Drift and
“noisy” input

Ecosystem Pushdown Optimization
• Support for ETL and ELT modes to

cloud data warehouse
• Dynamically create and execute cloud
data warehouse specific
transformation language
• Decouples data transformation logic
from physical execution plan

Data Integration with Elasticity
• Process data-integration workloads in a
fully managed scalable cloud-
environment with ephemeral Kubernetes
cluster and serverless Spark engine
• Automatically manage and intelligently
auto-scale and provision Spark-based
serverless clusters.
• Built-in high availability and
automatic recovery capabilities to
eliminate a single point of failure of data
workloads.

Orchestration Capabilities
Orchestration tool purpose

built for data integration tasks
•Supports parallel + conditional
execution of data integration tasks
•Loop and jump within the flow
•Embed ingest task into task flows
•Send custom email notifications
•Allow command-level OS processing
•File-watch capabilities

Orchestration Capabilities
• Orchestration tool purpose built for data integration tasks

• Supports parallel + conditional execution of data integration • Send custom email notifications
tasks • Allow command-level OS processing
• Loop and jump within flow • File-watch capabilities
• Embed ingest task into task flows

Template and Wizard-based Development
Pre-built templates for common

integration patterns
Integration patterns that support
citizen-integrator, business user
and developers to quickly onboard
data via a wizard-driven approach
Integration flows that Data
Engineers can build to meet any
complexity using drag and drop
interface

Developer Productivity & Reusability
Standardize integration tasks with parameterization Reusable mapplets and user-defined functions
• Automated parameter file generation with default values.
• Support for connection and object parameter overrides for cloud
databases + data warehouses
• Support for partial parameterization of SQL overrides, Pre-Post
SQL queries in mapping

Demo
Integrated Data Discovery
Integrated experience with

Enterprise Data Catalog to search
and discover data objects
Discovered objects can be used as
part of mappings
Connections automatically import

metadata from catalog

Intelligent Operational Insights
• Increase operational efficiency of Cloud DI

deployments
- Predictive Analytics, Actionable
Recommendations, Integrated Monitoring built
on CLAIRE
• Analyze impact integration jobs and

optimize resources + schedules with heat
maps.
• View all upcoming scheduled jobs to detect
any overloads on runtime environments.

Enable self-service enterprise class
data quality
• Build rules to ensure data is always fit

for purpose
• Unit test to verify data accuracy and
expected outcomes at design time
• Integrate data quality logic seamlessly
into your integration pipelines

Cloud Data Quality
Enable self-service enterprise class data quality
Build rules to ensure data is always fit

for purpose
Unit test to verify data
accuracy and expected outcomes at
design time
Integrate data quality logic seamlessly
into your integration pipelines

Demo
Reuse Your PowerCenter Skills… and Upskill
Skills PowerCenter Informatica Intelligent Cloud
Mapping Design
Connectivity
Transformations
Debugging
Transformation Functions & Language
Parameters & Variables
Workflows
Scheduling
Monitoring
Cloud Data Quality Upskill
Cloud Mass Ingestion Upskill
Application Integration Upskill
Cloud Integration Hub Upskill

FREE TRIAL – Cloud Data Integration!
• Register TODAY here: https://www.informatica.com/trials.html

Call to Action: Cloud Data Warehouse Modernization
https://www.informatica.com/solutions/move-to-the-cloud
Learn & Prepare Deep Dive Get Started

Whitepapers & Workbooks Reference Architecture Guides Free Trials
Free 30 day trial
© Informatica. Proprietary and Confidential.

Partner Resources - PARC
For assistance,
contact the PARC
Helpdesk
partners@informati
ca.com
IU Product Training/Certifi Implementation Workshops

cations Partner Learning Series
Sales Training
Log into PARC
first, and then click
these resources.
Competitive Technical Webinars Informatica Network Success Portal

Don’t Forget the Quiz!
Thank You

Module 2 Edcmicdiforcdw 21588742042605

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Module 2 Edcmicdiforcdw 21588742042605

Uploaded by

Copyright:

Available Formats

April 2020

Cloud Data Warehouse and

2 Accelerate Cloud Modernization and Migration with an Enterprise Data Catalog

2 © Informatica. Proprietary and Confidential.

On-Premises Warehouse Line of Business /

Spark Processing Data

Data Catalog &

4 © Informatica. Proprietary and Confidential.

On-Premises Warehouse Line of Business /

Spark Processing Data

Data Catalog &

5 © Informatica. Proprietary and Confidential.

Explosion in New New Data Data in Machine

Data Governance Advanced Analytics Cloud Modernization Customer Experience

Worldwide Online Large Regional Health Care

7 © Informatica. Proprietary and Confidential.

8 © Informatica. Proprietary and Confidential.

• Find the data you need with simple, powerful

9 © Informatica. Proprietary and Confidential.

Open APIs, Full Access

Breadth of Active Metadata

AWS Glue Azure Data Catalog Google Data Catalog

Relationship Discovery Natural Language Translation

Data Similarity Schema Inference

Data Tagging Anomaly Detection

Entity Discovery Recommendations

11 © Informatica. Proprietary and Confidential.

Which data consumers

Is there a need to modify data

12 © Informatica. Proprietary and Confidential.

Goal: Identify data assets and reports to

• Data Lineage describes the flow of data

13 © Informatica. Proprietary and Confidential.

Goal: Identify and understand the

• Data domain discovery identifies the

14 © Informatica. Proprietary and Confidential.

Goal: Assess data quality for content and

• Column profiles determine characteristics of

15 © Informatica. Proprietary and Confidential.

Goal: Understand the relationship between

• Discover and understand related data sets,

16 © Informatica. Proprietary and Confidential.

Goal: Understand schema structures

• Drill-down lineage view expands any lineage

17 © Informatica. Proprietary and Confidential.

Goal: Identify data sources that contain

• Leverage column similarity that identifies

18 © Informatica. Proprietary and Confidential.

Goal: Help developer accelerate data

• Search, view and select assets for data

19 © Informatica. Proprietary and Confidential.

Download this eBook to learn more about how to drive your

Leverage this workbook, “

21 © Informatica. Proprietary and Confidential.

On-Premises Warehouse Line of Business /

Spark Processing Data

Data Catalog &

23 © Informatica. Proprietary and Confidential.

On-Premises Warehouse Line of Business /

Spark Processing Data

Data Catalog &

24 © Informatica. Proprietary and Confidential.

DATA DATA METADATA

5. Runs autonomously with minimal intervention (247365)