Chapter 4
DATA WAREHOUSING AND MANAGEMENT
ETL Process in Data Warehouse
• ETL stands for Extract, Transform, Load
and it is a process used in data
warehousing to extract data from
various sources, transform it into a
format suitable for loading into a data
warehouse, and then load it into the
warehouse.
• The ETL process is an iterative process
that is repeated as new data is added to
the warehouse. The process is
important because it ensures that the
data in the data warehouse is accurate,
complete, and up-to-date. It also helps
to ensure that the data is in the format
required for data mining and reporting.
Extract
• The first stage in the ETL process is to
extract data from various sources such
as transactional systems, spreadsheets,
and flat files. This step involves reading
data from the source systems and
storing it in a staging area.
• Before data can be moved to a new
destination, it must first be extracted
from its source — such as a data
warehouse or data lake. In this first step
of the ETL process, structured and
unstructured data is imported and
consolidated into a single repository.
Volumes of data can be extracted from
a wide range of data sources, including:
• Existing databases and legacy systems
• Cloud, hybrid, and on-premises
environments
• Sales and marketing applications
• Mobile devices and apps
• CRM systems
• Data storage platforms
• Data warehouses
• Analytics tools
Transform
• In this stage, the extracted data is
transformed into a format that is
suitable for loading into the data
warehouse. This may involve cleaning
and validating the data, converting data
types, combining data from multiple
sources, and creating new data fields.
During this phase of the ETL process,
rules and regulations can be applied
that ensure data quality and
accessibility. You can also apply rules to
help your company meet reporting
requirements. The process of data
transformation is comprised of several
sub-processes:
• Cleansing — inconsistencies and
missing values in the data are resolved.
• Standardization — formatting rules are
applied to the dataset.
• Deduplication — redundant data is
excluded or discarded.
• Verification — unusable data is
removed and anomalies are flagged.
• Sorting — data is organized according to
type.
• Other tasks — any additional/optional
rules can be applied to improve data
quality.
• Transformation is generally considered
to be the most important part of the
ETL process. Data transformation
improves data integrity — removing
duplicates and ensuring that raw data
arrives at its new destination fully
compatible and ready to use.

Load
• After the data is transformed, it is
loaded into the data warehouse. This
step involves creating the physical data
structures and loading the data into the
warehouse. Data can be loaded all at
once (full load) or at scheduled intervals
(incremental load).
• Full loading — In an ETL full loading
scenario, everything that comes from
the transformation assembly line goes
into new, unique records in the data
warehouse or data repository. Though
there may be times this is useful for
research purposes, full loading
produces datasets that grow
exponentially and can quickly become
difficult to maintain.
• Incremental loading — A less
comprehensive but more manageable
approach is incremental loading.
Incremental loading compares incoming
data with what’s already on hand, and
only produces additional records if new
and unique information is found. This
architecture allows smaller, less
expensive data warehouses to maintain
and manage business intelligence.
Advantages of ETL process in data
warehousing:
• Improved data quality: ETL process
ensures that the data in the data
warehouse is accurate, complete, and
up-to-date.
• Better data integration: ETL process
helps to integrate data from multiple
sources and systems, making it more
accessible and usable.
• Increased data security: ETL process
can help to improve data security by
controlling access to the data
warehouse and ensuring that only
authorized users can access the data.
• Improved scalability: ETL process can
help to improve scalability by providing
a way to manage and analyze large
amounts of data.
• Increased automation: ETL tools and
technologies can automate and simplify
the ETL process, reducing the time and
effort required to load and update data
in the warehouse.
Disadvantages of ETL process in data
warehousing
• High cost: ETL process can be expensive
to implement and maintain, especially
for organizations with limited resources.
• Complexity: ETL process can be
complex and difficult to implement,
especially for organizations that lack the
necessary expertise or resources.
• Limited flexibility: ETL process can be
limited in terms of flexibility, as it may
not be able to handle unstructured data
or real-time data streams.
• Limited scalability: ETL process can be
limited in terms of scalability, as it may
not be able to handle very large
amounts of data.
• Data privacy concerns: ETL process can
raise concerns about data privacy, as
large amounts of data are collected,
stored, and analyzed.
WHAT IS ELT?
• ELT is an alternative to the traditional
extract/transform/load (ETL) process. It
pushes the transformation component
of the process to the target database for
better performance. Bectakes
advantage of the processing capability
already built into a data storage
infrastructure, ELT reduces ause it the
 time data spends in transit and boosts
efficiency.
• Extract — This step works similarly in
both ETL and ELT data management
approaches. Raw streams of data from
virtual infrastructure, software, and
applications are ingested either in their
entirety or according to predefined
rules.
• Load — Here is where ELT branches off
from its ETL cousin. Rather than deliver
this mass of raw data and load it to an
interim processing server for
transformation, ELT delivers it directly to
the target storage location. This
shortens the cycle between extraction
and delivery.
• Transform — The database or data
warehouse sorts and normalizes the
data, keeping part or all of it on hand
and accessible for customized reporting.
The overhead for storing this much data
is higher, but it offers more
opportunities to mine it for relevant
business intelligence in near real-time.
ELT vs. ETL
Benefits of ELT
• Simplifying management — ELT
separates the loading and
transformation tasks, minimizing the
interdependencies between these
processes, lowering risk, and
streamlining project management.
• Future-proofed data sets — ELT
implementations can be used directly
for data warehousing systems, but
oftentimes ELT is used in the data lake
approach in which data is collected
from a range of sources. This, combined
with the separation of the
transformation process, makes it easier
to make future changes to the
warehouse structure.
• Leveraging the latest technologies —
ELT solutions harness the power of new
technologies in order to push
improvements, security, and compliance
across the enterprise. ELT also leverages
the native capabilities of modern cloud
data warehouses and big data
processing frameworks.
• Lowering costs — Like most cloud
services, cloud-based ELT can result in
lower total cost of ownership, because
an upfront investment in hardware is
often unnecessary.
• Flexibility — The ELT process is
adaptable and flexible, so it’s suitable
for a variety of businesses, applications,
and goals.
• Scalability — The scalability of a cloud
infrastructure and hosted services like
integration platform-as-a-service (iPaaS)
and software-as-a-service (SaaS) give
organizations the ability to expand
resources on the fly. They add the
compute time and storage space
necessary for even massive data
transformation tasks.
• The differences between ELT and a
traditional ETL process are more
significant than just switching the L and
the T. The biggest determinant is how,
when and where the data
transformations are performed.
• With ETL, the raw data is not available
in the data warehouse because it is
transformed before it is loaded. With
ELT, the raw data is loaded into the data
warehouse (or data lake) and
transformations occur on the stored
data.
• Staging areas are used for both ELT and
ETL, but with ETL the staging areas are
built into the ETL tool being used. With
ELT, the staging area is in a database
used for the data warehouse.