Data Description

and Diagnostics
PREPROCESSING TECHNIQUES
Preprocessing techniques refer to a set of steps and
methods applied to raw data before it undergoes
analysis. The primary goal is to clean, organize, and
enhance the data, making it more suitable for accurate
and efficient analysis. This process is crucial because
real-world data is often messy, containing errors,
missing values, or unnecessary details that can hinder
the effectiveness of analytical tools.
 1. DATA CLEANING
Data cleaning is the process of identifying and fixing
errors or inconsistencies in a dataset to ensure that it is
accurate, reliable, and ready for analysis. Imagine you
have a bunch of puzzle pieces, but some are missing,
others are damaged, and a few don't quite fit. Data
cleaning is like sorting through these pieces, fixing or
removing the problematic ones, and making sure
everything fits together smoothly.
 Data Cleaning
Handling Missing Data:
Sometimes, data is incomplete,
and certain values are missing.
Data cleaning involves
figuring out how to fill in these
missing pieces, either by
estimating values or removing
incomplete records.
Dealing with Noisy Data:
Noisy data includes outliers or
errors that don't reflect the
actual trends in the dataset.
Data cleaning identifies and
corrects these anomalies,
ensuring that they don't skew
the analysis.
 Real Life Example
Imagine you are managing a customer database for an e-commerce platform. In
this database, there's a column for customer ages, and upon closer inspection,
you notice a few entries that seem off. Some customers have ages recorded as
150 or even 999.

Identifying and Handling Outliers: Handling Missing Data: As you

You recognize these extreme age
values as outliers that could distort any
age-related analysis. Through data
cleaning, you decide to either correct
these entries by using a more realistic
age or, if necessary, remove these
outliers to maintain the accuracy of
your customer age data.
continue to review the age column,
you also notice some entries marked
as "NA" or left blank. In the data
cleaning process, you decide on a
strategy to fill in these missing values,
perhaps by using the average age of
customers or another suitable method.
 2. DATA INTEGRATION
Data integration is like bringing together pieces of
information from different sources to create a complete
and unified picture, much like combining various
ingredients to make a delicious recipe. It's the process
of merging data from different places to form a cohesive
and comprehensive dataset for better understanding
and analysis.
3. DATA TRANSFORMATION
Data transformation is like giving your data
a makeover to make it more
understandable and ready for analysis. It
involves converting and organizing data in
a way that makes it easier to work with and
reveals valuable insights.
 DATA TRANSFORMATION
1. NORMALIZATION. Think of
normalization as putting different things
on the same scale. If you have data with
varying ranges, like temperatures in
Celsius and Fahrenheit, normalization
transforms them into a common scale,
making comparisons and analysis more
straightforward. It's like converting
temperatures to either all Celsius or all
Fahrenheit for consistency.
3. Smoothing. Sometimes data can be a
bit noisy, with lots of ups and downs.
Smoothing is like applying a filter to
remove unnecessary fluctuations, helping
to identify trends more easily. It's like
smoothing out a bumpy road to reveal the
underlying path.
2. Concept Hierarchy Generation. Imagine
you have data about products, and some are
categorized differently, like "Electronics" and
"Gadgets." Data transformation creates a
concept hierarchy, organizing these
categories into a structured framework. It's
like creating a clear hierarchy that groups
related concepts together, making the data
more organized and meaningful.
4. Aggregation. Aggregation involves
combining detailed data into summary
information. For instance, if you have
daily sales data, you might aggregate it
into monthly totals. It's like summarizing
the fine details into more manageable and
informative chunks.
 4. DATA REDUCTION
Data reduction is like simplifying a large
and complex puzzle by focusing on the
most important pieces. It involves
techniques to streamline and condense
data while retaining its essential
information. Let's break it down:
 DATA REDUCTION
1. Dimensionality Reduction. 2. Concept Hierarchy
Imagine you have a puzzle with Generation. Imagine you have
many pieces, and each piece data about products, and some are
represents a different aspect of categorized differently, like
your data. Dimensionality "Electronics" and "Gadgets." Data
reduction is like combining or transformation creates a concept
eliminating some pieces, hierarchy, organizing these
reducing the overall complexity. categories into a structured
It's like focusing on the main framework. It's like creating a clear
shapes and colors that capture hierarchy that groups related
the essence of the puzzle concepts together, making the data
without getting lost in more organized and meaningful.
unnecessary details.
Questions?
 Group Activity. Pick a company and
research about how big companies applied
the preprocessing techniques.
Instructions:
Company Selection:
Pick a well-known and sizable company for your research. Consider companies from various
industries such as technology, retail, finance, or healthcare.
Dataset Identification:
Research and identify a dataset or data-related challenge the chosen company faced. Look for
information on their website, annual reports, or reputable business articles.
Preprocessing Techniques:
Investigate how the company applied preprocessing techniques to clean, integrate, transform, or
reduce their data. Explore the specific challenges they aimed to address and the impact on
their analytical processes.
Success Stories:
Look for success stories or case studies related to the company's data preprocessing efforts.
Identify any significant improvements in decision-making, efficiency, or overall business
performance attributed to these techniques.
Presentation Preparation:
Prepare a brief presentation summarizing your findings. Include details on the chosen company,
the dataset, preprocessing techniques applied, and the outcomes or benefits achieved.
Presentation Delivery:
Present your research to the class, sharing insights into how the selected company strategically
employed preprocessing techniques to enhance their data quality and analysis.