WHITEPAPER

Data Engineering -
Beginner’s Guide

By 2025, it is anticipated
that the world will have
created and stored 200
Zettabytes of data. While
storing this amount of
data is a challenge in and
of itself, it is significantly
more difficult to extract
value from this quantity of
data.
It is crucial for businesses to derive value from their data in order
to better inform business decisions, protect their enterprise and
customers, and expand their operations. In order to accomplish this,
businesses must hire individuals with specialised data governance
and strategy skill sets, such as data engineers, data scientists, and
machine learning engineers.
This exhaustive guide will cover the fundamentals of data. You
will also gain a deeper understanding of the significance of data
engineering and how to begin extracting more value from your data.
WHAT IS DATA ENGINEERING?
There are many factors to consider when it comes to adding value
to data, both inside and outside the organisation. Your organisation
most likely generates data from internal systems or products,
integrates with third-party applications and vendors, and must
provide data in a specific format for various users (internal and
external) and use cases.
The data generated and collected by your business likely has
compliance requirements, such as SOC2 or Personally Identifiable
Information (PII), that you are legally obligated to safeguard.
When this is the case, data security becomes the highest priority,
introducing additional technical challenges for data in transit and at
rest. We continue to hear about massive data breaches in the news,
which can cripple your company and its reputation if they occur.
Not only must your data be secure, but it must also be accessible
to your end users, perform to your business’s specifications, and
possess integrity (accuracy and consistency). If your company’s data
is secure but unusable, it cannot add value. Numerous aspects of a
data governance strategy require specialised knowledge.
Data engineering comes into play at this point.
Data Team at a Glance
Data Engineer ML Engineer Data Scientist
• Builds data • Applies and • Extracts value
pipelines deploys data from data
• Analyzes and models • Data modeling
organizes • Bridges gap creation
data between data • Measures and
• Data ingestion engineer and improves results
and quality data scientist
checks • More emphasis
on
mathematics
 HOW DOES A DATA ENGINEER ADD VALUE?
In lieu of an abstract explanation, consider the following scenario:
the CEO wants to know how much money the company could save
by purchasing and distributing materials in bulk.

You must be able to determine how to charge back to different

business units any unused materials. This will likely necessitate
You’re probably familiar
the collection of data from your ERP system, supply chain system,
with the term “Big potential third-party vendors, and internal business structure. Some
Data,” and the size of companies may have attempted to create this report in Excel in the
this market continues past, with multiple business analysts and engineers contributing to
data extraction and manipulation.
to expand. By 2023,
the market for big data Data engineers enable an organisation to collect data from various
analytics is projected to sources efficiently and effectively, typically storing the data in a data
lake or several Kafka topics. After each system’s data has been
reach $103 billion, with
collected, a data engineer can determine how to optimally join the
poor data quality costing data sets.
the US economy up to
With this infrastructure in place, data engineers can construct data
$3.1 trillion annually. pipelines that allow data to flow out of source systems. The output
of this data pipeline is then stored in a separate location, typically
in a highly accessible format that can be queried by a variety of
business intelligence tools.

Data engineers are also responsible for ensuring that the inputs
and outputs of these data pipelines are accurate. This frequently
involves data reconciliation or additional data pipelines for source
system validation. Using various monitoring tools and site reliability
Simply by increasing engineering (SRE) practises, data engineers must also ensure that
data accessibility by 10 data pipelines flow continuously and that information is kept up-to-
percent, Fortune 1000 date.
companies can generate Data engineers add value by automating and optimising complex
an additional net income systems, thereby transforming data into a usable and accessible
of over $65 million. business asset.

ETL and ELT

It is the responsibility of the data engineer to know which
data pipeline strategy to implement and why. The two most
prevalent strategies revolve around data extraction, loading, and
transformation (ELT). Data must always be extracted in some fashion
from a data source, but the next step is not as straightforward.

The ELT use case is frequently observed in data lake architectures

or systems that require multiple source raw extracted data. This
allows multiple processes and systems to process data extracted
from the same source. When combining data from multiple systems
and sources, it is advantageous to co-locate and store the data in a
single location prior to transforming the data.
In contrast, an ETL (extract, transform, load) process transforms
the extracted data before loading it into a file system, database, or
data warehouse. This style is frequently less efficient than an ELT
process, as data for each batch or stream is frequently required
from related systems. On each execution, you would be required to
re-request data from the required systems, resulting in an increase
in system load and additional time spent waiting for the data to
become available.

However, when simple transformations are applied to a single

source of data, ETL may be preferable because it reduces the
complexity of your system, potentially at the expense of data
enablement.

To improve data performance, availability, and enablement, the

general recommendation is to utilise ELT processes whenever
possible.

Performance
Data must not only be accurate and accessible to a data engineer,
but also efficient. When processing gigabytes, terabytes, or even
petabytes of data, processes and checks must be implemented to
ensure that the data meets service level agreements (SLAs) and
quickly adds value to the business.

Continuous Integration and Delivery Continuity

Code is never a solution of the “set it and forget it” variety.
Data governance requirements, tools, best practises, security
procedures, and business requirements are constantly changing
and adapting; consequently, your production environment must also
be fluid and adaptable.

This necessitates automated and verifiable deployments. Older

methods of software deployment typically involved running a
build, copying and pasting the result onto your production server,
and conducting a manual “smoke test” to determine whether the
application was functioning as expected.

This is not scalable and poses a threat to your business.

Any bugs or issues that you may have missed during testing (or any
environment-specific influences on your code) that are presented
to the end user during live testing on a production environment
will result in a negative customer experience. The best practise for
promoting code is to implement automated processes that verify
code’s functionality in various scenarios. This is a common practise
for unit and integration tests.

Unit tests ensure that, given a set of inputs, individual pieces of

code produce the expected outputs independently of other code

Data Governance Structure
Least
Authentication and Authorization
Data Availability
Data Security
Maturity and Complexity
Information Architecture
Provisioning and Rights Management
Data Stwardship
Data Asset Definitions
Data Catalog and Data Classification
Data Lineage
Data Mastering
that uses that piece of code. These add value by validating complex
logic within each piece of code and providing evidence that the
code executes as expected.
Integration testing is the next level up from that. This ensures that
code fragments produce the expected output(s) for a specified set
of inputs. This is frequently the most important testing layer, as it
ensures that systems integrate as expected.
By combining unit tests and integration tests with modern
deployment strategies such as blue-green deployments, the
likelihood that new code will have an adverse effect on your
customers and business is significantly reduced.
Disaster Recovery
It is essential to have a plan in place in the event of a system
failure, despite the fact that many businesses prioritise providing
customers with maximum value as quickly as possible. Failure is
inevitable, despite the fact that many businesses rely heavily on
cloud providers to minimise downtime and guarantee SLAs. This
necessitates that systems be built to withstand a critical system
failure.
DATA GOVERNANCE VS. DATA ENGINEERING
Data administration is the primary focus of data governance,
whereas data execution is the primary focus of data engineering.
While data engineers are a part of the overall strategy for data
governance, data governance encompasses much more than data
collection and curation. It is unlikely that your organisation will have
an effective data governance practise without the implementation of
data engineers.

WHO MAY ACCESS MY INFORMATION?

In a data governance practise, rules and regulations specify who
within your organisation should have access to specific pieces of
data.

A data engineer is responsible for applying classification and

tagging rules upon collection of data from multiple systems. This
may involve adding data points to the collected data or storing
data on disc separately. Then, when the data is aggregated or
transformed, this information must be included in the final product.

HOW CAN I AUDIT AND PROVISION ACCESS?

To be considered compliant with the numerous regulations imposed
on businesses, you must be able to monitor who has access to
your data and how that access is modified. This includes informing
data consumers of any modifications made to the data. If you are a
consumer of data and it changes without your knowledge, it is likely
that your systems will fail. It is therefore essential to monitor who is
and should be consuming data.

While data governance practises determine what these rules should

be, data engineers are responsible for implementing them. This
may involve configuring IAM rules in AWS or Microsoft Azure so that
specific roles can only read data from various sources and systems.
The security team is then responsible for ensuring that users only
have access to the appropriate roles.

WHAT IS DATA SCIENCE?

Where would you start if you wanted to extract value from multiple
data sets?

If you have information about customers and their orders, for

instance, you could determine what additional products you could
sell to them based on the orders of other customers. If you could
successfully link customers to their purchases, you could likely
upsell on future orders.

This may be straightforward if you have a small number of

customers and orders. You could hire business analysts who are
experts in your industry and have worked with your customers for
years to infer what your customers want.

What would you do, however, if you had millions of customers and
millions of transactions? What if external vendors were to provide
you with additional customer information? What happens if your data
is unstructured and cannot be easily combined with other datasets?
How can you be sure that information is correlated and make
decisions based on data rather than intuition?

This is where data science enters the equation. Data scientists are
responsible for applying scientific methods, processes, algorithms,
and systems to structured and unstructured data in order to extract
valuable business insights.

HOW DO DATA SCIENTISTS ADD VALUE?

Data scientists use programming to validate theories and statistical
models with data. They typically have solid backgrounds in
mathematics, statistics, and programming. In our data model
example, we determined that customers who purchased diapers are
80% more likely to purchase hand sanitizer. While this is a simple
and logical conclusion, the relationship between an organization’s
data and its business value is frequently more complex. It is also
possible that your organisation has so much data that you do not
know where to begin. By increasing their data accessibility by 10
percent, Fortune 1000 companies can generate over $65 million
in additional net income. For this reason, it is crucial for businesses
to have data scientists who create data models and analyse data,
making it accessible to business units. It is highly plausible that your
enterprise could cross-sell or up-sell services to customers more
effectively, or that it could save money by using data models to
predict resource usage.

WHAT IS A MACHINE LEARNING ENGINEER?

Data engineering and data science converge at the intersection
of machine learning engineering. Typically, these engineers
have a stronger mathematical background than the average data
engineer, but not to the extent of a data scientist. These engineers
can leverage data engineering tools and frameworks in a big data
ecosystem, apply data models developed by data scientists to the
data, and automate the deployment of these models. This task is not
simple.

Engineers in machine learning must be proficient in data structures

and algorithms from both a mathematical and computational
standpoint. Data must be ingested into the model and computations
must run in a high-performance environment for a data model to be
productionized. This may necessitate the management of terabytes
of real-time data to inform business decisions.

HOW DO MACHINE LEARNING ENGINEERS WORK WITH DATA

SCIENTISTS?
When data scientists work with data to validate models, they
typically use environments like Python or R within an analytical
notebook like Jupyter. This notebook communicates with a cluster
to translate queries into a big data platform-specific engine, such
as Spark. While this method reduces the required development
experience and time to derive value, it necessitates additional
production work. This includes: • Data quality checks • Optimizing
query performance • Creating a Continuous Integration Continuous
Delivery (CI/CD) ecosystem around model changes • Incorporating
data from multiple sources into the data model

Application of machine learning and data science to distributed

systems

Despite the fact that some of these skills overlap with those of a
data engineer (data ingestion, data quality checks, etc.), the required
responsibilities and skills are significantly concentrated in a few
areas of data engineering.

DATA ENGINEERING IS NOW MORE IMPORTANT THAN EVER

In today’s society, the adage “knowledge is power” could not be
truer. Large organisations generate, consume, and process more
data than ever before.

As numerous examples have demonstrated, data is a crucial

component of knowledge, and the process of transforming data
into knowledge can be quite complex. There are various levels of
data processing and analysis, and there may be instances in your
organisation in which a person’s experience in a particular business
practise and field gives them a level of knowledge that can be
supported by the data. Nonetheless, the amount of knowledge that
Big Data can generate about your business and its effect on your
business is frequently neglected (and overwhelming).

Experts such as data engineers, data scientists, and machine

learning engineers are typically very expensive and experienced
resources for a company to hire, creating a formidable barrier to
entry.

Disclaimer
CloudAngles is a global IP-based technology consulting and services firm that enables
enterprises across industries to achieve superior competitive advantage, customer
experiences, and business outcomes through the utilisation of digital and cloud technologies.
CloudAngles is a digital transformation partner to the world’s most avant-garde businesses,
bringing extensive domain, technology, and consulting expertise to assist in reimagining
business models, accelerating innovation, and maximising growth. As a socially and
environmentally responsible organisation, CloudAngles focuses on both growth and
sustainability in order to create long-term stakeholder value. Using our differentiated
intellectual property, we assist global business services in preparing for the next normal. Ritz
Global (Azure Partner), and MindMach (Niche AI Player) are the members of our group. Visit
https://www.cloudangles.com/ to discover how CloudAngles enables clients to lead with
digital.

CloudAngles, Copyright 2022 All privileges reserved Without the express written permission
of CloudAngles, no portion of this document may be reproduced, stored in a retrieval system,
or transmitted in any form or by any means, electronic, mechanical, photocopying, recording,
or otherwise. This information is subject to change without prior notice. All trademarks
mentioned in this document belong to their respective owners.