Analyzing Big Data
Databricks
#1
Scale Up vs Scale Out
Up – there is a limit. Adding bigger, more
powerful machines SMP: Symmetric
multiprocessing
Out – Add more commodity data servers. Adding
more, smaller and cheaper machines MPP:
Massive parallel processing
Big Data vs Traditional DB technologies
Traditional DBMS Systems
• Rigid data models
• Weak fault-tolerance architecture
• Scalability constrains
• Expensive to scale
• Limitation on unstructured data
• Proprietary HW and SW
Big Data Technologies
• Schema free data models
• Fault tolerance architecture
• Highly scalable
• Lower costs
• Handles unstructured and streaming data
• Commodity hardware and open-source
software
The Data Lake (the new kid on the block)
• Term coined by James Dixon – CTO Pentaho
(~2010)
• Concept to contrast with the traditional “Data
Mart”
• The objective is to break data out of silos
• Stores “non modeled” data i.e., “raw data” with
no formal governance and (business) structure
• Stores data of any type: structured,
unstructured, streaming
• Tries to solve the data accessibility problems
• Data is used by the Business Users for
experimentation, not for operations
• Similar terms: Data Sandbox, Analytical Lab
#2
We’ll move with a strong focus and detail on
Databricks because:
• It is a leading platform for Big Data processing
• It is Spark based and leads Spark adoption
(along side with ASF)
• It provides a free platform for learn and work
with Spark
Delta Lake is an open source storage layer that
brings reliability to data lakes. Delta Lake
provides ACID transactions, scalable metadata
handling, and unifies streaming and batch data
processing. Delta Lake runs on top of your
existing data lake and is fully compatible with
Apache Spark APIs
Databricks (just what you need)
Workspaces allow you to organize all the work
that you are doing on Databricks. Like a folder
structure in your computer, it allows you to save
notebooks and libraries and share them with
other users.
Notebooks are a collection of runnable cells
(commands). Cells hold code in any of the
following languages: Scala, Python, R, SQL, or
Markdown. Notebooks need to be connected to a
cluster in order to be able to execute commands.
Dashboards can be created from notebooks as a
way of displaying the output of cells without the
code that generates them.
Clusters are groups of computers that you treat
as a single computer. In Databricks, this means
that you can effectively treat 20 computers as
you might treat one computer. Clusters allow you
to execute code from notebooks or libraries on
set of data.
Using Notebooks
The primary language of each cell is shown in ()
next to the notebook name:
You can override the primary language by
specifying the language magic command
%<language> at the beginning of a cell. The
supported magic commands are: %python, %r,
%scala and %sql
%sql select * from diamonds
Additionally:
%sh: allows you to execute shell code in your
notebook. Add the -e option in order to fail the
cell and (subsequently a job or a run command) if
the shell command has a non-zero exit status
%fs: allows you to use dbuilts filesystem
commands
%md: Allows you to include various types of
documentation, including text, images, and
mathematical formulas and equations
Databricks utilities – dbuilts.help()
Lists
• Lists are collections of items where each item in
the list has an assigned index value
• A list is mutable (meaning you can change its
contents)
• Lists are enclosed in square brackets [ ] and
each item is separated by a comma
Dictionaries
• A dictionary maps a set of objects (keys) to
another set of objects (values)
• Dictionaries are mutable and unordered (the
order that the keys are added doesn't necessarily
reflect what order they may be reported back)
• Use {} curly brackets to construct the dictionary,
and [] square brackets to index it. Separate the
key and value with colons : and with commas ,
between each pair. Keys must be quoted
Tuples
• A tuple consists of a set of ordered values
separated by commas
• Tuples are immutable
• Tuples are always enclosed in parentheses
Lambda expressions
The Lambda operator or lambda function is used
for creating small, one-time and anonymous
function objects. lambda operator can have any
number of arguments, but it can have only one
expression. It cannot contain any statements and
it returns a function object which can be assigned
to any variable.
map and filter Functions
map(function, iterable)
The map() function applies a given function to
each item of an iterable (list, tuple etc.) and
returns a list of the results.
filter(function, iterable)
The filter() function constructs an iterator from
elements of an iterable for which a function
returns true.
The filter() function filters the given iterable with
the help of a function that tests each element in
the iterable to be true or not.
#3
Hadoop
The Apache Hadoop project develops open-
source software for reliable, scalable, distributed
computing.
The AH software library is a framework that
allows for the distributed processing of large data
sets across clusters of computers using simple
programing models. It is designed to scale up
from single servers to thousands of machines
each offering local computation and storage.
Rather than rely on hardware to deliver high-
availability, the library itself is designed to detect
and handle failures at the application layer, so
delivering a highly-available service on top of a
cluster of computers, each of which may be
prone to failures.
• It’s a framework
• Distributed storage (HDFS) and computing
(MapReduce, Hive etc.)
• Reliable (designed for failure)
• Designed for commodity hardware
• Optimized for batch processing
• Open source
• Low cost
Assumptions and goals
• Hardware Failure
Detection of faults and quick, automatic recovery
from them is a core architectural goal of HDFS
• Streaming Data Access
HDFS is designed more for batch processing
rather than interactive use, the emphasis is on
high throughput of data access rather than low
latency
• Large Data Sets
HDFS is tuned to support large files. It should
provide high aggregate data bandwidth and scale
to hundreds of nodes in a single cluster
• Simple Coherency Model
A file once created, written, and closed need not
be changed except for appends and truncates.
This assumption simplifies data coherency issues
and enables high throughput data access
• “Moving Computation is Cheaper than Moving
Data”
A computation requested is much more efficient
if it is executed near the data it operates on. It’s
often better to migrate the computation closer to
where the data is located rather than moving the
data to where the application is running
• Portability Across Heterogeneous HW and SW
Platforms
HDFS has been designed to be easily portable
from one platform to another
Naming conventions
• A Cluster is a group of computers working
together - Provides data storage, data processing
and resource management
• A Node in an individual computer in the cluster
- Master nodes manage distribution of work and
data to works nodes
• A Daemon is a program running on a node -
Each Hadoop daemon (runs on a JVM) performs a
specific function in the cluster
Big Data architecture
File System
• The file system is responsible for storing data
on the cluster
• HDFS is a file system written in Java - Based on
Google GFS
• Sits on top of a native file system Ex: FAT, NTFS,
ext3, ext4, xfs
• Provides redundant storage - For massive
amounts of data
 Performs best with large files
 Files are “write once”
 The file system is optimized for reading
large streams of file (rather than random
files)
• Data is split into blocks and distributed across
multiple nodes Each block is typically 64MB or
128MB (it is configurable)
• In HDFS each block is replicated multiple times
(standard is 3 times)
• Name Node stores metada
Resource Manager (YARN)
• YARN - Yet Another Resource Negotiator
o Introduced in Hadoop 2 to improve the
MapReduce implementation
• YARN is Hadoop Resource Manager
o Resource Manager
o Job scheduler
• YARN allows multiple data processing engines
to run on a single cluster
o Batch processing (Spark, MapReduce)
o Interactive SQL (Hive, Impala)
o Advanced Analytics (Spark ML, mahout)
o Streaming (Storm, Spark Streaming)
Resource Manager (YARN daemons)
• Resource Manager
o Runs on a master node
o Global resource scheduler
o Arbitrates system resources between
competing applications
• Node Manager
o Runs on slave nodes
o Communicates with RM
Processing with MapReduce (the starting point)
• MapReduce is a programming model for
processing data in a distributed way on many
machines (nodes)
• It’s a “batch query processor”
• Although the Hadoop framework is
implemented in Java, MapReduce applications
need not be written in Java
• The concept behind MapReduce is to Map the
original data into a collection of key-value pairs
and then, Reduce the output with the same key
• Consists of two main step functions:
o Map (plus shuffle & sort)
o Reduce
• Each node processes data stored on that node
(when possible)
• The Mapper
o Map tasks run on the node where the file
(block) is stored
o Each Map task operates on a single HDFS
node/file block
• Shuffle & Sort
o Transfers the mappers output it to the
reducers as inputs
• The Reducer
o Operates on the shuffled/sorted
intermediate data
o Produces the final product
MapReduce drawbacks
• Complex code maintenance (originally
developed in Java)
• No shell
• Disk based: data stored (/processing) on disk
• Slow data access (many disk reads)
• No memory operations (caching)
• No abstract concepts for data manipulation and
processing
• Not optimized for iterative algorithms
• Limited to batch processing
• No additional libraries: ML, Graph, SQL,
Streaming etc.
Spark – Next generation MapReduce
Data formats4
Text files
• Text files are the most basic file type in Hadoop
o Can be read or written from a virtually
any programming language
 o Comma and tab delimited files are
compatible with many applications
• Text files are human readable since everything
is a string
o Useful when debugging
• However, the format is inefficient at scale
o Representing numeric values as
strings wastes storage space
o Difficult to represent binary data
such as images
o Conversion to/from native types
adds performance penalty
Avro data files - data serialization
• To understand Avro, you must first understand
serialization
• It’s the process of converting data into a
sequence of bytes/bits
o A way of representing data in memory as
a series of bytes
o Allows you to save data to disk or send it
across the network
o Deserialization allows you to read the
data back into memory
• Many programming languages and libraries
support serialization
o Ex: Serializable in Java or pickle in Python
• Backwards compatibility and cross-language
support can be challenging
o Avro was developed to address these
challenges
Avro data files
• Avro is a row-based storage format
• It stores data in a binary format using
serialization with a compact binary format
• It’s an efficient data serialization framework
o Top-level Apache project (created by
Dough Cutting)
o Widely supported throughout Hadoop
and its ecosystem
• Uses JSON for defining the data schema
• Schema metadata is embedded in the file
• Offers compatibility without sacrificing
performance
o Data is serialized according to a schema
you define
o Serializes data using a highly-optimized
binary encoding
o Specifies rules for evolving your schemas
over time
• Data is stored according with specific rules →
The Schema (type, order, format)
• The schema describes the fields and their data
types
• The schema is created using JSON format
• Efficient storage to optimized binary encoding
• Widely support throughout the Hadoop
ecosystem
o Can also be used outside of Hadoop
• Ideal for long-term storage of important data
o Can read and write from many languages
o Embeds schema in the file, so will always
be readable
o Schema evolution can accommodate
changes
Columnar formats
• These organize data storage by column, rather
than by row
• Very efficient when selecting only a small
subset of a table’s column
• Ex: RCFile, ORC (Optimized Row Columnar),
Parquet (default in Spark)
Parquet files
• Parquet is a columnar format developed by
Cloudera and Twitter
o Supported in Spark, mapReduce, hive,
Pig, Impala, and others
o Schema metadata in embedded in the file
(like Avro)
• Uses advanced optimizations (from Google’s
Dremel project)
o Reduces storage space
o Increases performance
• Most efficient when adding many records at
once
o Some optimization rely on identifying
repeated patterns
Delta Lake
• Is an open-source storage layer based on
parquet
• It was developed by Databricks and donated to
Linux Foundation
• Adds reliability, quality, performance to data
lakes
• Includes a transaction log on top of parquet
files
• Provides ACID transactions, scalable metadata
handling, and unifies streaming and batch data
processing
• Brings together the best of data warehousing
and data lakes
• Components:
o Delta engine
o Delta tables
o Delta format and storage layer
• Key Features:
o ACID transactions
o Scalable metadata
o Time travel
o Open source
o Unified Batch/Streaming
o Schema Evolution/ Enforcement
o Audit History
o DML Operations
Streaming data is data that is continuously
generated by different sources. Such data should
be processed incrementally using Stream
Processing techniques without having access to
all of the data. In addition, it should be
considered that concept drift may happen in the
data which means that the properties of the
stream may change over time. It is usually used in
the context of big data in which it is generated by
many different sources at high speed.
Flume
• Apache Flume is a real time data ingestion tool
• It’s a distributed, reliable, and available system
for efficiently collecting, aggregating and moving
large amounts of streaming data from many
different sources to a centralized data store
• The use of Apache Flume is not only restricted
to log data aggregation. Since data sources are
customizable, Flume can be used to transport
massive quantities of event data including but
not limited to network traffic data, social-media-
generated data, email messages and pretty much
any data source possible
• Apache Flume is a top-level project at the
Apache Software Foundation
SUMMARY
• HDFS/DBFS is the storage layer for Big Data
processing
o Chunks data into blocks and distributes
them across the cluster when data is
stored
o Slave nodes run DataNode daemons,
managed by a single NameNode on a
master node
o Access storage via UI or APIs
• The resource manager schedules jobs manages
resouces
o YARN works with the File System, to run
tasks where the data is stored
o Slave nodes run NodeManager daemons,
managed by a ResourceManager on a
master node
Monitor jubs using UI
• Data Formats are available with different
purposes /advantages
o Text formats
o Row based and binary/serialized formats
o Columnar formats
#4
Apache Spark
Was designed to be a modern distributed
computation framework as a response to the
limitations of MapReduce
Databricks have written over 75% of the code in
Apache Spark and have contributed more than 10
times more code than any other organization
Databricks makes big data simple by providing
Apache Spark as a hosted solution
• Apache Spark is an open-source, distributed
computation framework for parallel data
processing across many machines (clusters)
• Complementary to data abstractions and
interfaces, it manages the clusters of computers
and ensures production-level stability (not simple
at all)
• It’s a fast, in-memory, parallel data processing
engine with development APIs to allow data
workers to execute batch, streaming, SQL and
machine learning workloads
• Unified engine for large scale data analytics
Spark’s philosophy (Databricks) and what Spark
is not
1. Unified: Spark’s key driving goal is to offer a
unified platform for writing big data applications
2. Computing Engine: At the same time that
Spark strives for unification, Spark limits its scope
to a computing engine
3. Libraries: Spark’s final component is its
libraries, which build on its design as a unified
engine to provide a unified API for common data
analysis tasks
Apache Spark main characteristics
In-memory distributed processing framework
written in Scala
• Functional programming language that runs on
a JVM
Support for many data abstractions concepts
• For data storage (RDD, DataFrame, Dataset etc.)
• For data processing (map, join etc.)
API based
• Requires an API to run Spark code
• APIs: Scala, Java, Phyton, SQL and R
Unified analytics evolving platform
• Spark core for managing the cluster and basic
data objects
• Specific libraries for additional functionalities
(Streaming, Graphs, ML etc.)
Spark Core
• Is the base engine for large scale parallel data
processing
• It’s responsible for:
o Memory management and fault recovery
o Scheduling, distributing and monitoring
jobs on a Cluster
o Interacting with the storage systems
• It’s also home to the API that defines RDDs,
DataFrames, Datasets etc.
Spark Use Cases
• Data integration, ETL and ELT Pipelines for data
integration to refresh Decision Support Systems
(DSS/DW)
• Interactive analytics or business intelligence
Gaining insight from massive data sets to inform
product or business decisions in ad hoc
investigations or regularly planned dashboards
• High performance computation Reducing time
to run complex algorithms against large scale
data
• Machine learning and advanced analytics
Applying sophisticated algorithms to predict
outcomes, detecting fraud, inferring hidden
information or making decisions based on input
data
• Real-time data processing Capturing and
processing data continuously with low latency
and high reliability
Spark applications
Spark applications consist of:
• A driver process (running on a node)
o Maintains info about the Spark App and
defines the SparkSession
o Executes program input and output
operations
o Articulates resources allocation with the
resource manager
o Analyses, distributes and schedules work
on the executers
• Executor processes
o Execute code given from the driver
o Report on the stat of the execution to the
drives
The Spark Context and Spark Session
• The driver process is instantiated as an object
called the (SparkContext and) SparkSession
• It represents the connection to the existing
Spark cluster and provides the entry point for
interacting with Spark
• The SparkContext is the 1st version and the
SparkSession was introduced in Spark 2.0 as the
new entry point that subsumes SparkContext,
SQLContext, StreamingContext, and HiveContext
(for backward compatibility, they are preserved)
• In Databricks notebooks and Spark REPL, the
SparkSession is created for you and stored in a
variable called spark
Spark terminology
• Application: set of jobs managed by a driver
• Job: a set of tasks executed as a result of an
action
• Stage: a set of tasks in a job (one per partition)
that can be executed in parallel
• Task: an individual unit of work sent to an
executor
Spark provides 3 main abstraction to work with
data
• RDDs: logical distributed abstraction of your
data. Immutable collection of data elements
partitioned across the cluster. Scala/Java/Python
objects representing data. The main approach to
work with unstructured data
• DataFrames: structured data objects (schema
RDD, row) Data organized into named columns -
tables
• Datasets: extensions of the DataFrame API
which provides a type-safe object-oriented
programming
Partitions and RDDs
• In order to allow every executor to perform
work in parallel, Spark breaks up the data into
chunks, called partitions
• A partition is an atomic chunk of data. It’s a
collection of rows that sit on one physical
machine in the cluster
• By default, Spark tries to create as many
partitions as cores in the cluster (or as many as
the ones already saved on the filesystem)
• Partitions are basic units of parallelism in
Apache Spark
• RDDs are a collection of partitions
Partitions and data locality
• By default, Spark partitions file-based RDDs by
blocks
• Each partition is created in a data node / core
• An action triggers execution of tasks in parallel
What is data locality?
• Spark is specifically designed to solve Big Data
problems; it must deal with larger amounts of
data. As it is not feasible to move large data sets
towards computation, Spark provides this feature
called data Locality
• Data locality means moving computation
(mapper code) close to data rather than moving
data towards the computation
• The filesystem splits files into blocks and
distribute among various Data Nodes. When a job
is submitted, it is divided into map jobs and
reduce jobs. A Map job is assigned to a Data
Node according to the availability of the data, i.e.,
it assigns the task to a Data Node which is closer
to or stores the data on its local disk
• Data locality refers the process of placing
computation near to data, which helps in high
throughput and faster execution
RDDs
• RDDs are Spark’s main data abstraction concept
(unit of data)
• RDDs represent a collection of items distributed
across many compute nodes that can be
manipulated in parallel
• RDDs are immutable
• Spark Core provides many APIs for building and
manipulating these collections
➔ Resilient: fault-tolerant with the help of
RDD lineage graph and so able to
recompute missing or damaged partitions
due to node failures
➔ Distributed: data residing on multiple
nodes in a cluster
➔ Dataset: a collection of partitioned data
with primitive values or values of values,
e.g., tuples, lists or other objects
• RDDs can hold any type of data element
o Primitive types: integers, characters,
Booleans etc.
o Sequence types: strings, lists, arrays,
tuples, dicts etc.
o Also support nested data types and
mixed types
• Some types of RDDs have additional
functionality
 o Pair RDDs → RDDs consisting of key-value Use transformations
pairs for:
• Double RDDs • Transform data in
o RDDs consisting of numeric data sequence to modify it
as needed
How to create an RDD? (i.e. loading data in the • Create a new RDD
from an existing one.
cluster memory)
Ex: i) map(function) ii)
• From data in memory → sc.parallelize
filter(function)
• From a file or set of files → sc.textFile
• From another RDD with operations → new-
• Transformations are executed in parallel on
rdd.map(rdd)
each partition
• When possible, tasks are executed on the
Spark operations
workers nodes where data is in memory
RDDs (and DFs) support two types of operations:
There are 2 types of transformations:
• Transformations: Define a
• Narrow transformations (preserves
new RDD based on the
partitioning)
current one Ex: map(), filter()
o Input partitions will contribute to only
• Actions: Return values Ex:
one output partition
collect(), count(), 50 Spark
o Are the result of map(), filter()
data abstraction concepts
• Wide transformations (shuffles)
Transformations are the ones
o Input partitions will contribute to more
that produce new
than one output partition
RDDs/Datasets Actions are the ones that trigger
o Are the result of groupbyKey() and
computation and return results
reducebyKey().

A Transformation is a An Action is a
Reading and writing text files (transformation
function that produces function that
and action)
a new RDD / DF produces a value •
Spark can create distributed datasets from the
• textFile(file) count()
• map(func) • collect() storage source
• flatMap(func) • take(n) • Local file system, HDFS/DBFS, Cassandra,
• filter(func) • reduce() Hbase, Cloud provider, etc.
• sortBy(func) • min() • Use sc.textFile method to create the dataset
• union(rdd) • max()
• intersection(rdd) • stats() Reading a text file (is a transformation): •
• distinct() • foreach() sc.textFile(file)
• join(key value pair • countByKey() o Reads a text file(s) and creates an RDD of
rdd) • countByValue() lines (list)
• groupByKey() • saveAsTextFile(file) o Each line in the file(s) is a separate RDD
• reduceByKey(func)
element
Use actions for:
o Works only with line-delimited text files
• RDDs are immutable, • View data in the
Accepts a single file, wildcard list or a comma
data in a RDD is never console i) collect(),
changed take(n), count() separated list of files
• Transformation are • Collect data from • sc.textFile(“file.txt”) > One file
the core of the native objects in the • sc.textFile(“data/*.log”) > Wildcards
program business logic respective format i) • sc.textFile(“file1.txt,file2.txt”) > List of files
• It’s how you define parallelize(collection)
the changes you want • Write data to Writing a text file (is an action): •
to make in your data to output data sources rdd.saveAsTextFile(“path”)
produce the desired i) saveAsTextFile(file)
outcome Operations in Spark are lazy
• Lazy means that they do not compute their
results right away. Instead, they just remember
the transformations applied to some base dataset
(e.g., a file)
• The transformations are only computed when
an action requires a result to be returned to the
driver program
• This provides immense benefits to the end user
because Spark can optimize the entire data flow
from end to end

RDD / Dataframe lineage is a graph of all the

parent RDDs of an RDD
It is built as a result of applying transformations
to the RDD and creates a logical execution plan
• mydata.toDebugString() - (Verbose output)
• mydataframe.explain() - Only on DataFrames

SUMMARY
• Spark is a distributed computation framework
• Can be used interactively via the Spark shell
(Python or Scala)
• Spark has 3 main data abstraction concepts:
RDD, DataFrames & Datasets
• RDD are the basic data unit in Spark and for
unstructured data. DataFrames are data
organized in columns for structured data
• Spark has 2 types of operations on data objects:
i) Transformations ii) Actions
• Lazy execution → Transformations are not
executed until required by an action
• Spark uses functional programming i) Passing
functions as parameters ii) Anonymous functions
• Spark terminology
o Application: set of jobs managed by a
driver
o Job: a set of tasks executed as a result of
an action
o Stage: a set of tasks in a job (one per
partition)
o Task: an individual unit of work sent to an
executor
• Spark keeps track of each RDDs lineage → To
provide fault tolerance
• RDD operations are executed on partitions in
parallel
o More parallelism means more speed
o The level of parallelism can be controlled
o Operations that depend on multiple
partitions are executed in separate stages
(ex: join, reduceByKey)
#5 Note: The input function to map returns a single
Summary statistics element, while the flatMap returns a list of
elements. The output of flatMap is flattened.

map() and filter() with text strings11

map() vs flatMap()

RDDs persistence (or caching)

• Sparks doesn’t guarantee RRD data will be kept
in memory
• However, having data in-memory can be very
useful
Access speed
No need for reprocessing
Very useful for interactive algorithms
• Use the rdd.persist() or rdd.cache() to keep
data in memory
Persist offers options for Storage Levels
o Storage location
o Format in memory
o Partition replication
• Stop persisting and remove data in memory
(and disk)
o rdd.unpersist()
• In-memory persistence is a suggestion to Spark
o If not enough memory is available,
persisted partitions will be cleared
o Transformations will be re-executed
using the lineage when necessary
Checkpointing
• Maintaining RDD lineage provides resilience but
can cause problems when the lineage is very long
• Recovery can be expensive
• Risk of stack overflow
• Checkpointing saves data to HDFS → Provides
fault-tolerance across nodes
• Lineage is not saved
Shared variables
• In Spark tasks are executed on a data node
within a contained processing environment (JVM)
• Having general, read-write shared variables
across tasks would be inefficient
• To support some level of driver and node
communication, Spark provides two shared
variables: • Broadcast variables • Accumulators
Broadcast variables
• Used to give to every node a copy of input data
• Broadcast variables are created by calling:
sc.broadcast(broadcast_var)
• Broadcast variables can be access by calling
(method value): broadcast_var.value
Accumulators
• Used to give to the driver a consolidated value
from the data notes
• Accumulator variables are created by calling:
acc1 = sc.accumulator(initial_value)
• Accumulator variables can be changed by: •
Calling the add method • Using the += operator
Why do we care about counting words
• Word count is challenging over massive
amounts of data
o Using a single compute node would be
too time-consuming
o Number of unique words could exceed
available memory
• Statistics are often simple aggregate functions
o Distributive in nature
o Ex: max, min, sum, count
• Map-Reduce breaks complex tasks down into
smaller elements which can be executed in
parallel
• Many common tasks are very similar to word
count • Ex: log files analysis
Pi estimation (using Monte Carlo method)
• Compute intensive task
o Data generation and many computations
o Distributed computing
• Iterative algorithms
o In-memory processing
o The more iterations, the better the
answer will be
• Many scientific algorithms use similar
approaches
o Generating data and iterative
computations
SUMMARY
Working with RDDs
• RDDs are the basic data concept (unstructured)
in Spark
• RDDs are partitioned and process in parallel and
in memory
• RDDs can have any type of data element
• Operations are the elements to work with RDDs
o Actions
o Transformations
• Pair RDDs are a special form of RDD consisting
of Key-value pairs
• Pair RDD operations are available for different
purposes
• Reading files: textFile()
• Writing files: saveAsTextFile()
Use persist if an RDD will be used multiple times
(avoid re-computation)
Use shared variable to communicate with the
cluster and driver
• Broadcast: passes variables to each node
• Accumulators: accumulates values from the
nodes to the driver
Spark is especially suited for big data problems
that require iteration and intensive computations
• Parallel processing and in-memory persistence
makes this very efficient
MapReduce is a generic programming model for
distributed processing
• Spark implements MapReduce with Pair RDDs
• Spark provides many Maps and Reducers
Common Spark algorithms are:
• Word count, Pi estimation
• PageRank, k-means
Spark includes examples and specialized libraries
to implement many common algorithms
• GraphX and GraphFrames
• Mllib and Spark ML
#6
Apache Spark SQL
• Spark module for structured data processing
- “Table” like data objects, SQL engine and syntax
• Built on top of core Spark
• Works with the concept of DataFrame
- Data organized in columns
- Data and Metadata
Spark SQL componentes
• The DataFrame API
o Library for working with data as tables
o Defines DataFrames containing Rows and
Columns
• The SQL Engine (Interpreter and Optimizer)
- Catalyst query optimizer and Tungstem memory
management
• A SQL command line interface
• Integration with Apache Hive (HiveQL) and
ODBC/JDBC drivers
The context / environments
The main Spark SQL entry point is a SQL Context
object sqlContext.
• Is similar to Spark Context in core Spark
• Requires a SparkContext -sc
The sqlContext makes the DataFrame
functionality available while the sparkContext
focuses on the Apache Spark engine itself
After Apache Spark 2.X there is one new context -
the SparkSession (named as spark by default)
with the SQL Context embedded
DataFrames
• DataFrames are the main abstraction in Spark
SQL (like RDDs in core Spark), representing a two-
dimensional labeled data structure
• DataFrames are built on base RDDs containing
Row objects (each representing a record with
named columns).
• A distributed collection of data elements
organized into named columns (meaning: data
and metadata) - Row’s, like Pandas in Python
• Conceptually equivalent to a table in a
relational database or a dataframe in R/Python,
but with richer optimizations
DataFrames can be created:
• From data in-memory
• From an existing RDD
• From an existing structured data source
(Parquet file, JSON file, ODBC/JDBC etc.)
• By performing an operation or query on
another DataFrame
• By programmatically defining a schema
• From a Table (in Spark catalog)
DF = spark.range(5) → Range is a method to
create a DF
DF.show() → Show is the standard method to
display a DF
DF. → Hit Tab to see the DataFrame available
method
DF.collect() → Show the DF as a RDD
Display(DF) → Use Display to show a DataFrame
in a tabular format with visual options
Creating a DF reading a json file
spark.read.json(“/…)
Creating a DF reading a csv file
spark.read.csv(“/…)
Creating a DF reading a text file
spark.read.text(“/…)
Generic Load and Save Functions
To read/write data, Spark relies on generic
load/save functions
• df = spark.read.load(“path/file.parquet”)
• df = spark.read.load(“path/file.json”,
format=“json”)
• df.select(“col1”,
“col2”).write.save(“file.parquet")
• df.select(“col1”,
“col2”).write.save(“file.json“,
format=“json”)
• df = spark.read.load(“path/file.json”,
format=“Json”)
• df.select(“col1”,
“col2”).write.save(“file.parquet“,
format=“parquet”)
• Data in DataFrames can be saved to a data
source (json, parquet, delta, jdbc, orc, libsvm, csv,
text)
• Spark SQL save functions are not atomic,
transactional nor they use any locking
• The generic save method has as optional “save
mode” parameter which determines what
happens if you attempt to save a file/table that
already exists
SaveMode options
Register a DataFrame as a View or Table
Useful for sharing the data or declarative SQL
• df.createTempView(“name”) - session
scope
• df.createOrReplaceTempView(“name”)
• df.createGlobalTempView(“name”) -
application scope
• df.createOrReplaceGlobalTempView(“na
me”) global_temp.view
• df.write.saveAsTable(“name”)
• spark.catalog.dropTempView("name")
• spark.catalog.dropGlobalTempView("nam
e")
• Registering a DataFrame as View saves the
metadata on the local cluster
- Database: global_temp
- Query: select * from
global_temp.temp_view_name
• You won't find it in the tables button list on the
left because it's not globally registered on the
(Hive) metastore
• If registered as GlobalTempView other users
and notebooks that attach to this cluster can
access this table
• If we restart our Application, our table will have
to be reregistered
Saving to persisting Tables
• DataFrames are objects from your spark session
• DataFrames can be saved as persistent tables
into metastore with the saveAsTable method –
df.write.saveAsTable(“table”)
• saveAsTable will materialize the contents of the
DataFrame and create a pointer to the data in the
metastore (unlike the createTempView method)
• An existing Hive environment is not necessary.
Spark will create a default Hive metastore if
necessary
• Persistent tables will still exist even after your
Spark program has restarted
• You can create a DataFrame from a Table with
df = spark.table() or df = spark.sql(“select
statement”)
Managed and Unmanaged Tables
• Managed Tables: Spark manages both the
metadata and the data in the file store
spark.sql(“ CREATE TABLE table_name (col1 Type,
col2 type, …)“) or
df.write.saveAsTable(“table_name”)
• Unmanaged Tables: Spark only manages the
metadata. You manage the data
spark.sql(””” CREATE TABLE table_name (col1
Type, col2 type, …)
USING format OPTION (PATH ‘path’ ”””)
or
df.write.option(“path” ,
“path”).saveAsTable(“table_name”)
Running SQL queries programmatically
• Spark SQL also supports free-form SQL
• The DataFrame must be registered as View or
Table
Note: triple quote syntax (”””) is an easy way of diamond.select(“color”,”clarity”).where(“color =
creating a string that contains special characters ‘E’”).show(5) The where() clause is equivalent to
(like quotes) without having to escape them. Ex: filter()15
spark.sql(””” select … from … ”””)
DataFrame Actions (examples)
Basic Operations with DataFrames (metadata) • collect
• schema - returns a schema object describing • take
the data • count
• printSchema - displays the schema as a visual • show
tree
• cache/persist - persists the DataFrame to disk Sort and orderBy
or memory sort orders at partition level, orderBy orders at
• columns - returns as array containing the DF level (less efficient)
names of the columns diamond.sort(‘table’).show()
• dtypes - returns an array of (columns-name,
type) pairs GroupBy
• explain - prints debug info about the DataFrame display(diamond.select(“color”,”price”).groupby(
• alias - renames column names “color”).agg(avg(“price”)))

Operations with DataFrames join

• Queries – create a new DataFrame
o DataFrames are immutable
o Queries are analogous to RDD
transformations
o Queries can be chained like
transformations

• Actions – return data to the drives

o Actions trigger “lazy” execution of
queries Spark User Defined Functions – UDF
• Functions are very useful to operate on data
DataFrame Queries (examples) • You must register a function in order to use it in
• join - joins DataFrames a query (meaning, on the data on your data
• union - a new DF with the content of the DF storage)
added to base DF
• limit - a new DF with the first n rows of the base
DF diamond.limit(3).show()
• distinct - returns a new DF with distinct
elements
• select - a new DF with data from one or more
columns of the base DF
diamond.select(“carat”,”cut”,”color”).show(5)
• filter - a new DF with rows meeting a specific
condition diamond.filter(diamonds[color] == DataFrames are built on RDDs
‘E’).show(5) • DataFrames are Base RDDs containing Row
objects
• Use the rdd method to get the underlying RDD
peopleRDD = peopledDF.rdd
Summary
• Spark SQL is a Spark API for handling structured
and semistructured data
• Requires SQLContext as entry point or
SparkContext
• Use a Catalog and SQL Optimizer
• DataFrames are the key unit for structured data
• DataFrames are based on RDD of Row objects
• DataFrames allow transformation
s and actions as RDDs
• DataFrames allow “SQL like” operations
• Spark is not a replacement for a Data Base
#7 - SQL examples with Spark
Building your Data Lake
• Bronze - Raw Ingestion
• Silver – Filtered, Cleaned, Augmented
• Gold – Business Level aggregates
Summary
• Spark SQL supports many data operations and
functions like:
• GroupBy and pivot
• join, union
• Many SQL functions
• ETL/ELT pipelines with Spark Dataframes have
important advantages like performance and data
locality
• The more recent Delta format allows ACID
transformations and more SQL like operations on
the data
• The Data lake evolved to more sophisticated
data management and quality operations by
using the Delta format
• The Delta format supported the creation of the
Data Lakehouse
#9 - Spark Streaming
Streaming Data
Streaming data is data that is continuously
generated by different sources. Such data should
be processed incrementally using Stream
Processing techniques without having access to
all of the data.
It is usually used in the context of big data in
which it is generated by many different sources
at high speed
Streaming Data sources
• Web Log Files / System Log Files
• Sensors
• Mobile Devices
• Social Media
• Network sockets
• Wearables/ IoT devices
Streaming Data use cases
• Fraud detection in credit cards / bank
transactions
• Continuous patient monitoring
• Dynamic inventory management
• Dynamic routing of traffic data
• Thresholds / anomalies detection in sensors
data
• Next Best Action in on-line user engagement
• Patterns detection in social streaming data
Processing Streaming Data
• Many small and fast data inputs/streams
• Can’t have all data ready up front
• Data streams contain raw and unprocessed
data: Noise; Data quality and integrity
• Focus on the analysis of the streams patterns:
Stream content and behavior
• Actions might be required in real-time
Batch vs Streaming Data
• Data Scope: batch processing can process all
the data in the dataset. Stream processing
typically only has access to the most recent data
or within a rolling time window
• Data Size: batch processing is suitable for
handling large datasets efficiently. Stream
processing is intended for individual records or
micro batches consisting of few records
• Performance: the latency for batch processing
is typically a few hours. Stream processing
typically occurs immediately, with latency in the
order of seconds or milliseconds. Latency is the
time taken for the data to be received and
processed
• Analysis: you typically use batch processing for
performing complex analytics. Stream processing
is used for simple response functions, aggregates
or calculations such as rolling averages
Impact on technology
• Need to process (large volumes of) data quickly
and on the fly
• Means in memory parallel processing
• You have the queries but not the data
• As opposed to traditional batch systems
• You store the outputs, not the input raw data
• As opposed to traditional batch systems
• Typically, a component of a larger system
• Means that connectivity and integration are
key
Lambda Architecture
Proposed by Nathan Marz to addresses the
latency problem of typical DW’s
Kappa Architecture
Proposed by Jay Kreps as alternative to Lambda
• All data follows the same path as stream events
• Based on a stream processing engine (event-
based and real-time)
Streaming Data processing engine (aka:ESP, CEP)
• Specific engine to deal with Streaming Data
• Able to process large volumes of data in real-
time
• Low latency and high scalability
• Fault tolerance and reliability
• Integration with on-line sources and RDBMS
Systems
Spark Structured Streaming
• Apache Structured Streaming - Is a scalable and
fault-tolerant stream processing engine
• Spark Structured Streaming
• Since Spark 2.0
• Built on top of Spark SQL Library
• Separate engine based on DataFrames
• Streams are unbounded DataFrames -
Streaming DataFrames
• Allow micro-batch processing and a new low-
latency processing mode - Continuous processing
• Stream processing engine built on the Spark
SQL engine and DataFrames / Datasets • Spark
SQL engine will run queries incrementally and
continuously and updating the result as
streaming data continues to arrive (small batch
jobs)
• Structured Streaming queries are processed
using a microbatch processing engine, which
processes data streams as a series of small batch
jobs thereby achieving end-to-end latencies as
low as 100 milliseconds
• Since Spark 2.3 was introduced a new low-
latency processing mode called Continuous
Processing, which can achieve endto-end
latencies as low as 1 millisecond

Programming basic concepts
• The streaming contexto
from pyspark.streaming import StreamingContex
ssc = StreamingContext(sc,
batch_interval_seconds)
• Spark Structured Streams
StreamingDataFrames: (Since Spark 2.0)
DataFrames and Datasets can represent
static, bounded data, as well as streaming,
unbounded data
• Reading a Stream The method
spark.readStream returns a
DataStreamReader used to configure the
stream
streamingDF = ( spark .readStream
.format("socket") .option("host", "localhost")
.option("port", port_number) .load() )
Input sources format
• File: Reads files written in a directory as a
stream of data. Ex: .json() .csv() .parquet()
.text() …
• Kafka: Reads data from Kafka
• Socket (for testing): Reads UTF8 text data
from a socket connection
• Rate source (for testing): Generates data at
the specified number of rows per second,
each output row contains a timestamp and
value
• Processing the Stream
streamingDFnew = ( streamingDF
.withColumnRenamed("Index", "User_ID") #
Pick a "better" column name
.drop("_corrupt_record") # Remove an
unnecessary column )
streamingDFnew =
streamingdf.select(“device”).where(“signal =
1”)
• Checking if it’s a Stream
streamingDF.isStreaming()
• Writing the Stream
• The method DataFrame.writeStream
returns a DataStreamWriter used to
configure the output of the stream
• Parameters of the DataStreamWriter
configuration - aka Query:
• Query's name (optional)
- Name must be unique among all the
currently active queries in the spark
context
• Trigger (optional)
- Default value is ProcessingTime(0) - it
will run the query as fast as possible
• Checkpointing directory (optional for
pup/sub sinks)
• Output Mode and Output Sink
• Configuration specific to the output
sink, such as:
• The host, port and topic of the receiving
Kafka server
• The file format and destination of files
• A custom sink via
writeStream.foreach(…)
Once the configuration is completed, we can
trigger the job with a call to .start()
Queries - The query that is continually being
executed over streaming data
• Managing Streaming Queries
• query.name()
• query.stop()
• query.explain()
• query.awaitTermination()

Triggers – specifies when the system should
process the next set of data
Checkpointing
• Checkpoint stores the current state of your
streaming jog on a reliable storage (Ex: HDFS)
• With checkpoint, a terminated stream can be
restarted and will continue from where it
stopped
• To enable checkpointing you need to specify
the location of the directory with:
.option(“checkpointLocation”, checkpointPath)
Output Modes
Note: Append mode is the default
Output Sinks
Structure of a Spark Streaming program
• Define the input sources by creating input
DStreams / StreamingDFs
• Define the streaming process/computations by
applying transformation
• Define the streaming output operation
• Start receiving data and processing it using
streamingContext.start() or query.start()
• Wait for the processing to be stopped
(manually or due to any error) using
streamingContext.awaitTermination()
• The processing can be manually stopped using
streamingContext.stop() or query.stop()
Remember: you want to have a program (agent)
always running
• Receiving the data
• The data is received from sources using
receivers
• Use readStream() – Create your input
StreamingDF
• Transforming the data
• The received data is transformed
• Use DF Transformations – Define your
query logic
• Pushing out the data
• The final transformed data is pushed
out (to external systems)
• Use writeStream() - Create your query
name and output
Remember: you have to start() and stop() your
agent
• Window operations on Event Time: apply
transformations over a sliding window of data
• window(w_length, slide_interval)
Return a new StrmDF computed based on
windowed batches of the source StrmDF
• countByWindow(w_length,
slide_interval) Return a sliding window
count of elements in the stream
• reduceByWindow(func, w_length,
slide_interval)
• Tumbling windows: fixed-sized, non-
overlapping and contiguous time intervals
• Sliding windows: overlapping and
contiguous time intervals
• Session windows (Spark 3.2): dynamic
size of window length
• Join operations → Join_output =
StrmDF1.join(StrmDF2, “join_type”)
• DataFrame and SQL operations →
Using DataFrames and SQL operations on
streaming data
StrmDF.createOrReplaceTempView(“New
_Strm”) spark.sql(“select count(*) from
New_Strm”)
Summary
• Streaming data is data that is continuously
being generated and need to be analyzed in near
or real time
• Streaming data has been growing in terms of
importance and use cases with the new sources
like IoT and Social Media
• Streaming data requires a specific computing
architecture to coupe with the speed and reliance
required to process the data
• Spark incorporates key advantages to process
streaming data like parallel computing and in-
memory processing
• Spark Streaming and later, Spark Structured
Streaming, are additional Spark libraries for
processing streaming data
#10 – Graph Analytics
Intro to Graphs
• Graph algorithms provide one of the most
potent approaches to analyzing connected data
because their mathematical calculations are
specifically built to operate on relationships
• Making sense of connected data by studying
their relationships is key to have new insights like
understanding the data structures and revealing
patterns
Graph analytics use cases
• In Spark, a job is associated with a chain of a
Data Object (RDD/Dataset) dependencies
organized in a direct acyclic graph (DAG)
• A DAG (finite direct graph) is a set of Vertices
and Edges, where vertices represents the Data
Objects (RDD/Dataset) and the edges represents
the Operations to be applied on the Data Object.
• Find influencers in a network
• Find vulnerable components in a network
• Find the best path
• Fraud detection
• Understanding customer behavior
• Understanding disease behavior
• Cost analysis and rout optimization in
transports
• Wheather analysis and prediction
• Web navigation analysis (web clicks path
analysis)
• Track objects and operations sequence
(logistics/computing)
Vertices = {a, b, c}
Edges = {{a,b}, {b,c}, {c,a}}
Types of Graph algorithms
• Pathfinding Finding the shortest path with the
fewest hops or lowest weight
• Centrality Understanding which nodes are
more important in a network like Ability to
quickly spread information or bridging distinct
groups
• Community detection Connectedness and
connectivity analysis is used to find communities
and quantify the quality of groupings
Graph programming concepts
Spark Graph Analysis library
• Spark library - API for Graph analysis and graph-
parallel computation
GraphX (legacy)
• Spark library, Scala
• Property Graph - RDD extension for Graph
abstraction
- Direct Multigraph with user defined
objects attached to vertex and edges
• Graph operators and Graph algorithms
GraphFrames
• Not Spark itself, it’s a separate package (GitHub
– neo4j)
- Collaboration between Databricks, UC
Berkeley & MIT
• GraphX extended, Scala, Python, Java
• DataFrame Graphs
- With Spark optimizations: Catalyst
optimizer and Tungsten memory management
• Included in Databricks runtime for ML
Creating a GraphFrame
myGraphFrame = GraphFrame( verticesDF,
edgesDF )
 • A GraphFrame is created from a vertex
DataFrame and an Edges DataFrame • The edges DataFrame must contain a column
• The vertex DataFrame must contain a column named src that stores the source of the edge and
named id that stores unique vertex IDs a column named dst that stores the destination
of the edge
• Both DFs can have additional columns, the
Graph operators List operators numEdges, numVertices, degrees
(GraphX, scala)
Property operators mapVertices, mapEdges, mapTriplets
Structural operators reverse, subgraph, goupEdges
Join operators joinVertices, outerJoinVertices
Neighborhood aggregateMessages (Primitives for developing graph algorithms)
Aggregation
GraphFrames show, display (vertices/edges)
operators
filter, groupBy, inDgrees/outDegress = # of edges pointing in/out
(GraphFrames,
inDegrees, g.vertices().filter(“column = ‘string’”).show()
Python)
outDegrees g.filterEdges(“column = ‘string’ “)
.dropIsolatedVertices().vertices().show()
g.vertices.groupBy().max(“column”).show()
g.inDegress.show()
GraphFrames Motif finding Searching for structural patterns in a graph. DSL - Domain
operators Specific Language: () - [] -> () g.find( “(v1) - [e1] -> (v2); (v2) - [e2]
(GraphFrames, -> (v1)” ) g.find( “(v1) - [] -> (v2); !(v2) - [] -> (v1)” )
Python)
Subgraphs Select subgraphs based on a combination of motif finding and
DataFrame filters. Ex: filterVertices(cond), filterEdges(cond),
dropIsolatedVertices()

Message passing Primitives for developing graph algorithms

GraphFrames Breadth-first search Finds the shortest path(s) from one vertex to another vertex
algorithms (BFS) paths = g.bfs(“column_start = ‘string’”, “columns_end = ‘string’”)
Connected Computes the connected component (id) membership of each
components vertex results = g.connectedComponents()
Strongly connected Compute the strongly connected component (SCC) of each
components vertex results = g.stonglyConnectedComponents(maxIter=10)
Label Propagation Run static LPA for detecting communities in networks result =
Algorithm (LPA) g.labelPropagation(maxIter=5)
Page Rank Compute a Rank/Score to Edges (identify important vertices in a
graph) results = g.pageRank(MaxIter=10, resetProbability=0.15)
Shortest paths Computes shortest paths from each vertex to the given vertices
results = g.shortestPaths( [ “a”, “c” ] )
Triangle count Computes the number of triangles (3 linked nodes) passing
through each vertex results = g.triangleCount()

Loading and Saving GraphFrames v = spark.read.parquet(“path_vertices”)

e = spark.read.parquet(“path_edges”)
g = graphFrame( v , e )
g.vertices.write.parquet(“path”)
g.edges.write.parquet(“path”)
vertices/edges attributes (depending on the
specific needs)
Summary Supervised Learning Unsupervised
• Graph are mathematical structures to model Learning
relationships between objects • Regression To • Clustering
• Graphs are everywhere in the world: Sociology, predict a real value • Association
Biology, Physics, Medicine, Computer Sciences output (linear - Association rules
etc. variables) - Collaborative
• Graphs are defined with Vertices and Edges • Classification To filtering
predict a discrete • Visualization
• In Spark we have two libraries to work with
value output
Graphs: GraphX (legacy) and GraphFrames
(nonlinear /
(GitHub) categorical variables)
• To operate with GraphsFrames we have the Predictive Modeling Descriptive Modeling
Graph Operators (show, filter,…,motif finding,
subgraphs) and GraphFrame Algorithms (BFS, The three “Cs”:
Connected Components, LPA, PAgeRank, Shortest • Collaborative filtering (or recommendation
Path, Triangle Count) systems) - Unsupervised Learning
• CF is a “unsupervised learning” technique for
#11 – Spark ML Introduction recommendations (recommendation systems)
Machine Learning (ML) • The algorithm discover groups or
• Many programs are process driven (procedural, recommendations based on similarities
tell the computer what to do) between users and items
• Database transactions and queries • Is a technique for making automatic
• Controlling systems: transport, predictions (by filtering) based on historical
manufacturing processes etc. events
• Search engines • Helps users navigate data by expanding to
• Social systems: blogs, chats, email etc. topics that have affinity with their established
• An alternative technique is to have computers interests
“learn” what to do • CF algorithms are agnostics to the different
• ML refers to programs that leverage collected types of data items involved
data to drive future program behavior • Examples: • Given people who each like certain
• Development of systems that can “learn” from books, learn to suggest what someone may like in
“past data” to identify patterns (describe the the future based on what they already like •
data) and apply it (to new data) Recommendations on music playlist, videos,
• ML is based on the construction and application locations
of algorithms and statistical models to perform a
specific task without using a • Clustering - Unsupervised Learning
procedural/declarative approach • Clustering algorithms discover structure in
• ML algorithms built a mathematical model collections of data – Where no formal structure
(based on sample data) that best represents the previously existed
data (reality) • They discover what groupings (clusters)
• ML requires a process to train the model and naturally occur in data
apply the model • Examples: • Finding related news articles •
• The process to train (built) the model requires Identify similar customers • Computer vision
heavy data processing capabilities to: (groups of pixels that cohere into objects)
• Prepare the data in the right format for
the model • Classification and regression - Supervised
• Build (train) the model Leaning
ML methods are classified in • Takes a set of data records with (a) known
• Supervised leaning - When we have (a target) label(s) - Learns how to label new records based
labelled data to train the model on that information
• Unsupervised leaning - When we don’t have • Examples: • Given a set of emails identified as
labeled data to train the model spam/not spam, label new emails as spam/not
spam • Given tumors identified as benign or Main concepts in Pipelines
malignant, classify new tumors Estimator: Learns (fits) parameters from the
inputDF and returns a model, which is a
Application of ML transformer
• Fraud detection MyModel = SparkModel.fit(inputDF)
• Speech recognition Transformer: Returns a new DF (transformed)
• Object Recognition / Computer vision based on the input DF
• Customer segmentation MyModel.transform(inputDF)
• Customer churn / attrition / erosion Pipeline: Combines multiples steps named
• Next Best Offer, Next Best Action stages, into a single workflow
• Recommendation Systems pipeline = Pipeline(stages=[stg1_object, stg2_obj,
• Classification systems …])
• Transport optimization
• Production Plan optimization Spark Mllib and ML Algorithms~

Apache Spark for Machine Learning

• ML brings many computational challenges:
• Large volumes of source data to develop and
train the model
• Very intensive in terms of computational
resources
• Using of iterative algorithms (optimization
functions)
• Use both structured and unstructured data General steps of a ML pipeline

• Spark is a good fit for a ML processing engine

• Large parallelized datasets
• In-memory processing
• Scalability • Data ingestion (load the data)
• Iterative algorithms (using an API of choice) • Data exploration and preparation (analyze,
• Doesn’t require movement of data from the clean and transform data)
Big Data cluster Ex: missing values, split data into training and
test, encoders etc.
Spark MLlib • Model creation and training
• Mllib is an Apache Spark library model = model_type(), model.fit(training_data)
on top of core Spark • Model analysis and test
print(coeficients), print(error/precision)
• Mllib already includes many common
• Make preditions
ML algorithms:
prediction = model.transform(test_data)
• K-means
• Model deployment
• Alternative least squares
• Model evaluation
• Logistics regression
• Linear regression
Data exploration and preparation for ML
• Gradient descent
• Eliminate duplicates
• Still work in progress • Filter NULLs
• Still limited set of operations and algorithms • Derive new features
• Moving from Mllib to Spark ML • Drop features
• Outliers' identification
Primary API, which is the original Spark MLlib • Vectorize features
High-level API, which is a DataFrame based API in • Index Strings
spark.ml
• One Hot Encoding

Summary
• ML system are design to “learn” from data by
building a mathematical model and make
predictions based on that model (“past data”)
• ML implies a standard process
• Load data and clean/transform the data
• Built/train and test the model
• Deploy and use the model
• Evaluate the model
• Spark has unique characteristics for ML models Derive a new feature based on existing one(s)
development and implementation
• Spark does ML with two libraries: Mllib and
Spark ML
• Spark implements many ML functions and
algorithms (but still with work in progress)

#12 – ML data preparation with Spark

Machine Learning example (the simple thing)

Drop features of a DataFrame

Eliminate duplicates
Filter Nulls (eliminate rows with null values)
Identifying Outliers
• Common rule: an outlier value is more than
1.5 * IQR
• IQR is the interquartile range, defined as the
difference between the upper (Q3) and lower
(Q1) quartiles → Meaning: there are no outliers if
all the values are roughly within the Q1−1.5*IQR
and Q3+1.5*IQR range
 • For every distinct category, create a feature
(vector)
• Better than just StringIndex to solve non
ordered categories

approxQuantile() method to identify approximate

quantiles of numerical columns
• 1st parameter is the name of the column
• 2nd parameter can be either a number
between 0 or 1 (where 0.5 means to calculated
median) or a list (as in this case)
Summary
• 3rd parameter specifies the acceptable level of
• ML requires a process with data loading,
an error for each metric (0 means an exact value -
analysis, preparation and modeling
it can be very expensive)
• The data preparation process requires many
specific operations for getting the data ready for
Vectorize features (models take features as a
modeling
vector)
• Spark includes many useful functions for data
preparation for ML
• dropna()
• dropDuplicates()
• selectExpr()
• UDFs
• VectorAssemble()
• StringIndexer
• OneHotEncoder

#13 – ML examples with Spark

Index Strings – StringIndexer
• Used to transform categorical string features
into numerical features
• To use algorithms like Logistic Regression, you
must first convert the categorical variables into
numeric variables
1) Regression - Supervised learning
• Using a set of features values (ex: claimed
amount/house area) to predict a value (ex:
amount of fraud/house price) – continuous
output variables
• Linear regression is a regression model that fits
a line into the data

One Hot Enconding – OHE

Regression info – on notebook
Regression models available in Spark
(continuous output variables)
• Linear regression
• Generalized linear regression
• Decision tree regression
• Random forest regression
• Gradient-boosted tree regression
• Survival regression
• Isotonic regression
• Factorization machines regressor
2) Classification - Supervised learning
• Instead of predicting a value, in classification
we predict a class (ex: 0 and 1)
• Logistic regression is a classification model that
fits a logistic function into the data
Classification models available in Spark (discrete
or categorical output variables)
• Logistic regression • Decision tree classifier
• Random forest classifier
• Gradient-boosted tree classifier
• Multilayer perceptron classifier
• Linear Support Vector Machine
• One-vs-Rest classifier
• Naive Bayes
• Factorization machines classifier
3) Clustering - Unsupervised learning
• Clustering is an unsupervised learning problem
where we aim to group subsets of entities with
one another based on some notion of similarity
• K-means aims to partition n observations into k
clusters in which each observation belongs to the
cluster with the nearest mean
Clustering models available in Spark
• K-means
• Latent Dirichlet Allocation (LDA)
• Bisecting k-means
• Gaussian Mixture Model (GMM)
• Power Iteraction Clustering (PIC)
4) Association rules - Unsupervised learning
• Association rules is used to discover relations
between variables in large datasets. In a
transaction with a variety of items, association
rules are meant to discover how or why certain
items are connected
• The FP-growth operates on item sets to identify
frequent items and association rules with the
confidence, lit and support indicators
Confidence: how often (0-1) the association rule
has been found to be true
Lift: measures the correlation between the
antecedent the consequent ( 1 yes)
Support: association rule occurrence / total
transactions (0-1)
Association Rules (FPM) models available in
Spark
• FP-Growth
• PrefixSpan
Summary
Spark has many functionalities to cover the
ML programming lifecycle
• Data exploration and preparation
• Model creation and training
• Model analysis and execution
• Model evaluation
Spark ML available algorithms
• Regression (continuous output variables)
• Classification (discrete or categorical output
variables)
• Clustering
• Frequent Pattern Mining (association)
• Collaborative Filtering (association)
#14 – Review Class and Examples
Summary
Anlysing Big Data and Apache Spark • Analyzing
Big Data means processing large datasets (over
multiple machines) with parallel processing
technologies like Apache Spark • Apache Spark is
a multi-language and multi-purpose engine for in-
memory data processing in distributed
environments • Batch and streaming data • SQL
analytics • Data engineering and Machine
learning • Support for building the Data
Lakehouse • Apache Spark is the most widely-
used engine for scalable computing Thousands of
companies, including 80% of the Fortune 500, use
Apache Spark™. Over 2,000 contributors to the
open-source project from industry and academia.
ABD 21 1st Exam - 11 January (pdf)
1. What is the meaning of the data
processing model: “Scale Out”?
a. Means adding bigger, more powerful
machines
b. Means adding more, smaller
machines
c. Means processing data with scalable
machines
d. Means processing data outside the
cluster
2. In Databricks notebooks you can?
a. Program only in Python and Spark
b. Program only in the language defined
in the Notebook creation
c. Select a language at the cell level
d. Select a language for a set of cells
3. In a Databricks notebook, to access the
cluster driver node console, what magic
command is used?
a. %fs
b. %drive
c. dbutils.fs.mount()
d. %sh
4. What is a DataFrame?
a. A data processing method in Spark
b. A distributed dataset in-memory
c. A distributed dataset on-disk
d. A distributed dataset on-disk and in-
memory
5. What is MapReduce?
a. A programming language
b. A programming model
c. A set of functions for processing big
data
d. The original Apache Software
Foundation query engine for big data
6. What is Avro in Hadoop?
a. A program to load data with high
parallelization
b. A column-based data format
c. A row-based data format
d. A text-based data format for
compatibility and portability
7. What is a Pair RDD?
a. Two RDDs in sequence in a
transformation statement
b. An RDD with only two rows
c. An RDD with a pair operation
d. An RDD with key-value pairs
8. What is the result of the Spark statement
bellow?
sc.parallelize(mydata,3)
a. Creates an RDD with a minimum of 3
partitions
b. Creates an RDDs named mydata and
the value 3
c. Generates an error of too many
parameters
d. Creates 3 RDDs with mydata
9. What Spark function has, as an output, a
Pair RDD?
Commented [GMS1]: Word diz que é esta, e chat gpt
a. map tambem
b. flatmap
Commented [GMS2]: Word diz que é esta, e o
c. textFile abd_exames passados jan2021 também
d. wholeTextFiles
10. Select the right statement to create a
DataFrame:
a. DF = spark.createDataFrame([“Line 1”,
“Line 2”])
b. DF = spark.table("mytable")
c. DF = spark.load("mytable")
d. DF = sc.parallelize([“Line1”, “Line2”])
11. Select the right statement regarding
reduceByKey():
a. reduceByKey() is a wide
transformation
b. reduceByKey() is a narrow
transformation
c. reduceByKey() is a lazy transformation
d. reduceByKey() is an action
12. Select the false statement regarding
Spark terminology:
a. A Job is a set of tasks executed as a
result of an action
b. A Stage is a set of tasks in a job that
can be executed in parallel
c. A task is an individual unit of work sent
to an executor
d. An Application can only contain one
job
13. What is a lambda function?
a. It’s a function defined without a name
and with only one parameter
b. It’s a function defined without a name
and with only one expression
c. It’s a function that can be reused with
many parameters
d. It’s a function that can be reused with
many expressions
14. collect() is a Spark function that?
a. You can extensively use to display
Dataframes content
b. It's not available for Dataframes
c. You can extensively use to display
RDDs content
d. You should use with caution to
display RDDs content
We should use the collect() on smaller
dataset usually after filter(), group(), count()
e.t.c. Retrieving on larger dataset results in
out of memory.
15. With the instruction
sc.textFile(“file:/data”) you are?
a. Reading a file from your hdfs file
system
b. Reading a file called “file:/data”
c. Reading a file from your local non-
Hadoop file system
d. Reading a file called “data” stored in a
folder called "file"
16. Window operations in Spark are used
for?
a. Define a window to display data
b. Select a window of data to display
c. Freeze data in memory from a window
of data
d. Apply transformations over a window
of data
17. What is the data type that results from
the following spark instruction:
spark.range(10)?
a. A list
b. A tuple
c. An RDD
d. A DataFrame
18. Select the right statement regarding
Spark transformations:
a. Wide transformations are very efficient
because they don’t move data from the
node
b. Narrow transformations are very
efficient because they don’t move data
from the node
c. Both wide and narrow
transformations move data from the
node
d. None of the narrow or wide
transformations move data from the
node
19. In Spark, lazy execution means that:
a. Execution will take some time because
it needs to be sent to the worker nodes
b. Execution will take some time because
the code is interpreted
c. Execution is done one line at the time
d. Execution is triggered only when an
action is found
20. What is the difference between
Spark Streaming and Structured
Streaming?
a) Structured Streaming is for
structured streaming data processing
and Spark Streaming is for
 unstructured streaming data
processing
b) Spark Streaming is the new ASF
library for Streaming Data and
Structured Streaming the old one
c) Structured Streaming is a stream
processing engine and Spark
Streaming is an extension to the
core Spark API to streaming data
processing
d) Structured Streaming relies on
micro batch and RDDs while Spark
Streaking relies on DataFrames and
Datasets
21. What is a Tumbling Window in Spark
Streaming?
a. A fixed-sized, non-overlapping and
contiguous window of data
b. An overlapping and contiguous
window of data
c. A non-contiguous window of data
d. A dynamic size window of data
22. Spark ML library can be classified as:
a. A mature ML library with a very wide
range of predictive and descriptive
models to choose from
b. A strong Deep Learning library
c. A complete ML framework for data
analysis
d. A ML library with a reasonable set of
models but still with work in progress
23. What is the main difference between
Spark MLlib and ML?
a. There is no difference apart from the
bigger set of algorithms available on
Spark ML
b. Spark ML works with Streaming Data
c. Spark MLlib is faster
d. Spark ML is based/ works on/with
Dataframes
Spark MLlib carries the original API built
on top of RDDs.
Spark ML contains higher-level API built
on top of DataFrames for constructing ML
pipelines.
24. In a Spark ML program, what is the
purpose of the code bellow?
model.transform(mydata)
a. Create a machine learning model
based on the data of ‘mydata’
b. Apply the model in ‘model’ to the
data in ‘mydata’
c. Create a new model based on ‘mydata’
d. Adjust the model based on the data in
‘mydata’
25. Is a Spark ML program, what is the
purpose of the code bellow?
model.fit(mydata)
a. Train a machine learning model based
on the data of ‘mydata’
b. Apply the model in ‘model’ to the data
in ‘mydata’
c. Create a new model based on ‘mydata’
d. Adjust the model based on the data in
‘mydata’
26. What is the result of the Spark ML
instruction bellow? lr =
LogisticRegression(maxIter=10)
a. A logistic regression object is declared
with a maximum of 10 interactions
b. A logistic regression is executed with a
maximum of 10 interactions
c. A logistic regression is trained with a
maximum of 10 interactions
d. A logistic regression is estimated with a
maximum of 10 interactions
27. The vertex DataFrame in a GraphFrame
is:
a. A free form DataFrame
b. A DataFrame that must contain a
column named 'id'
c. A DataFrame that must contain a
column named 'src' and 'dst'
d. A DataFrame that must contain a
column named 'id', 'src' and 'dst'
28. What is DSL used for in GraphFrames?
a. Formatting the output of a
GraphFrame query
b. Declare a GraphFrame object
c. Search for patterns in a graph
d. Define properties in a GraphFrame
29. Based on the figure bellow, explain the
shown algorithm line by line and the
expected outcome with an example (of
the outcome). Complementary, explain
what is supposed to be input content for
the algorithm and where is it (the name
of the the object) identified in the code.
The code is a map-reduce algorithm to
perform a word count.
In the first line, the outcome is a list of
words (one word per line). Each word is
split on the delimiter“ “, after applying
the flatMap, which flattens the results. In
the second line, the map function applies
a lambda function to each word to create
a tuple with (word, 1).
This way, the outcome is a key-value pair,
being the word the key and the value 1.
In the third line, an aggregation is
performed by key (word), being the
outcome a list of the distinct words, with
the count of occurrences registered.
The input content of the algorithm is an
rdd representing at least a line of text,
which is called in the first line before the
first lambda function.
Example: rdd = sc.parallelize([“This is the
first line”, “This is the second line”, “This
is the last line”])
With this rdd, the outcome for the first
line (after applying the method collect()
)would be:
['This', 'is', 'the', 'first', 'line', 'This', 'is',
'the', 'second', 'line', 'This', 'is', 'the', 'last',
'line']
With this rdd, the outcome for the
second line (after applying the method
collect() ) would be:
[('This', 1), ('is', 1), ('the', 1), ('first', 1),
('line', 1), ('This', 1), ('is', 1), ('the', 1),
('second', 1), ('line', 1), ('This', 1), ('is', 1),
('the', 1), ('last', 1), ('line', 1)]
Finally, the outcome after the line (after
applying the method collect() ) would be:
[('line', 3), ('This', 3), ('last', 1), ('the', 3),
('first', 1), ('is', 3), ('second', 1)]
30. What are the general steps of a Spark ML
program? Give an example.
Regarding a Spark ML pipeline, the
general steps are
Data ingestions: in this step, the data is
loaded
Data preparation: after loading the data,
an exploratory data analysis should be
performed, followed by cleaning the
dataset (incoherences, missing values).
Then the data should be properly split
into training and test datasets
Model creation and training, with the
training data
Model analysis and testing
Make predictions, by applying the trained
model to the test dataset
(model.transform(test_data)
Model evaluation, where we should print
the evaluation metrics
Example for a multiple linear regression
problem: We have data regarding clients
from a insurance customer data and we
want to predict the health insurance
charges based on their ages, BMI,
smoking habits and the number of
children. 1 - First we need to create a
dataframe with the customer data -
which is in a list called clients df =
spark.createDataFrame(records,
["age","bmi","smoker","charges"]) 2 -
Convert the smoker column to a binary
column using the String Indexer:
smokerIndexer = StringIndexer(inputCol =
"smoker", outputCol = "smoker_binary")
df2 = smokerIndexer.fit(df).transform(df)
df3 = df2.drop("smoker") 3 - Then we
should vectorise the data, to be used by
the model, creating a features column
assembler =
VectorAssembler(inputCols=["age",
"bmi", "children", "smoker_binary"],
outputCol="features")
Stages += [assembler]
pipeline = Pipeline(stages=stages)
pipelineModel = pipeline.fit(df3) dataset
= pipelineModel.transform(df3) 4 - Then
keep relevant columns (label and
features) and rename the column
"charges" to "label" df_final =
dataset.select(["features",
 "charges"]).selectExpr("features as
features", "charges as label") 5 - The next
step is to split the data into training and
test sets (example 80% train, 20% test)
(trainingData, testData) =
df_final.randomSplit([0.8,0.2]) 6 - Create
a linear regression object lr =
LinearRegression(maxIter = 10) 7 - Chain
Linear Regression in a Pipeline pipeline =
Pipeline(stages = [lr]) 8 - Train the Model
model = pipeline.fit(trainingData) 9 -
Make Predictions predictions =
model.transform(testData) 10 - Evaluate
the model eval = RegressionEvaluator
(labelCol = "label", predictionCol =
"prediction") print("Coefficients: " +
str(model.stages[0].coefficients))
print("R2:", eval.evaluate(predictions,
{eval.metricName: "r2"}))
31. Explain the main differences between
real time data processing and batch data
processing.
Batch data processing deals with groups
of transactions that have already been
collected over a period of time. The goal
of a batch processing system is to
automatically execute periodic jobs in a
batch. It is ideal for large volumes of
data/transactions, as it is increases
efficiency in comparison with processing
them individually. However, it can have a
delay between the collection of data and
getting the result after the batch process,
as it is normally a very time consuming
process. The latency in this systems is the
order of a few hours. An example of
batch processing are the processing of
the salary within a company every
month.
On the other hand, real-time data
processing deals with continuously
flowing data (individual records or micro
batches) in real-time. Real-time
processing systems need to be very
responsive and active all the time, in
order to supply immediate response at
every instant. In this systems, the
information is always up to date.
However, the complexity of this process
is higher than in batch processing. The
latency in this systems is the order of a
few seconds or miliseconds . An example
can be a radar system, weather forecast
or temperature measurement and
normally involve several IoT sensors
ABD_Apoio_exame_vf9
1. Which of the following is NOT a
component of big data architecture?
a) Data storage
b) Data sources
c) Machine Learning
d) Anonymization
2. Only one of the following sentences is
NOT true. Choose it.
a. Workspaces allow you to organize all the
work that you are doing on Databricks
b. The objective of data lake is to break data
out of silos
c. Clusters are single computer that you
treat as a group of computers
d. The data lake stores data of any type:
structured, unstructured, streaming
3. Which of the following is the most
appropriate definition for “Jobs”
a. Are packages or modules that provide
additional functionality that you need to solve
your business problems
b. Are structured data that you and your team
will use for analysis
c. Are the tool by which you can schedule
execution to occur either on an already
existing cluster or a cluster of its own
d. Are third party integrations with the
Databricks platform
4. Only one of the following options is
NOT true. Choose it.
a. Notebooks in order to be able to execute
commands
b. Dashboards can be created from
notebooks as a way of displaying the output
of cells without the code that generates them
c. Clusters allow you to execute code
from apps
d. Workspaces allow you to organize all the
work that you are doing on Databricks

5. Only one of the following sentences is
correct. Choose it.
a. Clusters allow you to execute code
from notebooks or libraries on set of data
b. Dashboards can not be created from
notebooks
c. Tables can not be stored on the cluster
that you're currently using
d. Applications like Tableau are jobs
6. Only one of the following sentences is
correct. Choose it. The command “%sh”
allows you:
a. To display the files of the folder
b. To execute shell code in your notebook
c. To use dbutils filesysytem commands
d. To include various types of documentation,
including text, images and mathematical
formula and equations
7. Concerning the characteristics of
“Lists” only one of the following options
is NOT true. Choose it.
a. Lists are collections of items where each
item in the list has an assigned index value
b. Lists consists of values separated by
commas
c. A list is mutable (meaning you can change
its contents)
d. Lists are enclosed in square brackets [ ]
and each item is separated by a comma
8. Only one of the following sentences is
correct. Choose it.
a. A dictionary maps a set of keys to
another set of values
b. Tuples are never enclosed in parentheses
c. Dictionaries are immutable
d. The Lambda function is used for creating
big and multimer function objects
9. Only one of the following sentences is
NOT true. Choose it.
Several types of data are stored in the
following DBFS root locations:
a. /databricks-datasets
b. databricks-results
c. /FileStore
d. databricks-finalizations
10. The example annex uses the lambda
function. The syntaxes are correct?
a. Yes
b. No
11. Concerning the use of Hadoop, only
one of the following sentences is correct.
Choose it.
a. A Node is a group of computers working
together
b. A Cluster is an individual computer in the
cluster
c. A Daemon is a program running on a
node
d. With Hadoop we can’t explore the nodes
(name or data)
12. Which of the following is NOT a
component of Hadoop data architecture?
a) HDFS
b) MapIncrease – o que é é MapDecrease
c) Yarn
d) Spark
13. Concerning HDFS, only one of the
following sentences is NOT true. Choose
it.
a. HDFS is responsible for storing data on the
cluster
b. HDFS s a File System written in Java
c. HDFS sits on top of a native file system
d. HDFS provides non-redundant storage
14. Only one of the following sentences is
correct. Choose it.
a. YARN allows multiple data processing
engines to run on a single cluster
b. MapReduce is a programming model for
processing data in a distributed way on a
unique node
c. MapReduce drawbacks is optimized for
iterative algorithms
d. MapReduce drawbacks is not limited to
batch processing
15. Only one of the following sentences is
NOT true. Choose it. Avro data files:
a. Is a row-based storage format for Hadoop
b. It stores data in a non-binary format
c. It’s an efficient data serialization framework
d. Uses JSON for defining the data schema
16. Concerning Parquet files, only one of
the following sentences is NOT true.
Choose it.
a. Parquet is supported in Spark,
mapReduce, hive, Pig, Impala, and others
b. Parquet reduces performance
c. Parquet is a columnar format developed by
Cloudera and Twitter
d. Parquet is most efficient when adding
many records at once
17. Which of the following is NOT a Delta
Lake key feature?
a. Closed Format:
b. Scalable Metadata Handling
c. Unified Batch and Streaming Source and
Sink
d. Schema Enforcement and Evolution
18. Choose the option with the correct
command to copy the file foo.txt from
local disk to user’s directory in HDFS
a) $hdfs dfs -put foo.txt foo.txt
b) $hdfs dfs -ls foo.txt foo.txt
c) $hdfs dfs -get foo.txt foo.txt (??)
d) $hdfs dfs -rm foo.txt foo.txt
19. Concerning Apache Flume, only one of
the following sentences is NOT true.
Choose it.
a. Apache Flame is a real time data ingestion
tool
b. The use of Apache Flume is only
restricted to log data aggregation
c. Apache Flame is a distributed, reliable, and
available system for efficiently collecting,
aggregating and moving large amounts of
streaming data from many different sources
to a centralized data store
d. Apache Flume is a top level project at the
Apache Software Foundation
20. Can you explain why use Apache
Storm?
Apache Storm is a free and open source
distributed realtime computation system.
Apache Storm makes it easy to reliably
process unbounded streams of data, doing for
realtime processing what Hadoop did for batch
processing. Apache Storm is simple, can be
used with any programming language, and is
a lot of fun to use!
Apache Storm has many use cases: realtime
analytics, online machine learning, continuous
computation, distributed RPC, ETL, and more.
Apache Storm is fast: a benchmark clocked it
at over a million tuples processed per
second per node. It is scalable, fault-tolerant,
guarantees your data will be processed, and is
easy to set up and operate.
21. Concerning Apache Spark, only one of
the following sentences is NOT true.
Choose it.
a. Apache Spark is a unified analytics engine
for large-scale data processing
b. Apache Sparks is a open-source
distributed computation framework for
executing code in parallel across many
different machines
c. Apache Sparks is a slow, in-memory
data processing engine without
development APIs
d. Apache sparks is ease to use and write
applications quickly in Java, Scala, Python, R
and SQL.
22. Which one of the following sentences
is NOT a key Spart characteristics?
a. Processing of unstructured, structured and
streaming data
b. Distributed processing
c. Unified analytics and evolving platform
d. Works only with Python language
23. Only one of the following sentences is
NOT true. Choose it.
Spark Core is:
a. Responsible for memory management and
fault recovery
b. The base engine for large scale parallel
data processing
c. Responsible for interacting with storage
systems
d. Home to the API that defines Data
warehouses
24. Define the Spark’s main benefits in
terms of performance and productivity
development
Spark’s main benefits in terms of performance
are using in-memory computing, Spark is
considerably faster than Hadoop MapReduce
(100x in some tests). It also can be used for
batch and real-time data processing. In terms
of Developer productivity are Easy-to-use
APIs for processing large datasets that
Includes 100+ operators for transforming.
25. Concerning main abstraction provides
by Spark to work with data, only one of
the following sentences is correct.
Choose it.
a. Data Frames are the main approach to
work with unstructured data Scala objects
representing data
b. RDD are structured data objects in Spark
Data organized into named columns -tables
c. Data sets are extensions of the Data
Frame API which provides type-safe,
object-oriented programming interface
and an optimizer
d. A partition is a collection of columns that sit
on one physical machine in our cluster
26. Only one of the following sentences is
NOT true. Choose it.
a. Data Locality means moving data
towards computation rather than moving
computation close to data
b. Hadoop stores data in HDFS, which splits
files into blocks and distribute among various
Data Nodes
c. With Hadoop, when a job is submitted, it is
divided into map jobs and reduce jobs
d. RDDs represent a collection of items
distributed across many compute nodes that
can be manipulated in parallel
27. Define the 3 ways to create an RDD in
Spark
Meaning: loading data in the cluster memory
From data in memory - sc.parallelize
From a file or set of files - sc.textFile,
spark.read
From another RDD - RDD operations,
ex: new-rdd.map(rdd)
28. Which one of the following sentences
is NOT a key RDDs characteristics?
a. Resilient
b. Distributed
c. Mutable
d. In-memory
29. Which one of the following functions
is NOT a single RDD transformation?
a. sortBy
b. flatMap
c. reduce
d. distinct
30. Only one of the following sentences is
NOT true. Choose it. Working with RDDs,
function “collect” means:
a. Return the first element of the dataset
b. Return an array with the first n elements of
the data set
c. Return all the elements of the dataset
as an array at the driver program
d. Return the first n elements of the RDD
31. Concerning Spark terminology, only
one of the following sentences is correct.
Choose it.
a. Task is an individual unit of work sent
to an executor
b. Application is a set of tasks executed as a
result of an action
c. Job is a set of tasks in a job that can be
executed in parallel
d. Stage is a set of jobs managed by a driver
32. Concerning RDDs persistence, only
one of the following sentences is NOT
true. Choose it.
a. Storage levels let you control partition
replication
b. The persist method offers other options
called persistence levels
c. Storage levels let you control storage
Commented [GMS3]: OpenAI diz que é esta, e as pessoas
location não discordam
d. By default the persist method stores data
in memory only
33. Concerning RDDs persistence,
describe how to choose the persistence
level
Memory only: when possible for best
performance and saves space by saving a
serialized object in memory
Disk: when re-computation is more expensive
than disk read. Ex: expensive functions or
filtering large datasets
Replication: when re-computation is more
expensive than memory
34. Concerning RDDs checkpointing, only
one of the following sentences is NOT
true. Choose it.
a. Risk of stack overflow
b. Maintaining RDD lineage provides
resilience but can cause problems when
the lineage is very short
c. Checkpointing saves data to HDFS
d. Recovery can be expensive
35. Only one of the following sentences is
NOT true. Choose it. In Spark:
a. Tasks are executed on a data node within
a contained processing environment
b. Broadcast variables are used to give to
every node a copy of input data
c. Read-write shared variables across
tasks would be very efficient
d. Accumulators are used to give to the driver
a consolidated value from the data notes
36. Only one of the following sentences is
NOT true. Choose it.
Spark is especially useful when working
with any combination of:
a. Iterative algorithms
b. Large amounts of data
c. Intensive computations
d. Plus privacy
37. Which one of the following functions
is NOT an iterative algorithm used in
Spark?
a. K-means
b. PI estimation
c. Sum count
d. PageRank
38. What is the Spark algorithm used in
the code of this picture?
a. Word Count
b. PI estimation
c. K-means clustering
d. PageRank implementation
39. Concerning K-means clustering, only
one of the following sentences is NOT
true. Choose it.
a. K-means partitions the data into k clusters
b. There are only one version on k-means
algorithms and implementations
c. K-means is a common iterative algorithm
used in Machine Learning for cluster analysis
d. The objective of k-means is to group the
observations into clusters where each
observation belongs to the cluster with the
nearest mean
40. Describe the 5 steps to implement the
K-means clustering algorithm in Spark
1.Choose the seeds (k)
2.Each individual is associated with the
nearest seed
3.Calculate the centroids of the formed
clusters
4.Go back to step 2
5.End when the centroids cease to be re-
centered
41. Which one of the following functions
is NOT a Spark SQL component?
a. A SQL command line interface
b. The DataFrame IPA
c. Integration with HiveQL
d. The SQL Engine
42. Concerning Spark SQL, only one of
the following sentences is NOT true.
Choose it.
a. DataFrames can be created by performing
an operation or query on another DataFrame
b. Instead of using Java serialization
datasets use a specialized encoder to
serialize the objects for processing or
transmitting over the network
c. Python supports the Dataset API
d. DataFrames are built on base RDDs
containing Row objects
43. In Spark SQL, when you operate with
DataFrame (DF) Queries, if you use the
command “limit” what result do you
obtain?
a. A new DF with data from one or more
columns of the base DF
b. Returns a new DF with distinct elements
c. A new DF with the content of the DF added
to base DF
d. A new DF with the first n rows of the
base DF
44. Concerning Spark SQL, only one of
the following sentences is NOT true.
Choose it.
a. DataFrames are the key unit for
unstructured data
b. Spark SQL is a Spark API for handling
structured and semi-structured data
c. Requires SQLContext as entry point or
SparkContext
d. Use a Catalog and SQL Optimizer
45. Using Spark SQL present an
instruction to join Table A and Table B
using the field “name”
join = ta.join(tb, ta.name == tb.name)
join.show()
46. Which one of the following functions
is NOT a streaming data source?
a. Sensors
b. Social media
c. Survey questionnaires
d. Mobile devices
47. Which one of the following functions
is NOT a streaming data use cases?
a. Dynamic routing of traffic data
b. Fraud detection in bank transactions
c. Anomalies detection in sensors data
d. Discrete patient monitoring
48. Concerning processing streaming
data, only one of the following sentences
are correct. Choose it.
a. Stream processing is used for simple
response functions, aggregates or
calculations such as rolling averages
b. Stream processing typically occurs
immediately, with latency in the order of
minutes
c. Stream processing is intended for grouped
records
d. Stream processing typically only has
access to the oldest data
49. Which one of the following values is
NOT accepted by the command
DataSreamWriter.format in Spark
Streaming?
a. Kafka
b. Context
c. Memory
d. Delta
50. Describe the main steps of the
structure of a Spark Streaming program.
Define the input sources by creating input
DStreams / StreamingDFs
Define the streaming process/ computations
by applying transformation and output
operations to Dstreams / StreamingDFs
Receiving the data - The data is received from
sources using receivers, use readStream() –
Create your input StreamingDF
Transforming the data - The received data is
transformed, use DF Transformations –
Define your query logic
Pushing out the data - The final transformed
data is pushed out (to external systems), Use
writeStream() - Create your query name and
output
Remember: you have to start() and stop() your
agent
51. Which one of the following is NOT a
Type of Graph?
a. Undirect graph
b. Total graph
c. Directed graph
d. Directed star
52. Which one of the following is NOT a
Type of Graph algorithms?
a. MapReduce
b. Community detection
c. Centrality
d. Pathfinding
53. Which one of the following is NOT a
Graph operator?
a. Neighborhood aggregation
b. Join operators
c. Spark operators
d. Property operators
54. What is the GraphFrames algorithm
used in the code annex? paths =
g.bfs(“column_start= ‘string’”, “columns_end=
‘string’”)
a. Strongly connected components
b. Breadth-first search
c. Label Propagation Algorithm
d. Page Rank
55. Concerning Graph Analysis, only one
of the following sentences is NOT true.
Choose it.
a. In Spark we have two libraries to work with
Graphs: GraphX and GraphFrames
b. Graph are mathematical structures to
model relationships between objects
c. Graphs are defined with Squares and
Edges
d. Graphs are everywhere in the world:
Sociology, Biology, Physics, Medicine,
Computer Sciences etc.
56. Describe the differences between ML
supervised learning and unsupervised
Learning.
Supervised leaning is when we have (a target)
labelled data to train the model. Regression (to
predict a real value output (linear variables)
and Classification (to predict a discrete value
output (nonlinear / categorical variables)).
Unsupervised leaning is when we don’t have
labeled data to train the model. Clustering,
association and Visualization.
57. Concerning Machine Learning, only
one of the following sentences is NOT
true. Choose it.
a. A classification / regression system takes a
set of data records with (a) known label(s)
b. Classification and regression are forms
of “unsupervised learning”
c. Clustering is classified as “unsupervised
learning”
d. Collaborative Filtering helps users navigate
data by expanding to topics that have affinity
with their established interests
58. Which one of the following is NOT an
application of ML?
a. Object Recognition
b. Transport detection
c. Speech recognition
d. Classification systems
59. Describe the reasons Spark is a good
fit for a ML processing engine.
Large parallelized datasets
In-memory processing
Scalability
Iterative algorithms (using an API of choice)
Doesn’t require movement of data from the
Big Data cluster
60. Concerning Machine Learning, only
one of the following sentences is NOT
true. Choose it.
a. Spark does ML with two libraries:
Hadoop ML and Spark ML
b. Spark has unique characteristics for ML
models development and implementation
c. Spark implements many ML functions and
algorithms
d. ML system are design to “learn” from data Select a language at the cell level
by building a mathematical model and make
predictions based on that model What is an RDD?
Select one:
61. Which one of the following is NOT a A dataset in-memory
step in data preparation for ML?
a. Filter NULLs What is a lambda function?
b. Eliminate duplicates Select one:
c. Index Strings It’s a function that can be reused with many
d. Vectorize tables expressions

62. Using Spark ML programming what In Spark, lazy execution means that:
results can we obtain with the code Select one:
annex? Execution is triggered only when an action
is found

What is the difference between Spark

a. Drop features of a DataFrame Streaming and Structured Streaming?
b. Identifying Outliers Select one:
c. Vectorize features Structured Steaming is a stream
d. StringIndexer processing engine and Spark Streaming is
an extension to the core Spark API to
63. Which one of the following is NOT a streaming data processing
regression model available in Spark?
a. Perceptron regression What is the main difference between Spark
b. Gradient-boosted tree regression MLlib and ML?
c. Random forest regression Select one:
d. Decision tree regression Spark ML works with Dataframes

64. Which one of the following is NOT a The vertex DataFrame in a GraphFrame is:
regression model available in Spark? Select one:
a. Multilayer perceptron classifier A DataFrame that must contain a column
b. Decision classifier named 'id'
c. Gradient-boosted tree classifier
d. Random forest classifier ABD_Exames passados_jan2021 (removi
as perguntas repetidas)
65. Concerning Machine Learning, only 3. What is an RDD?
one of the following sentences is NOT Select one:
true. Choose it. a. A Hadoop data format
b. A dataset in-memory
a. ML requires a process with data loading,
c. A dataset in-disk
analysis and preparation d. A dataset in-disk and in-memory
b. Spark provides two libraries for ML:
“pyspark.ml” and “pyspark.mllib” 8. What is the output object type that results from
applying a map() function to an RDD that was created
c. Spark includes “dropna()” function for data
from a text file with the sc.textFile() method?
preparation for ML Select one:
d. Spark includes “UDFs” function for a. String
data modelling b. Tuple
c. List
d. Dictionary
In Databricks notebooks you can?
Select one:
9. Select the right statement to create an RDD:
Select one:
a. myRDD = [“Alice”, “Carlos”, “Frank”, “Barbara”]
b. myRDD = sc.load(“Alice”, “Carlos”, “Frank”,
“Barbara”)
c. myRDD = sc.parallelize([“Alice”, “Carlos”, “Frank”,
“Barbara”])
d. myRDD = load(“Alice”, “Carlos”, “Frank”, “Barbara”)
16. Select the right instruction to create a
Dataframe?
Select one:
a. Dataframe = spark.range(10)
Out[1]: DataFrame[id: bigint]
b. Dataframe = sc.textFile(“mydata”)
c. Dataframe = spark.dataFrame(“Mydata”)
d. Dataframe = sc.parallelize(“mydata”)
19. What is the difference between Spark Streaming
and Structured Streaming?
Select one:
a. Structured Streaming is for structured streaming
data processing and Spark Streaming is for
unstructured streaming data processing
b. Spark Streaming is the new ASF library for
Streaming Data and Structured Streaming the old one
c. Structured Steaming is a stream processing engine
and Spark Streaming is an extension to the core
Spark API to streaming data processing
d. Structured Streaming relies on micro batch and
RDDs while Spark Streaming relies on DataFrames and
Datasets
20. What are DStreams?
Select one:
a. Data abstractions provided from Spark core library
b. Data abstractions provided from Spark MLlib
c. Data abstractions provided from Spark Streaming
d. Data abstractions provided from Spark Structured
Streaming
• Both map() and flatMap() are an transformations
that apply a function to the elements of an RDD an
return a new RDD with the transformed elements.
• On the one hand, map() transformation takes one
element and produces one element (one-to-one
transformation). On the other hand, flatMap() takes
one element and produces zero, one or more
elements (one-to-many transformation)
• In this example, let’s create an RDD which has a
list of 2 lines of text o rdd =
sc.parallelize(["First Word",
"Second Word"])
• If we perform an upper case transformation using
the map(), the output will be a list with the two
lines of text with all the words in uppercase: o
code: rdd.map(lambda line: line.upper()).collect()
o output: ['FIRST WORD', 'SECOND
WORD']
• If we perform an upper case transformation using
the flatMap(), the output will be a list with all the
characters character in upper case, as the results
are flattened: o code: rdd.flatMap(lambda line:
line.upper()).collect()
o output: ['F', 'I','R','S', 'T','
','W','O','R','D','S','E','C','O','N','D',' ','W','O','R','D']
2nd Exam 2021
Question 1
Write a Databricks Notebook program to do the
following tasks
1. Display your Spark session version, master and
AppName

Exams ABD (removi as perguntas

repetidas)
6 – What is a pair RDD?
2. Print the list of the files in /FileStore/tables
a) Two sets of RDDs in a transformation
b) An RDD with only two rows
c) An RDD with two data types
d) An RDD with only two columns

30 – Explain the differences between the map()
and flatMap() transformations in Spark.
Complementary, show and example created by
you (with code) and explain the output
differences.
3. Copy one of the files (you may suggest a non-
existing) name to a new version with the name
prefix “New_”
 1. Create a RDD with the following 7 lines of text:
"First Line", "Now the 2nd", "This is the 3rd line",
"This is not the 3rd line?", "This is the 5th ", "This
is the 6th", "Last Line"

1 – sc
2 – %fs ls /FileStore/tables
3 - dbutils.fs.cp("dbfs:/FileStore/tables/File.csv",
2. Create a RDD (based on the previous one with
"dbfs:/FileStore/tables/New_File.csv", True) Spark functions) with only the lines that start
with the word "This"
Question 2
Write a Spark program to do the following tasks:
1. Create a Python list of Temperatures in ºF as
in: [50, 59.2, 59, 57.2, 53.5, 53.2, 55.4, 51.8, 53.6,
55.4, 54.7]
2. Create an RDD based on the list above

3. Create a RDD (based on the previous one with

Spark functions) without lines with the word
"not"

3. Display the main stats of the RDD (like count,

mean, stdev etc.)

4. Create a program to convert the elements of

the RDD from ºF to ºC
5. Display the result of the new RDD with the 4. Create a RDD (based on the previous one with
elements in ºC Spark functions) with the text in capital letters

5. Display only two elements of the resulting RDD

1 – List = [50, 59.2, 59, 57.2, 53.5, 53.2, 55.4, 51.8, 53.6,
55.4, 54.7]
2 – rdd = sc.parallelize(List)
rdd.collect()
3 – rdd.stats() 1–
4 – rdd_Celsius = rdd.map(lambda T: (T-32)*5/9) text = ["First Line", "Now the 2nd", "This is the 3rd line",
5 – rdd_Celsius.collect() "This is not the 3rd line?", "This is the 5th ", "This is the 6th",
"Last Line"]
rdd = sc.parallelize(text)
Question 3 rdd.collect()
Write a Spark program to do the following tasks: 2–
rdd_2 = rdd.filter(lambda line: line.startswith("This"))
rdd_2.collect() Question 5
3– Write a Spark algorithm to count the number of
rdd_3 = rdd.filter(lambda line: "not" not in line)
rdd_3.collect() distinct words on the text line below (or similar
4– input you may write).
rdd_4 = rdd.map(lambda line: line.upper()) • o "First Line"
rdd_4.collect() • o "Now the 2nd line"
5 – rdd_4.take(2) • o "This is the 3rd line"
• o "This is not the final
Question 4 line?"
Write a Spark program to do the following tasks: • o "This is the 5th "
1. Create an RDD with the data below (simulating • o "This is the 6th"
customer acquisitions): o • o "Last Line is the 7th"
'Client1:p1,p2,p3'
o 'Client2:p1,p2,p3' The output must be a pair RDD with the distinct
o 'Client3:p3,p4' words and the corresponding number of
occurrences.

2. Write a Spark program to convert the above

RDD in the following output: o 'Client1',
'p2'
o 'Client1', 'p3'
o 'Client2', 'p1'
o 'Client2', 'p2'
o 'Client2', 'p3'
o 'Client3', 'p3'
o 'Client3', 'p4'

1–
rdd = sc.parallelize(['Client1:p2,p3', 'Client2:p1,p2,p3',
'Client3:p3,p4'])
rdd.collect()
2- text = ["First Line", "Now the 2nd line", "This is the 3rd line",
rdd2 = rdd.map(lambda line: line.split(":")) "This is not the final line?", "This is the 5th", "This is the 6th",
rdd3 = rdd2.map(lambda fields: (fields[0],fields[1])) "Last Line is the 7th"]
rdd4 = rdd3.flatMapValues(lambda p: p.split(",")) rdd = sc.parallelize(text)
rdd4.collect() rdd2 = rdd.flatMap(lambda line: line.split()) \
.map(lambda word: (word, 1)) \
.reduceByKey(lambda w1,w2: w1+w2).collect()
Question 6
Write a Spark program to do the following tasks:
1. Create a RDD based on the list of the following
4 tuples:
[('Mark',25),('Tom',22),('Mary',20),('Sofia',26)]
2. Create a Dataframe based on the previous RDD
with the 2 columns named "Name" and "Age"
3. Display the new DataFrame
4. Create a new DataFrame based on the
previous one, adding a new column named
"AgePlus" with the content of Age multiplied by
1.2
5. Write the new DataFrame (with the columns
"Name", "Age" and "AgePlus") in dbfs in Delta
format
6. Check that the written DataFrame/file is in
dbfs
1 – rdd =
sc.parallelize([('Mark',25),('Tom',22),('Mary',20),('Sofia',26)])
2 – df = spark.createDataFrame(rdd).toDF("Name","Age")
3 – display(df)
4 – df2 = df.withColumn("AgePlus", df["Age"]*1.2)
5 – df2.write.format("delta").save("/FileStore/tables/df2")
6 - %fs ls /FileStore/tables/df2
Question 7
Explain the major differences between Spark SQL,
Hive, and Impala. Give examples supporting your
explanation for each of the 3 cases.
• Spark SQL is a distributed in-memory computation engine.
It is a spark module for structured data processing which is
built on top of Spark Core.
o Spark SQL can handle several independent processes and
in a distributed manner across thousands of clusters that are
distributed among several physical and virtual clusters.
o It supports several other Spark Modules being used in
applications such as Stream Processing and Machine
Learning
• On the other hand, Hive is a data warehouse software for
querying and managing large distributed datasets, built on
top of Hadoop File System (HDFS) o Hive is designed for
Batch Processing through the use of Map Reduce
Programming
• Finally, Impala is a massively parallel processing (MPP)
engine developed by Cloudera.
o Contrarly to Spark, it supports multi-user environment
while having all the qualities of Hadoop it supports column
storage, tree architecture, Apache HBase storage and HDFS.
o It has significantly higher query throughput than Spark
SQL and Hive.
o However, in large analytical queries Spark SQL and Hive
outperform Impala.
Question 8 - Write a Spark program to do the
following tasks:
1. Create a Graph that depicts the following data and
relationships: o Alice: age=31; Esther:
age=35; David: age=34; Bob age=29
o Alice is married to Bob and is
friend of Esther
o Esther is married to David and is
friend of Alice
o Bob and David are friends
2. Print the edges and vertices of your graph
3. Create a subgraph with only the friend relationships
and show the result
1 - from graphframes import *
# Create the vertices
vertices = sqlContext.createDataFrame([
("a", "Alice", 31),
("b", "Esther", 35),
("c", "David", 34),
("d", "Bob", 29)], ["id", "name", "age"])
# Create the edges
edges = sqlContext.createDataFrame([
("a", "d", "married"),
("a", "b", "friend"),
("b", "c", "married"),
("b", "a", "friend"),
("c", "d", "friend")], ["src", "dst", "relationship"])
# Create the graph
g = GraphFrame(vertices, edges)
2–
display(g.vertices)
display(g.edges)
3–
friends = g.edges.filter("relationship = 'friend' ")
friends.show() 1–
records = [ [19,27,0,168,"y"], [18,33,1,177,"n"],
Question 9 - Write a Spark program to do the [28,35,2,191,"y"], [32,38,3,208,"n"] ]
following tasks: df = spark.createDataFrame(records,
["age","bmi","children","charges","smoker"])
1. Create a DataFrame simulating insurance
2–
customer data with: o The columns: from pyspark.ml.feature import StringIndexer
["age","bmi","children","charges" smokerIndexer = StringIndexer(inputCol = "smoker",
,"smoker"] outputCol = "smokerIndex")
o 4 records with the following values: [ df2 = smokerIndexer.fit(df).transform(df)
3-
[19,27,0,168,"y"], [18,33,1,177,"n"], df_final = df2.drop("smoker")
[28,35,2,191,"s"], [32,38,3,208,"n"]] display(df_final)

2. Create a new DataFrame based on the
previous one with the values in the smoker
column encoded in 1/0 values
3. Display the new DataFrame without the
original "smoker" column
1. Write a Spark program to do the following
tasks:
o Create a DataFrame simulating insurance
customer data, with (or assume you already have
a file with the data and read it from dbfs):
o The columns:
["age","bmi","children","charges"
,"smoker"]
o 4 records with the following values: [
[19,27,0,168,"y"], [18,33,1,177,"n"],
[28,35,2,191,"s"], [32,38,3,208,"n"] ]

2. Write a Spark ML program to perform a
multiple linear regression in an attempt to
predict the health insurance charges based on
the clients age, bmi, number of children and
his/hers smoking habits
3. At the end of your program print also the:
coefficients, RMSE and the r2 of your model
1–
records = [ [19,27,0,168,"y"], [18,33,1,177,"n"],
[28,35,2,191,"y"], [32,38,3,208,"n"] ]
df = spark.createDataFrame(records,
["age","bmi","children","charges","smoker"])’
2–
from pyspark.ml.feature import StringIndexer
from pyspark.ml.feature import VectorAssembler
from pyspark.ml import Pipeline
from pyspark.ml.regression import LinearRegression
from pyspark.ml.evaluation import RegressionEvaluator
# Convert the smoker column to a binary column
smokerIndexer = StringIndexer(inputCol = "smoker",
outputCol = "smoker_binary")
df2 = smokerIndexer.fit(df).transform(df)
df3 = df2.drop("smoker")
# Vectorize the data
from pyspark.ml import Pipeline
stages = []
assembler = VectorAssembler(inputCols=["age", "bmi",
"children", "smoker_binary"], outputCol="features")
stages += [assembler]
pipeline = Pipeline(stages=stages)
pipelineModel = pipeline.fit(df3)
dataset = pipelineModel.transform(df3)
# Keep relevant columns and rename the column "charges"
to "label"
df_final = dataset.select(["features",
"charges"]).selectExpr("features as features", "charges as
label")
# Split the data into training and test sets (80% train, 20%
test)
(trainingData, testData) = df_final.randomSplit([0.8,0.2])
# Create a linear regression object
lr = LinearRegression(maxIter = 10, regParam = 0.3,
elasticNetParam = 0.8)
# Chain Linear Regression in a Pipeline
pipeline = Pipeline(stages = [lr])
# Train the Model
model = pipeline.fit(trainingData)
# Make Predictions
predictions = model.transform(testData)
3–
eval = RegressionEvaluator (labelCol = "label", predictionCol
= "prediction")
print("Coefficients: " + str(model.stages[0].coefficients))
print("RMSE:", eval.evaluate(predictions, {eval.metricName:
"rmse"}))
print("R2:", eval.evaluate(predictions, {eval.metricName:
"r2"}))
20 – Can you explain why use Apache Storm?
Apache Storm is a free and open source
distributed realtime computation system. Apache
Storm makes it easy to reliably process
unbounded streams of data, doing for realtime
processing what Hadoop did for batch
processing. Apache Storm is simple, can be used
with any programming language, and is a lot of
fun to use!
Apache Storm has many use cases: realtime
analytics, online machine learning, continuous
computation, distributed RPC, ETL, and more.
Apache Storm is fast: a benchmark clocked it at
over a million tuples processed per second per
node. It is scalable, fault-tolerant, guarantees
your data will be processed, and is easy to set up
and operate.