Q1) Define big data. Why is big data required?

How does traditional BI environment differ Q9) Write a short note on Classification of Analytics? Analytics can be classified into three
from big data environment? - Big Data: Big data refers to extremely large and complex
datasets that exceed the capabilities of traditional data processing methods. It is characterized
by its volume, velocity, variety, and sometimes veracity, encompassing vast amounts of
information that traditional systems struggle to handle. Why Big Data is Required: Big data is
necessary for in-depth analysis, informed decision-making, predictive analytics, and gaining a
competitive advantage. It enables organizations to process and derive insights from massive
datasets, providing a comprehensive view of business operations and customer behavior.
Traditional BI vs. Big Data Environment: Traditional BI environments primarily deal with
structured data in relational databases, operate in batch mode, and may face scalability
challenges. In contrast, big data environments handle diverse data types, support real-time
processing, scale horizontally, and offer cost-effective solutions for storing and processing large
datasets. The shift to big data reflects the need for more flexible, scalable, and real-time data
processing capabilities in today's digital landscape.
Q15) Discuss data locality in detail and how it helps improve performance in Hadoop?
main categories: 1. Descriptive Analytics: - Descriptive analytics focuses on summarizing
historical data to provide insights into what has happened in the past. It involves the
examination of patterns, trends, and key performance indicators (KPIs) to understand the
current state of affairs. 2. Predictive Analytics: - Predictive analytics uses statistical algorithms
and machine learning techniques to analyze historical data and predict future outcomes. It
enables organizations to make informed decisions by forecasting trends, identifying potential
risks, and understanding probable scenarios. 3. Prescriptive Analytics: - Prescriptive analytics
goes beyond predicting future outcomes; it recommends actions to optimize results. By
considering various possible actions and their potential impacts, prescriptive analytics helps
organizations make strategic decisions that lead to desired outcomes. This classification
provides a framework for organizations to leverage analytics at different levels, from
understanding historical data (descriptive) to predicting future trends (predictive) and
prescribing optimal actions (prescriptive) for improved decision-making.
Explain scenarios where data locality may or may not be optimal. (10 marks) - Data locality
in Hadoop refers to scheduling map tasks on the same server where the input data resides
rather than transferring data across nodes. This provides several benefits: • Improves network
bandwidth utilization as data transfer is minimized • Lowers latency for task execution as data
movement not required • Better cluster utilization as tasks are distributed based on data
location • Enables scaling out linearly by adding nodes without network bottlenecks. However,
strict data locality is not always optimal. Scenarios where data locality may not help: •
Unbalanced or skewed data distribution can lead to uneven cluster utilization • Nodes storing
critical intermediate data become hotspots • Fetching data from remote nodes may be faster
than waiting for a busy node • Some data processing requires cross-rack communication
Hadoop tries to achieve optimized data locality by allowing customizable data locality
relaxation. The scheduler relaxes locality constraints if tasks are waiting for a long time. This
provides a balance between locality and optimal utilization.

Q2) What are the challenges with big data?- •Challenge in incorporating diverse, large
datasets into analytical platforms leading to gaps and inaccurate insights. •Acute shortage of
skilled professionals in Big Data analysis, creating high demand for data scientists. •Ensuring
meaningful insights from Big Data analytics and managing access to relevant departments
pose challenges. •Handling growing volumes of data daily is a constant challenge for
businesses. •Uncertainty in choosing suitable technologies for Big Data analytics without
introducing new problems. •Managing storage of massive data presents challenges in data
lakes/warehouses, leading to quality issues. •Security and privacy concerns arise with the use
of Big Data tools, increasing the risk of data exposure.
Q3) Define big data. Why is big data required? Write a note on data warehouse
environment? Big Data: Big data refers to extremely large and complex datasets that exceed
the capabilities of traditional data processing methods. It is characterized by its volume,
velocity, variety, and sometimes veracity, encompassing vast amounts of information that
traditional systems struggle to handle. Why Big Data is Required: Big data is necessary for in-
depth analysis, informed decision-making, predictive analytics, and gaining a competitive
advantage. It enables organizations to process and derive insights from massive datasets,
providing a comprehensive view of business operations and customer behavior. •Big Data
handles large, diverse datasets for insights and decision-making. •Big Data analytics uncovers
patterns, relationships, and market trends for informed decisions. •Data Warehouse (DW)
consolidates and analyzes corporate information for business decisions. •DW sourced from
operational systems and external data, populated via ETL tools. •Business intelligence
software accesses and analyzes data within the Data Warehouse. •Both Big Data and Data
Warehouse serve organizations in making informed business decisions.
Q4) What are the three characteristics of big data? Explain the differences between BI and
Data Science? Characterized by high-volume, velocity, and variety information assets 1.
Objective: BI: The primary objective of BI is to provide insights into past and current business
performance. It focuses on reporting, querying, and data visualization to support strategic and
operational decision- making. Data Science: Data Science is forward-looking, emphasizing the
discovery of patterns, correlations, and trends in data to make predictions and inform future
strategic decisions. 2. Scope: BI: BI typically deals with structured data, often sourced from
internal databases and transactional systems. It is oriented towards predefined reports and
dashboards. Data Science: Data Science has a broader scope, encompassing both structured
and unstructured data. It involves exploring diverse datasets, including text, images, and social
media, and uses advanced analytical techniques. 3. Methodology: BI: BI follows a structured
and predefined approach. It focuses on key performance indicators (KPIs), standardized
reporting, and delivering actionable insights based on historical and current data. Data
Science: Data Science adopts an exploratory and iterative approach. It involves hypothesis
testing, experimentation, and the development of predictive models to uncover insights and
patterns in data. 4. Tools and Techniques: BI: BI tools are designed for data visualization,
reporting, and simple analytics. SQL queries and predefined metrics are commonly used for
analysis. Data Science: Data Science employs a more extensive range of tools and techniques,
including statistical modeling, machine learning algorithms, and programming languages such
as Python and R. It involves creating and deploying complex models to handle predictive
analytics. 5. Decision-Making Horizon: BI: BI supports short to medium-term decision-making
by providing insights into current and historical performance. Data Science: Data Science
supports both short-term and long-term decision-making by forecasting future trends and
outcomes.
Q5) Describe the current analytical architecture for data scientists? - The current analytical
architecture for data scientists typically revolves around a cloud-based ecosystem. It involves
scalable storage solutions such as data lakes, allowing the storage of vast and diverse datasets.
Data processing and transformation are facilitated by distributed computing frameworks like
Apache Spark. Advanced analytics and machine learning models are developed using
programming languages such as Python and R, with frameworks like TensorFlow and PyTorch
gaining prominence. Containerization technologies like Docker facilitate seamless deployment
of models. Data visualization tools like Tableau or Power BI are integrated for communicating
insights effectively. This architecture ensures flexibility, scalability, and accessibility,
empowering data scientists to analyze, model, and visualize data efficiently.
Q6) What are the key roles for the New Big Data Ecosystem? 1.Data Engineer: - Responsible
for designing, constructing, and maintaining the architecture that enables the processing of
large volumes of data. They develop data pipelines, ensuring efficient data ingestion, storage,
and retrieval. 2.Data Scientist:- Focuses on analyzing and interpreting complex datasets to
extract valuable insights. Data scientists utilize statistical methods, machine learning, and
predictive modeling to derive actionable information from big data. 3. Data Analyst:
Examines and interprets data trends, patterns, and correlations to provide insights that
support business decision-making. Data analysts play a crucial role in transforming raw data
into understandable and actionable information. 4. Machine Learning Engineer: - Specialized
in designing, implementing, and deploying machine learning models within the big data
ecosystem. They work closely with data scientists to integrate and operationalize machine
learning solutions. 5.Cloud Architect: - Designs and oversees the cloud infrastructure to
support the storage, processing, and analysis of big data. Cloud architects ensure the
scalability, reliability, and security of the entire big data ecosystem. These roles collaboratively
contribute to the effective functioning of the new big data ecosystem, facilitating the
extraction of meaningful insights and driving data-driven decision-making within
organizations.
Q7) What are key skill sets and behavioural characteristics of a data scientist? 1. Analytical
Skills: - Proficiency in analyzing complex datasets, identifying patterns, and extracting valuable
insights using statistical methods and machine learning algorithms. 2. Programming
Proficiency: - Strong programming skills, particularly in languages like Python or R, for data
manipulation, statistical analysis, and the development of machine learning models. 3.
Domain Knowledge: - A solid understanding of the specific industry or domain in which the
data scientist operates, enabling them to contextualize analyses and provide more relevant
insights. 4.Communication Skills: - Effective communication skills to convey complex findings
and insights to non-technical stakeholders. Data scientists should be able to articulate their
results in a clear and understandable manner. 5. Curiosity and Problem-Solving Attitude: - A
natural curiosity and a problem-solving mindset, encouraging data scientists to explore data,
ask relevant questions, and derive actionable solutions from the information at hand.
Q8) What is big data analytics? Explain in detail with its example. - Big Data Analytics: Big
data analytics refers to the process of examining and uncovering meaningful patterns, trends,
and insights from large and complex datasets, commonly known as big data. It involves using
advanced analytical techniques, including statistical analysis, machine learning, and data
mining, to extract valuable information that can guide business decision-making and strategic
planning. Example: Consider a retail company that collects vast amounts of data from various
sources, including sales transactions, customer interactions, and social media. Through big
data analytics, the company can analyze this data to gain insights into customer preferences,
buying patterns, and market trends. For instance, the analytics may reveal that certain
products are often purchased together, allowing the company to optimize product placement
in stores. Additionally, sentiment analysis on social media data may provide insights into
customer opinions and preferences. By leveraging big data analytics, the retail company can
make informed decisions on inventory management, marketing strategies, and customer
engagement, ultimately improving overall business performance.
Q10) What is big data analytics? Also write and explain importance of big data. - Big Data
Analytics: Big data analytics refers to the process of examining and uncovering meaningful
patterns, trends, and insights from large and complex datasets, commonly known as big data.
It involves using advanced analytical techniques, including statistical analysis, machine
learning, and data mining, to extract valuable information that can guide business decision-
making and strategic planning. The importance of Big Data are: • Big Data's importance lies in
efficient utilization, not just the volume of data. • Enables cost savings through tools like
Hadoop and Cloud-Based Analytics. • Time reductions achieved with high-speed tools for quick
data analysis and decision-making. • Enhances understanding of market conditions through
customer behavior analysis. • Controls online reputation via sentiment analysis tools. • Utilizes
Big Data Analytics for customer acquisition, retention, and loyalty. • Solves advertising
problems and provides marketing insights through data observation.
Q11) Write a short note on data science and data science process? Data Science: Data science
is a multidisciplinary field that utilizes scientific methods, processes, algorithms, and systems
to extract knowledge and insights from structured and unstructured data. It combines
expertise from various domains, including statistics, mathematics, computer science, and
domain- specific knowledge, to uncover patterns, trends, and valuable information from large
datasets. Data Science Process: The data science process involves several iterative steps: •
Problem Definition: Clearly define the problem or question at hand and establish objectives
that data science aims to address. • Data Collection: Gather relevant data from various
sources, ensuring it is comprehensive and representative of the problem domain. • Data
Cleaning and Preprocessing: Cleanse and preprocess the data to handle missing values,
outliers, and ensure it is in a suitable format for analysis. • Exploratory Data Analysis (EDA):
Conduct exploratory data analysis to understand the characteristics of the data, identify
patterns, and gain initial insights. • Feature Engineering: Create new features or transform
existing ones to enhance the predictive power of models. • Model Development: Utilize
statistical and machine learning techniques to build models that address the defined problem
or question. • Model Evaluation: Evaluate the performance of models using appropriate
metrics to ensure their effectiveness. • Model Deployment: Deploy the model into a
production environment, making it accessible for real-world applications. • Monitoring and
Maintenance: Continuously monitor the model's performance in a live environment and make
necessary adjustments to ensure its accuracy and relevance over time. The data science
process is iterative, with feedback loops allowing for refinement and improvement at each
stage. This structured approach enables data scientists to derive actionable insights and
solutions from data, contributing to informed decision-making in various domains.
Q12) Write a short note on Soft state eventual consistency? - Soft state eventual consistency
is a concept in distributed computing that aims to strike a balance between system availability
and data consistency. In systems employing eventual consistency, the goal is to ensure that,
given enough time and no new updates, all replicas of a piece of data will converge to the same
state. Key Points: 1. Eventual Consistency:- Soft state eventual consistency acknowledges that,
in distributed systems, achieving immediate consistency across all nodes may not always be
practical or efficient. 2. Trade-off between Consistency and Availability: - It represents a trade-
off between ensuring data consistency at all times (strong consistency) and allowing systems
to remain available for read and write operations even during network partitions or failures.
3. Acceptance of Temporary Inconsistency: - Soft state eventual consistency accepts the
existence of temporary inconsistencies among replicas but expects that, over time and without
further updates, all replicas will converge to a consistent state. 4. Use Cases: - This approach
is often suitable for systems where real-time consistency is not critical, and temporary
divergences in data state can be tolerated for the sake of system availability and fault
tolerance. 5. Implementation:- Soft state eventual consistency is implemented by allowing
replicas to continue serving read and write requests independently, with eventual
reconciliation to synchronize data across nodes over time. In summary, soft state eventual
consistency is a pragmatic approach in distributed systems that acknowledges the challenges
of immediate consistency and prioritizes system availability while allowing for eventual
convergence of data state across all replicas.
Q13) What are different phases of the Data Analytics Lifecycle? Explain each in detail? - 1.
Business Understanding: - In this initial phase, the focus is on defining the business problem
or question that data analytics aims to address. Understanding the business context, goals,
and stakeholders' requirements is crucial to guide the entire analytics process. 2. Data
Collection and Preparation: - This phase involves gathering relevant data from various sources.
Data is then cleaned, preprocessed, and transformed to ensure quality and compatibility with
analytics tools. Handling missing values, outliers, and formatting issues is essential during this
stage. 3. Exploratory Data Analysis (EDA):- EDA involves analyzing and visualizing the data to
- gain insights into its characteristics, patterns, and potential relationships. Techniques such as
statistical summaries, visualizations, and correlation analyses help in understanding the
dataset's structure and informing subsequent analytical decisions. 4. Model Development:- In
this phase, statistical and machine learning models are built to address the defined business
problem. This involves selecting appropriate algorithms, splitting the data into training and
testing sets, and tuning model parameters to achieve optimal performance. 5. Model
Evaluation and Validation:- The developed models are rigorously evaluated using various
metrics to assess their performance and generalization to new, unseen data. Validation
ensures that the models are reliable and capable of providing accurate predictions or
classifications. 6. Deployment:- Once a satisfactory model is obtained, it is deployed into a
production environment. This involves integrating the model into the operational workflow,
making predictions on new data, and ensuring that the model's output aligns with the
intended business objectives. 7. Monitoring and Maintenance:- Continuous monitoring of the
deployed model is essential to ensure its ongoing accuracy and relevance. If the model's
performance degrades over time, adjustments and updates may be necessary. Monitoring
helps maintain the model's effectiveness in a changing business environment. 8.
Communication of Results:- Throughout the entire lifecycle, effective communication of
results is crucial. Findings and insights are communicated to stakeholders through reports,
visualizations, and presentations, ensuring that the analytics outcomes are comprehensible
and actionable.This lifecycle is iterative, allowing for refinements, adjustments, and additional
analyses based on feedback and changing business requirements. It forms a structured
framework for conducting data analytics projects effectively.
Q14) Explain in detail the Hadoop architecture and how the various components achieve
reliable, distributed processing of big data. (10 marks)- The core Hadoop architecture consists
of: • HDFS (Hadoop Distributed File System): HDFS has a master-slave architecture with a
Namenode (master) and Datanodes (slaves). Namenode manages the filesystem metadata and
datanodes store the actual data in blocks. The data is replicated across datanodes for reliability.
• YARN (Yet Another Resource Negotiator): YARN has a Resource Manager (master) which
manages resources and schedules jobs. The NodeManagers (slaves) run on all nodes in the
cluster to launch and monitor containers running tasks. • MapReduce: MapReduce utilizes
YARN to execute data processing tasks in parallel. The developer creates map and reduce
functions to process data. Map tasks are distributed to nodes and run parallelly on the local
data. The framework sorts the outputs which are then input to the reduce tasks. This
architecture provides reliability through replication and parallel processing to achieve
tremendous scalability. Data locality optimizes task scheduling to move computation closer to
the data. Hadoop can scale linearly by simply adding commodity hardware. Failures are
automatically handled by rerunning failed tasks. This enables Hadoop to handle very large
datasets across hundreds or thousands of nodes.
Q16) Analyse the small file problem in HDFS and discuss various solutions with their pros
and cons. (10 marks)- The small file problem in HDFS arises when storing a large number of
small files. Challenges with small files: • Namenode memory overhead to store file metadata
• Low storage utilization due to lots of under-filled blocks • Excessive seeks and network
chatter during file access Solutions to the small file problem include: • Sequence files to pack
small files into larger files. Con is code changes needed. • HAR files to archive small files. Con
is accessing archives needs unpacking. • Compression to reduce storage usage. Con is
compressed files use CPU. • In-memory filesystems like Alluxio to cache files. Con is memory
overhead. • Hive/HBase to store small records in own system. Integration complexity. •
Increase block size to pack more small files. Con is reduced parallelism. There is no single
optimal solution. Based on access patterns, storage overhead, and need for native HDFS, an
appropriate technique should be used to optimize handling of small files.
Q17) Explain the key characteristics of NoSQL databases and how they differ from traditional
RDBMS. (10 marks) - NoSQL databases are designed for storing and managing large volumes
of structured, semi- structured and unstructured data. Key characteristics: • Flexible schema -
No fixed schema allows easy evolution of data models. • Scalability - Built with horizontal
scaling to distribute data across servers. • Eventual consistency - Favors availability over ACID
guarantees for high performance. • Non-relational - Uses document, key-value, graph or wide
column storage models rather than relational tables. • High performance - In-memory caching
and distributed architecture allow high throughput and low latency. In contrast, RDBMS have
fixed schemas, vertical scaling, strong consistency, and table- relational models suitable for
structured transactional data. NoSQL is better suited for web, mobile and Big Data applications
dealing with variably structured data at scale.
Q18) Compare and contrast Graph databases and Document databases - their data models,
query languages, and use cases. (10 marks) - Graph databases like Neo4j use nodes and edges
as the data model to represent networked data. Relationships are first class citizens enabling
fast traversal across networks. Query languages are based on graph patterns and traversal. Use
cases include social networks, knowledge graphs, IoT networks. Document databases like
MongoDB use documents (JSON/BSON) as the primary data model. Documents allow nested
structures without fixed schema. Query languages provide declarative ways to filter and
aggregate documents. Use cases include content management, ecommerce, web apps. While
graphs focus on relationships, documents focus on structure within records. Graphs allow
efficient traversal whereas documents make it easy to query nested data. Graphs are suited
for interconnected data while documents suit applications with semi-structured records.
Q19) Discuss the trade-offs between consistency, availability and partition tolerance in
distributed NoSQL systems. Explain with examples. (10 marks) - According to the CAP
theorem, distributed systems can only guarantee two of the following: • Consistency - all
nodes see the latest updated data • Availability - guarantees response despite failures •
Partition tolerance - system works despite arbitrary partitioning. Most NoSQL systems favor
availability and partition tolerance over strong consistency. For example, Cassandra offers
tunable consistency levels like ANY and QUORUM to allow flexible consistency guarantees.
DynamoDB and MongoDB also relax consistency for high availability and low latency.
Applications have to account for inconsistency - like reading stale data or merging conflicting
writes. However, for large scale web and mobile apps, availability and performance often
trump strong consistency requirements. The trade-offs allow NoSQL systems to scale, perform
and remain highly available at internet scale.
Q20) What are the characteristics of big data and what are the challenges in analyzing them
using traditional statistical techniques? How do machine learning techniques help in big data
analytics? - Big data refers to large, complex datasets characterized by 3V - Volume, Velocity
and Variety. Traditional statistical methods are inadequate for big data due to: • Scalability
issues for large volumes of data • Difficulty in integrating heterogeneous and unstructured
data types • Requirement of complex modelling and sampling techniques Machine learning
provides solutions through: • Distributed learning algorithms like MapReduce that scale
across clusters • Ability to uncover patterns from all kinds of unstructured data • Techniques
like neural networks that learn complex models from raw data. Thus, machine learning,
especially deep learning, provides the capabilities to effectively process big data and drive
predictive analytics.
Q21) Explain the role of dimensionality reduction in analyzing high dimensional big data
using machine learning. Discuss various techniques, their working and comparative
advantages.- Real-world big data is often characterized by very high dimensionality with
numerous features. Analysing such data directly leads to computational complexity and
overfitting. Dimensionality reduction transforms data to lower dimensions while preserving
useful information. Key techniques: • Feature selection - identify and retain most salient
features based on correlations. Works well for sparse data. • Feature extraction - combine
features using methods like PCA. Useful for dense data. • Autoencoders - neural network-
based models that compress input into lower dimensional code. Captures non-linear
relationships. Dimensionality reduction removes noise, collapses redundant information and
results in more robust models. The choice of technique depends on data characteristics and
modelling objectives.
Q22) What are the additional steps required in the machine learning workflow for analyzing
streaming big data? Explain with examples. -The lifecycle for analysing streaming data
involves additional considerations: • Preprocess and extract features from continuously
incoming data streams • Train models on smaller data batches in real-time rather than full
datasets • Adaptive learning algorithms like online learning to update models incrementally •
Predictions need continuous evaluation and model retraining to handle concept drift • Store
only summary statistics rather than full data history. For instance, real-time anomaly
detection on user activity streams. The model trains on latest batches, makes predictions,
measures error, retrain to address drift. Summary stats like rolling averages are tracked to
detect anomalies. The workflow must operate continuously at low latency on streaming data.
Q23) A. Explain the evolution of big data processing frameworks from MapReduce to current
platforms. B. Discuss how YARN improved resource management over classic MapReduce.
C. Explain how Hive and Impala enabled SQL-like data warehousing capabilities over
Hadoop. D. Discuss the role of Cloudera and Snowflake as data platform vendors. Answer
A - MapReduce provided the initial breakthrough in distributed processing of big data on
commodity hardware. However, it had limitations like lack of resource management and reuse
across jobs. B - YARN (Yet Another Resource Negotiator) opened up Hadoop for diverse
workloads beyond MapReduce. It introduced a Resource Manager to allocate resources across
clusters and Application Master to negotiate resources for each job. This enabled running
multiple applications on Hadoop without contention.C - Hive provides a SQL interface and data
warehouse capabilities on top of Hadoop. It converts SQL queries into MapReduce jobs. Impala
made SQL query processing on Hadoop faster through a distributed query engine. This enabled
analysts to leverage Hadoop using familiar SQL syntax. D - Cloudera offers a robust open-source
based Hadoop distribution along with proprietary management and governance tools. It aims
to make Hadoop usable in enterprise environments with better security, automation and
optimization. Snowflake provides a cloud data platform with separation of storage and
compute. It delivers data warehouse capabilities built specially for the cloud. Snowflake allows
enterprises to leverage cloud infrastructure for analytics at scale. The evolution from
MapReduce to YARN to SQL interfaces shows how big data processing frameworks have
matured over time by becoming more versatile, scalable and easier to use for enterprises.
Vendors like Cloudera and Snowflake have further enabled Hadoop and cloud adoption in
mainstream business analytics.