⛪

Assignment - 3 (Big Data)

Q-1, Define Command line interface using HDFS files and give a brief note on Hadoopspecific file system types
and HDFS commands.
Ans, A Command Line Interface (CLI) using HDFS files refers to a method of interacting with the Hadoop
Distributed File System (HDFS) through text-based commands entered in a terminal or console window.

1. Connecting to HDFS: Before interacting with HDFS via CLI, you typically need to connect to the Hadoop
cluster where HDFS is running. This involves using commands like ssh or hadoop fs to access the filesystem.

2. Navigating the File System: Once connected, you can navigate the file system using commands similar to
those used in traditional filesystems, such as ls to list files, cd to change directories, pwd to print the current
directory, etc.

Hadoop-Specific File System Types:

1. HDFS (Hadoop Distributed File System): The primary distributed file system used by Hadoop. It is designed
to store large datasets reliably and provides high throughput access to the data.

2. Local File System (file://): Though not distributed, Hadoop can also interact with data stored on local file
systems. This is often used in local development environments.

Common HDFS Commands:

1. ls : List files and directories in the current directory.

2. mkdir : Create a new directory.

3. cp : Copy files from source to destination.

Q2. Discuss the following terms

a. Streaming information access.
b. Low latency information access.
c. Rest and thrift
d. Org.apcahe.hadoop.io.package

Ans, a. Streaming Information Access: Streaming allows real-time processing of data as it arrives, enabling
continuous analysis without storing the entire dataset locally, crucial for real-time analytics and event processing.

b. Low Latency Information Access: Low latency minimizes the delay between data request and retrieval, vital for
time-sensitive applications such as financial trading and real-time decision-making systems.
c. REST and Thrift: REST is an architectural style for designing networked applications, emphasizing stateless
communication over standard HTTP methods; Thrift is a framework for building scalable cross-language
services, facilitating efficient communication between different systems using a simple interface definition
language.
d. org.apache.hadoop.io package: Part of Apache Hadoop, it offers classes and interfaces for input/output
operations, serialization, and data handling, providing a standardized approach for working with data in Hadoop
environments, enhancing interoperability and efficiency.

Q3. Explain HDFS Read and write operations in detail.

Ans. HDFS Write Operation:

1. Client Interaction: The write process begins when a client application interacts with the HDFS cluster to write
data. The client sends a request to the NameNode to write a file.

Assignment - 3 (Big Data) 1

 2. NameNode Processing: The NameNode receives the request and determines the data nodes where the file
will be stored. It then returns the list of DataNodes to the client.

HDFS Read Operation:

1. Client Request: A client application initiates a read operation by sending a request to the NameNode for a
specific file.

2. NameNode Processing: The NameNode receives the request and retrieves metadata about the requested
file, including block locations.

Q4. What are the features of HDFS Interface?

Ans. The Hadoop Distributed File System (HDFS) interface offers several key features that facilitate efficient
storage and processing of large datasets in a distributed environment. Here are some of its notable features:

1. Scalability: HDFS is designed to scale horizontally, allowing it to handle petabytes of data across thousands
of commodity hardware nodes. It can seamlessly accommodate growing storage needs by adding more
nodes to the cluster.

2. Fault Tolerance: HDFS achieves fault tolerance through data replication. It automatically creates multiple
copies of data blocks and distributes them across different nodes in the cluster. In case of node failures or
data corruption, HDFS can retrieve the data from replicated copies stored on other nodes.

Q5. List and explain any five essential Hadoop tools with their features.
Ans.

1. HDFS (Hadoop Distributed File System): Provides scalable and fault-tolerant storage for large datasets
across distributed clusters.

2. MapReduce: Framework for parallel processing of big data, enabling scalable computation across Hadoop
clusters.

3. Apache Hive: Data warehouse infrastructure for querying and analyzing large datasets using SQL-like
HiveQL.

4. Apache Spark: Fast and general-purpose distributed computing system with in-memory processing
capabilities for big data.

5. Apache Pig: High-level data flow scripting language and execution framework for parallel data processing in
Hadoop.

Assignment - 3 (Big Data) 2