Scribd Logo
Upload

Welcome to Scribd!

  • 28 views

    The Hadoop Ecosystem: So Much Free Stuff!

    Uploaded by

    Kosuru ratnasai
    The Hadoop ecosystem consists of several layers including core frameworks like HDFS, YARN, and MapReduce as well as higher-level tools for structured querying (Hive), dataflows (Pig), graph processing (Giraph), stream processing (Storm, Flink), NoSQL databases (HBase, Cassandra, MongoDB) and coordination (Zookeeper). These open-source tools provide scalable storage, scheduling, programming models and specialized processing and can be downloaded separately or as pre-built images to build flexible big data systems.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Flag for inappropriate content

    You might also like

    The Hadoop Ecosystem: So Much Free Stuff!

    Uploaded by

    Kosuru ratnasai
    28 views21 pages
    The Hadoop ecosystem consists of several layers including core frameworks like HDFS, YARN, and MapReduce as well as higher-level tools for structured querying (Hive), dataflows (Pig), graph processing (Giraph), stream processing (Storm, Flink), NoSQL databases (HBase, Cassandra, MongoDB) and coordination (Zookeeper). These open-source tools provide scalable storage, scheduling, programming models and specialized processing and can be downloaded separately or as pre-built images to build flexible big data systems.

    Original Description:

    HadoopEcosystem

    Original Title

    HadoopEcosystem

    Copyright

    © © All Rights Reserved

    Available Formats

    PDF, TXT or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    Report this Document
    The Hadoop ecosystem consists of several layers including core frameworks like HDFS, YARN, and MapReduce as well as higher-level tools for structured querying (Hive), dataflows (Pig), graph processing (Giraph), stream processing (Storm, Flink), NoSQL databases (HBase, Cassandra, MongoDB) and coordination (Zookeeper). These open-source tools provide scalable storage, scheduling, programming models and specialized processing and can be downloaded separately or as pre-built images to build flexible big data systems.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Flag for inappropriate content
    Download as pdf or txt
    28 views21 pages

    The Hadoop Ecosystem: So Much Free Stuff!

    Uploaded by

    Kosuru ratnasai
    The Hadoop ecosystem consists of several layers including core frameworks like HDFS, YARN, and MapReduce as well as higher-level tools for structured querying (Hive), dataflows (Pig), graph processing (Giraph), stream processing (Storm, Flink), NoSQL databases (HBase, Cassandra, MongoDB) and coordination (Zookeeper). These open-source tools provide scalable storage, scheduling, programming models and specialized processing and can be downloaded separately or as pre-built images to build flexible big data systems.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Flag for inappropriate content
    Download as pdf or txt
    of 21

    The Hadoop Ecosystem:

    So much free stuff!


    After this video you will be able to..
    • Differentiate the major layers in the Hadoop
    ecosystem

    • Recognize key tools of the Hadoop


    ecosystem including HDFS, YARN, and
    MapReduce
    Yahoo created
    Hadoop in 2005
    More Big Data frameworks released
    Now there’s over a 100!
    Layer Diagram
    D

    B C

    A
    One possible layer diagram for Hadoop

    Hive Pig

    Giraph

    Spark
    Storm

    Flink
    MapReduce

    HBase

    Cassandra

    MongoDB
    Zookeeper

    YARN

    HDFS
    One possible layer diagram for Hadoop
    Higher levels:
    Interactivity

    Hive Pig

    Giraph

    Spark
    Storm

    Flink
    MapReduce

    HBase

    Cassandra

    MongoDB
    Zookeeper

    YARN

    HDFS

    Lower levels:
    Storage and scheduling
    Distributed file system as foundation
    Scalable storage
    Fault tolerance

    Hive Pig Giraph

    Spark
    Storm

    Flink
    MapReduce

    HBase

    Cassandra
    Zookeeper

    MongoDB
    YARN

    HDFS
    Flexible scheduling and
    resource management

    YARN schedules jobs on


    Hive >Pig40,000 servers at Yahoo
    Giraph

    Spark
    Storm

    Flink
    MapReduce

    HBase

    Cassandra
    Zookeeper

    MongoDB
    YARN

    HDFS
    Simplified programming model

    Map  apply()
    Reduce  summarize()
    Hive Pig Giraph

    Spark
    Storm

    Flink
    MapReduce

    HBase

    Cassandra
    Zookeeper

    MongoDB
    YARN
    Google used MapReduce
    for indexing web sites
    HDFS
    Higher-level programming models
    Pig = dataflow scripting
    Hive = SQL-like queries

    Hive Pig Giraph

    Spark
    Storm

    Flink
    MapReduce

    HBase

    Cassandra
    Zookeeper

    Pig created at Yahoo,

    MongoDB
    YARN
    Hive created at Facebook
    HDFS
    Specialized models
    for graph processing
    Giraph used by Facebook
    to analyze social graphs
    Hive Pig Giraph

    Spark
    Storm

    Flink
    MapReduce

    HBase

    Cassandra
    Zookeeper

    MongoDB
    YARN

    HDFS
    Real-time and
    in-memory processing
    In-memory  100x faster
    for some tasks

    Hive Pig Giraph

    Spark
    Storm

    Flink
    MapReduce

    HBase

    Cassandra
    Zookeeper

    MongoDB
    YARN

    HDFS
    NoSQL for non-files
    Key-values
    Sparse tables
    Hive Pig Giraph

    Spark
    Storm

    Flink
    MapReduce

    HBase

    Cassandra
    Zookeeper

    YARN for Facebook’s

    MongoDB
    HBase used
    Messaging Platform
    HDFS
    Zookeeper for management
    Synchronization
    Configuration
    High-availability
    Hive Pig Giraph

    Spark
    Storm

    Flink
    Created by Yahoo to wrangle
    MapReduce

    HBase

    Cassandra
    Zookeeper

    MongoDB
    services
    YARN named after animals
    HDFS
    All these tools are open-source
    All these tools are open-source

    Large community
    for support
    All these tools are open-source

    Large community
    for support

    Download separately
    or part of pre-built image
    All these tools are open-source

    Large community
    for support

    Download separately
    or part of pre-built image
    Hive Pig

    Giraph

    Spark
    Storm

    Flink
    MapReduce

    HBase

    Cassandra

    MongoDB
    Zookeeper

    YARN

    HDFS

    Growing number of open-source tools

    You might also like