APACHE MESOS

A MODERN OPEN-SOURCE CLUSTER MANAGER FOR

OPTIMIZING RESOURCE UTILIZATION
OUTLINE
• INTRODUCTION TO APACHE MESOS
• FRAMEWORK
• COMPONENTS
• SETUP
• SUMMARY
INTRODUCTION
DESCRIPTION
ARCHITECTURE
INTRODUCTI
ON

WHAT IS APACHE MESOS?

• Open-source cluster manager.

• System resource abstraction from hardware

• Facilitates effective resource isolation and sharing across

distributed applications or frameworks.

• Cross platform.

• Meta scheduler

• Supports Mesos and Docker containers.

• Distributed systems kernel / Datacenter OS.

INTRODUCTI
ON

ARCHITECTURE –
FEATURES
 Scales to over 50,000 nodes
 Fault Tolerance
 Multi Latency
 Flexibility
 High availability via Zookeeper
 Resource Isolation using Docker containers or Linux cgroups
 A Master / Slave based system
 Web UI for monitoring cluster states
INTRODUCTI
ON
INTRODUCTI
ON
INTRODUCTI
ON

ARCHITECTURE –
OVERVIEW
INTRODUCTI
ON

ARCHITECTURE OVERVIEW
 FRAMEWORK
SERVICE SCHEDULING AND FRAMEWORK MANAGEMENT
BIG DATA FRAMEWORKS
FRAMEWORK
S

SERVICE SCHEDULING AND FRAMEWORK

MANAGEMENT
 Marathon
– Launch and monitor long-running applications. Controlling a high availability environment,
checking the application’s health, and restarting services if nodes go down.

 Chronos
– Distributed, fully fault-tolerant fob scheduler. Can interact with systems even if the Mesos
worker machine does not have the system installed.

 Apache Aurora
– Manages long-running services, cron jobs, and ad-hoc jobs.

 Consul
– Service Discovery and orchestration
FRAMEWORK
S

BIG DATA FRAMEWORKS

 Apache Hadoop
 - Distributed Data Processing, MapReduce and Distributed File System

 Apache Spark
 - Execution engine for large-scale data processing.

 Apache Storm
 - Real-time distributed stream event-processing engine.

 Apache Cassandra
 - Open source scalable NoSQL database.
 Provides query language Cassandra Query Language or CQL to communicate with database.
COMPONENTS
ZOOKEPER
MARATHON
COMPONENT
S

APACHE ZOOKEEPER
 A centralized service for maintaining configuration information, naming, providing distributed
synchronization, and providing group services.

 Independent project

 Fast and Simple

 Central Key/Value Store

 Designed for High Availability

 Containing additional information like time stamps, versioning

COMPONENT
S

APACHE ZOOKEEPER
 High Availability
 One Leader
 Two or more followers
 Quorum

Note: A minimum of 3 servers are required to cater for any node failure.
COMPONENT
S

APACHE ZOOKEEPER
 High Availability
 Hierarchical Namespace
 Z-Nodes
 Nodes identified by Path
 Ephemeral Nodes
 Leader Election
 Conditional updates and watches
 Atomic Transactions
 Takes hand in everywhere Mesos needs to store and call up distributed dynamic information
 - Service Locations
 - Service States
 - Client Notification
 - Leader election for Mesos Masters and Frameworks (e.g. Marathon)
COMPONENT
S

MARATHON
 An Apache Mesos framework for container orchestration
 - designed for long-running and stateful apps

 A meta framework
 - Used for managing other Mesos Frameworks (Chronos, Storm,
…)

 Container Support
 - Mesos Containers (default)
 - Docker Constrainers (must be configured)meta-framework
COMPONENT
S

MARATHON
 Features
 - High Availability

 - Constraints

 - Service Discovery & Load Balancing

 - Health Checking

 - Event Subscription

 - Marathon UI

 - Basic Security (Authorization, SSL, …)

 - User Interfaces for managing services (start, stop, check, …)

SETUP
INSTALLATION
CONFIGURATION
REQUIREMENTS
SETUP

INSTALLATION
 http://mesos.apache.org/gettingstarted

 Different between CentOS 6 / 7

 Different Requirement (OS Packages, Libraries) with Zookeeper

SETUP

CONFIGURATION
 http://mesos.apache.org/documentation/lates/configuration

 Provide parameters by config file or as startup parameter

 Required parameters (M: on Master, A: on Agent)

SETUP

CONFIGURATION

 Requirements
 Apache Mesos 1.1.1+
 Apache Zookeeper
 JDK 1.8+
 Default JVM set to Java 8

 Installation
 https://mesosphere.github.io/marathon/docs/
 Download/unpack Kit (tar)
Platform Use & HA Container Ease of Use & Communit Bare Metal Support for
Compatibility with Capabilit Orchestration Setup Cost y Support & Hypervisor
COMPARISON TABLE
Database Service y /Ecosystem Setup s

Apache Large-scale distributed High Built-in support Auto-scalable Large, active, Yes, LOW
apps, resource integrates with (Medium), extensive Medium
Mesos abstraction Kubernetes, Docker flexible,
Swarm, Apache Aurora User-friendly
High
HIGH
AppScale Run Google App Engine High NOT primarily focused Auto-scalable Large, active Yes, Medium
apps on any on container (High), flexible, Medium
infrastructure orchestration User-friendly

LOW (Relatively) Low

Cloud Stack For building & managing High Primarily focuses on VM Auto-scalable Active, Yes, High
cloud environments orchestration (HIGH), User- extensive Medium/
friendly Difficult
High
Medium
Eucalyptus On-premises private Medium Primarily focuses on VM Auto scalable SMALL, Yes, Medium
cloud, compatibility with orchestration (Medium), INACTIVE Medium
AWS APIs flexible, User-
friendly
Medium
Medium
Open Stack For building & managing High Supports container Auto-scalable Largest, active, Yes, High
cloud environments orchestration through (High), extensive Difficult
projects like Magnum NOT Flexible
High and Kubernetes
HIGH
Private cloud, traditional Medium Support Containers Auto-scalable Large, active, Yes, High
THANK YOU
KOFI ABAASAH