APACHE MESOS
A MODERN OPEN-SOURCE CLUSTER MANAGER FOR
OPTIMIZING RESOURCE UTILIZATION
OUTLINE
• INTRODUCTION TO APACHE MESOS
• FRAMEWORK
• COMPONENTS
• SETUP
• SUMMARY
INTRODUCTION
DESCRIPTION
ARCHITECTURE
INTRODUCTI
ON
WHAT IS APACHE MESOS?
• Open-source cluster manager.
• System resource abstraction from hardware
• Facilitates effective resource isolation and sharing across
distributed applications or frameworks.
• Cross platform.
• Meta scheduler
• Supports Mesos and Docker containers.
• Distributed systems kernel / Datacenter OS.
INTRODUCTI
ON
ARCHITECTURE –
FEATURES
Scales to over 50,000 nodes
Fault Tolerance
Multi Latency
Flexibility
High availability via Zookeeper
Resource Isolation using Docker containers or Linux cgroups
A Master / Slave based system
Web UI for monitoring cluster states
INTRODUCTI
ON
INTRODUCTI
ON
INTRODUCTI
ON
ARCHITECTURE –
OVERVIEW
INTRODUCTI
ON
ARCHITECTURE OVERVIEW
FRAMEWORK
SERVICE SCHEDULING AND FRAMEWORK MANAGEMENT
BIG DATA FRAMEWORKS
FRAMEWORK
S
SERVICE SCHEDULING AND FRAMEWORK
MANAGEMENT
Marathon
– Launch and monitor long-running applications. Controlling a high availability environment,
checking the application’s health, and restarting services if nodes go down.
Chronos
– Distributed, fully fault-tolerant fob scheduler. Can interact with systems even if the Mesos
worker machine does not have the system installed.
Apache Aurora
– Manages long-running services, cron jobs, and ad-hoc jobs.
Consul
– Service Discovery and orchestration
FRAMEWORK
S
BIG DATA FRAMEWORKS
Apache Hadoop
- Distributed Data Processing, MapReduce and Distributed File System
Apache Spark
- Execution engine for large-scale data processing.
Apache Storm
- Real-time distributed stream event-processing engine.
Apache Cassandra
- Open source scalable NoSQL database.
Provides query language Cassandra Query Language or CQL to communicate with database.
COMPONENTS
ZOOKEPER
MARATHON
COMPONENT
S
APACHE ZOOKEEPER
A centralized service for maintaining configuration information, naming, providing distributed
synchronization, and providing group services.
Independent project
Fast and Simple
Central Key/Value Store
Designed for High Availability
Containing additional information like time stamps, versioning
COMPONENT
S
APACHE ZOOKEEPER
High Availability
One Leader
Two or more followers
Quorum
Note: A minimum of 3 servers are required to cater for any node failure.
COMPONENT
S
APACHE ZOOKEEPER
High Availability
Hierarchical Namespace
Z-Nodes
Nodes identified by Path
Ephemeral Nodes
Leader Election
Conditional updates and watches
Atomic Transactions
Takes hand in everywhere Mesos needs to store and call up distributed dynamic information
- Service Locations
- Service States
- Client Notification
- Leader election for Mesos Masters and Frameworks (e.g. Marathon)
COMPONENT
S
MARATHON
An Apache Mesos framework for container orchestration
- designed for long-running and stateful apps
A meta framework
- Used for managing other Mesos Frameworks (Chronos, Storm,
…)
Container Support
- Mesos Containers (default)
- Docker Constrainers (must be configured)meta-framework
COMPONENT
S
MARATHON
Features
- High Availability
- Constraints
- Service Discovery & Load Balancing
- Health Checking
- Event Subscription
- Marathon UI
- Basic Security (Authorization, SSL, …)
- User Interfaces for managing services (start, stop, check, …)
SETUP
INSTALLATION
CONFIGURATION
REQUIREMENTS
SETUP
INSTALLATION
http://mesos.apache.org/gettingstarted
Different between CentOS 6 / 7
Different Requirement (OS Packages, Libraries) with Zookeeper
SETUP
CONFIGURATION
http://mesos.apache.org/documentation/lates/configuration
Provide parameters by config file or as startup parameter
Required parameters (M: on Master, A: on Agent)
SETUP
CONFIGURATION
Requirements
Apache Mesos 1.1.1+
Apache Zookeeper
JDK 1.8+
Default JVM set to Java 8
Installation
https://mesosphere.github.io/marathon/docs/
Download/unpack Kit (tar)
Platform Use & HA Container Ease of Use & Communit Bare Metal Support for
Compatibility with Capabilit Orchestration Setup Cost y Support & Hypervisor
COMPARISON TABLE
Database Service y /Ecosystem Setup s
Apache Large-scale distributed High Built-in support Auto-scalable Large, active, Yes, LOW
apps, resource integrates with (Medium), extensive Medium
Mesos abstraction Kubernetes, Docker flexible,
Swarm, Apache Aurora User-friendly
High
HIGH
AppScale Run Google App Engine High NOT primarily focused Auto-scalable Large, active Yes, Medium
apps on any on container (High), flexible, Medium
infrastructure orchestration User-friendly
LOW (Relatively) Low
Cloud Stack For building & managing High Primarily focuses on VM Auto-scalable Active, Yes, High
cloud environments orchestration (HIGH), User- extensive Medium/
friendly Difficult
High
Medium
Eucalyptus On-premises private Medium Primarily focuses on VM Auto scalable SMALL, Yes, Medium
cloud, compatibility with orchestration (Medium), INACTIVE Medium
AWS APIs flexible, User-
friendly
Medium
Medium
Open Stack For building & managing High Supports container Auto-scalable Largest, active, Yes, High
cloud environments orchestration through (High), extensive Difficult
projects like Magnum NOT Flexible
High and Kubernetes
HIGH
Private cloud, traditional Medium Support Containers Auto-scalable Large, active, Yes, High
THANK YOU
KOFI ABAASAH
