MariaDB + Docker

Scaling MariaDB
Conclusion
Containers + Databases = Happy Developers
Ephemeral Containers + Databases = DevOps headaches
4 Things you must use to evaluate
Data Redundancy
Dynamic Self Discovery & Cluster formation
Self Healing (as containers enter and leave)
Application Tier discovery of Database Cluster
Part One
Heres another nice mess
you have got me into
Laurel & Hardy circa 1929
Existing Deployment Models Are Broken

1. Development 2. Test 3. Stage / Production

Version
control

Developer QA / QE Sysadmin
 Architectures Are Complex & Static

Pricing / Real-time Real-time Streaming

Inventory / Decisioning Consumer Data
Billing facing Challenges
Orchestration
Data Warehouse Velocity
Data Lake
Caching Layer

Maintainability
Durability

Operational Database Consistency

Scalability
Cost ($)
Enterprise Environment (Hybrid) Cloud
Legacy Mainframe RDBMS Transactional
Systems
Part Two
Containers
What do Containers give me?
Encapsulation of Dependencies
O/S packages & Patches
Execution environment (e.g. Python 2.7)
Application Code & Dependencies

Process Isolation
Isolate the process from anything else running

Faster, Lightweight virtualization

Containers vs. Virtual Machines

App 1 App 2 App 3

Bins/Libs Bins/Libs Bins/Libs

Guest OS Guest OS Guest OS App 1 App 2 App 3

Bins/Libs Bins/Libs Bins/Libs

Hypervisor Docker Engine

Host Operating System Operating System

Infrastructure Infrastructure
Deployment Simplicity

Build Ship Run

Clustering Distribution
Platform

Plumbing
Runtime Trust

Open Standards
Container Image
run-time spec spec
Dockerfile - Example
FROM python:2.7
ADD . /code
WORKDIR /code

RUN apt-get update && apt-get -y install python-dev libssl-dev

RUN pip install Flask MySQL-python

EXPOSE 5000
CMD python app.py
Open Container Initiative (OCI) Polyglot Vendors

Coalition of industry leaders join

forces to eliminate fragmentation
Form a vendor-neutral, open source governance model under the Linux Foundation
Establish common standards for container format and runtime
Docker donated its container format, runtime and associated specifications
Appoint maintainers for the libcontainer project
 Docker Toolchain in pictures
Docker Swarm
Containers encapsulates
your code, dependencies
Swarm clusters
Containers are run Docker Engines
by Docker Engine

Container
Compose orchestrates
Container deployment

Docker Engine
Machine provisions
Docker Engines
Docker Machine Docker Compose
But Docker evolves: 1.13 / 17.3 GA features

Docker Engine in Swarm mode

Engine based clustering versus Container based clustering
Master based, RAFT for consensus

Docker Stacks
Bundles of Services
Sort of compatible with docker-compose

Docker Services
Image + Configuration
Replicable across the cluster
Scale Up & Down
Part Three
Databases & Docker
Requirements

Data Redundancy
Containers are Ephemeral Need more than one copy of the data

Dynamic Self Discovery & Cluster formation

Need to start and stop Containers when needed
Clusters needs to grow and shrink dynamically

Self Healing
Loss of nodes must not be fatal to the cluster integrity
Addition of nodes must scale capacity

Application Tier discovery of Database Cluster

Automatic discovery of nodes
Automatic routing of requests to the correct nodes
Part Four
MariaDB
MariaDB Portfolio

MARIADB SERVER MARIADB MAXSCALE MARIADB CLUSTER

Enterprise-grade secure, Next-generation database Multi-Master, synchronous
highly available and proxy that manages security, replication - improves
scalable relational database scalability and high availability availability and scales
with a modern, extensible in scale-out deployments reads and writes
architecture
MariaDB MaxScale
MariaDB MaxScale is a next-generation database proxy that manages security,
scalability and high availability in a scale out deployment.

Security Scalability High Availability Data Streaming

Secure database Manage your Ensure uptime Stream transactional
firewall to prevent scaled-out with no single data to data lake for
cyber attacks like SQL infrastructure point of failure real-time analytics
injection and DDoS without changing and minimize
application code downtime
during upgrade
 Application /
App Server
MariaDB Cluster

Load Balancing Multi-Master

and Failover Synchronous replication

Faster Failover
All nodes synchronized,
therefore equal

MariaDB Scale reads and writes

MariaDB MariaDB
Galera Cluster + R/W split routing

MaxScale + Galera
Use Case

Each application server

uses only 1 connection
Max
Scale
MaxScale selects one node
as master and the other
nodes as slaves

If the master node fails,

a new one can be elected
immediately
Part Five
Demo
Demo: Development Through Production

Development
Build & Run an App in Development
Python + MariaDB

Production
Deploy to a Swarm cluster in Production
Scale Web nodes
Add more Web containers behind HAProxy
Database High Availability
Deploy 3 nodes Galera cluster
Deploy 2 node MaxScale
Lets Build An App!

app

Python / Flask

Development
Then Scale in Production...

HAProxy
app Virtual
IP

app1 app2 app3 app4 appN

Virtual
IP
MaxScale 1 MaxScale 2

1 2 3
Development Production
Demo 1
Build an Application
Dockerfile
FROM python:2.7
RUN apt-get update && apt-get -y install python-dev libssl-dev
WORKDIR /app

RUN pip install Flask MySQL-python

ADD . /app
EXPOSE 5000

CMD ["python", "app.py"]

docker-compose.yml
services:
web:
build: .
ports:
- "5000:5000"
links:
- mariadb
hostname: dev.myapp.com
environment:
APP_MARIADB_HOST: dev_mariadb_1
APP_PASSWORD: foo
mariadb:
image: mariadb:10.1
environment:
MYSQL_ROOT_PASSWORD: foo
Roll The Application Behind Haproxy

HAProxy
app Virtual
IP

app1

Virtual
IP
MaxScale 1

1
Development Production
Scale the Application Tier

HAProxy
app Virtual
IP

app1 app2 app3 app4 appN

Virtual
IP
MaxScale 1 MaxScale 2

1 2 3
Development Production
 Docker Networking
Docker Host (swarm-0) Docker Host (swarm-1)

HAProxy App
Container Container
Endpoint Endpoint Endpoint

front Network

Docker Host (swarm-2) Docker Host (swarm-3)

MaxScale MariaDB
Container Container

Endpoint Endpoint

back Overlay Network

Docker Networking
$ docker network create -d overlay --attachable --opt encrypted myapp_back

$ cat docker-compose.stack.yml
...
networks:
front:
back:
external:
name: myapp_back
Demo 2
Scale Web Tier
Secrets Opppsss
services:
web:
build: .
ports:
- "5000:5000"
links:
- mariadb
hostname: dev.myapp.com
environment:
APP_MARIADB_HOST: dev_mariadb_1
APP_PASSWORD: foo
mariadb:
image: mariadb:10.1
environment:
MYSQL_ROOT_PASSWORD: foo
Docker secrets
$ more ./app_password.txt
appfoo

$ cat docker-compose.stack.yml
...
secrets:
app_password:
file: ./app_password.txt
mariadb_root_password:
file: ./mariadb_password.txt
xtrabackup_password:
file: ./xtrabackup_password.txt
 haproxy & web services
services: web:
haproxy: image: alvinr/demo-webapp-vote:mariadb
image: dockercloud/haproxy environment:
networks: SERVICE_PORTS: "5000"
- front VIRTUAL_HOST: "prod.myapp.com"
- back APP_MARIADB_HOST: "maxscale"
volumes: APP_USER: "app"
- /var/run/docker.sock:/var/run/docker.sock APP_PASSWORD_FILE: "/run/secrets/app_password"
ports: APP_DATABASE: "test"
- 80:80 networks:
deploy: - back
placement: deploy:
constraints: [node.role == manager] placement:
constraints: [node.role != manger]
secrets:
- app_password
Demo 3
Hardened
Database Tier
 Container Placement
Docker Host (swarm-0) Docker Host (swarm-1)

HAProxy App
Container Container
Endpoint Endpoint Endpoint

front Network

Docker Host (swarm-2) Docker Host (swarm-3)

MaxScale MariaDB MariaDB

Container Container Container

Endpoint Endpoint Endpoint

back Overlay Network

 Container Placement
Docker Host (swarm-0) Docker Host (swarm-1)

HAProxy App
Container Container
Endpoint Endpoint Endpoint

front Network

Docker Host (swarm-2) Docker Host (swarm-3)

MaxScale MariaDB MariaDB

Container Container Container

Endpoint Endpoint Endpoint

back Overlay Network

Container Placement
mariadb_cluster:
image: alvinr/mariadb-galera-swarm
...
labels:
com.mariadb.cluster: "myapp-prod-cluster"
...
deploy:
replicas: 1
placement:
constraints: [engine.labels.com.mariadb.cluster != myapp-prod-cluster]
Restarting on Failure

maxscale:
image: alvinr/maxscale-swarm
...
labels:
com.mariadb.cluster: "myapp-maxscale"
networks:
- back
deploy:
replicas: 1
restart_policy:
condition: on-failure
delay: 10s
placement:
constraints: [engine.labels.com.mariadb.cluster != myapp-maxscale]
secrets:
- app_password
Part Six
Considerations
& Conclusions
 Service Discovery - How to mesh nodes?

DNS RESOLUTION
Docker assigns VIP to Service, each Task has
own IP
nslookup, dig, getent etc.

3rd PARTY
consul, etcd, zookeeper etc.

DOCKER EVENTS
https://docs.docker.com/engine/
reference/api/docker_remote_api/
Interlock -
ttps://github.com/ehazlett/interlock
 Storage: Inside or Outside the Container?

Host

Docker Daemon Docker Daemon

Container Container
Local Disk e.g.
/dev/xvdb
SSD / NVMe
Networked
/mnt/xx:/var/lib/mysql e.g. EBS
Volume

Inside Outside
Encapsulation Separation of Concerns
of Concerns Storage features (e.g. Snapshots)
3rd Party options
NetApp, Google Compute Engine, Rancher Convoy
Flocker
 Storage: Data Container?

Host Host

Docker Daemon Docker Daemon

Container Container
--volumes-from
{container name}

Inside
Managed like
other containers
Special rule for
Destruction
TBD: Performance
And...

Swarm locking
Image verification (trusted Images)
AppArmor / Seccomp profiles
Monitoring
Heathchecks
Rolling Upgrades
Summary

One Solution Development -> Production

Define Images & Orchestration once
Reuse when needed, inject required behaviours

MariaDB in Production with Docker

Ops define the whitelisted images, security policies
Dev approve images to build upon
Eliminate complexity (and cost) of Deployment
Scale easily, maintain SLA requirements of component
Thanks and Q&A

Code
https://github.com/alvinr/docker-demo/tree/master/mariadb/vote

Docker Images
https://hub.docker.com/_/mariadb/

MariaDB & Docker deployment guide

https://mariadb.com/kb/en/mariadb/installing-and-using-mariadb-via-docker/

Contact me!
alvin@mariadb.com
@jonnyeight