Matthias Sohn

Adel Zaalouk
SAP
 From Containers to Kubernetes Container Scheduler

Benefits
Isolation
Container Immutable infrastructure
Portability
Faster deployments Kubernetes
Versioning
Ease of sharing
Orchestration of cluster of containers across
multiple hosts
Container Runtime •Automatic placements, networking, deployments,
Challenges scaling, roll-out/-back, A/B testing
Host OS
Networking Declarative – not procedural
VM Deployments •Declare target state, reconcile to desired state
Service Discovery •Self-healing
Auto Scaling
Persisting Data Workload Portability
Logging, Monitoring •Abstract from cloud provider specifics
Access Control •Multiple container runtimes

Docker
What does Kubernetes not cover ?

• Install and manage many clusters

• Across Multi-Cloud Gardener
• Public Cloud Providers
• Private Cloud
• Zero Ops
• Minimal TCO
• Manage Nodes
• Manage Control Planes
• Day 2 Operations
WHAT do we want to achieve with the
Gardener?
Provide and establish solution for Kubernetes Clusters as a
Service
Central Provisioning
Engage with Open Source community,
foster adoption, become CNCF project
Large scale organisations need hundreds or
thousands of clusters
WHAT do we want to achieve with the
Gardener?
Homogenously on Hyper-Scale Providers and for the Private
Cloud
Full Control of Kubernetes,
Homogeneous Across All Installations

AWS, Azure, GCP, Alibaba and Others

Private DCs for Data Privacy:

OpenStack
and eventually Bare Metal
WHAT do we want to achieve with the
Gardener?
with Minimal TCO and Full Day-2 Operations Support

Full Automation, Backup & Recovery,

High Resilience and Robustness, Self-Healing,
Auto-Scaling, …

Rollout Bug Fixes, Security Patches,

Updates of Kubernetes, OS, Infrastructure,
Certificate Management,
…
Gardener Mission

Provide and establish solution for Kubernetes Clusters as a

Service
Homogenously on Hyper-Scale Providers and for the Private
Cloud
with Minimal TCO and Full Day-2 Operations Support
Primary Gardener Architecture Principle
Following the definition of Kubernetes…
Kubernetes is a system for automating
deployment, scaling, and management
of containerized software

…we do the
following:
We use Kubernetes to deploy, host and operate
Kubernetes
Control planes are “seeded” into already existing clusters
 Common Kubernetes Cluster Setup
Master Worker Master Worker Master Worker

Master Worker Master Worker Master Worker

Master Worker Master Worker Master

HA HA HA
Worker

Master Worker Master Worker

Master Master Worker Worker Worker

Master Worker Master Worker

HA
Master Worker Worker

HA Master

Worker
Worker

The green machines host the control plane,

often in HA and on separated hardware
(usually underutilized or, worse, overutilized) The blue machines host the actual workload and
are managed by Kubernetes (usually pretty well utilized)
 Gardener Kubernetes Cluster Setup
Gardener Cluster Seed Cluster Shoot Clusters

Master Worker Master Worker Worker Worker

Master Worker Master Worker Worker Worker

Master Worker Master Worker Worker

HA HA
Worker

manages
Worker Worker Worker

Zooming into the ETCD API Server Worker

Seed Cluster reveals… Controller Mgr Scheduler Worker Worker

Multiple Shoot Cluster

Control Planes ETCD API Server Worker

Controller Mgr Scheduler

ETCD API Server … Gardener Machine Controller Manager
Machine Provisioning
Inside a Seed Self-Healing
Cluster Worker Auto-Update
Auto-Scaling
Primary Gardener Design Principle

Do not reinvent the wheel …

“Let Kubernetes drive the design

of the Gardener.”
Lingua Franca – Gardener Cluster Resource
apiVersion: garden.sapcloud.io/v1
Native Kubernetes Resource kind: Shoot
metadata:
name: my-cluster
namespace: garden-project
spec:
dns:
provider: aws-route53
Gardener or Self-Managed DNS domain: cluster.ondemand.com
cloud:
aws:
networks:
vpc:
Define Your Infrastructure Needs cidr: 10.250.0.0/16
workers:
- name: cpu-worker

Specify Worker Pools machineType: m4.xlarge

autoScalerMin: 5
autoScalerMax: 20
kubernetes:
Set Kubernetes Version version: 1.11.2
kubeAPIServer:
featureGates: ...
Tweak Kubernetes Control Plane runtimeConfig: ...
admissionPlugins: ...
kubeControllerManager:
featureGates: ...
kubeScheduler: Avoid Vendor Lock-In
featureGates: ...
kubelet:
featureGates: ...
maintenance:
timeWindow:

Define When and What to Update begin: 220000+0000

end: 230000+0000
autoUpdate:
kubernetesVersion: true

Gardener Reported Status status:

...
 Garden Cluster Seed Cluster Shoot Cluster
Administrator End-User
R R R R R
R R
Kubernetes Gardener
Kubify kubectl gardenctl Kubernetes
Dashboard Dashboard kubectl
R R R R R Dashboard
R R

HTTPS

Garden Cluster Seed Cluster Shoot Cluster

Garden Cluster API LB Ingress LB Seed Cluster API LB Shoot Cluster API LB Shoot Cluster VPN LB
R R R R

Garden Cluster Storage Seed Cluster Storage

[K8s] DS, RS, SS, J, ... [K8s] DS, RS, SS, J, ...
Control Plane [CRD] Shoot, Seed, ... Control Plane [CRD] Machine Deployment
R
R R R R
R
Worker W + Container Runtime
Kubelet Worker Worker
Kubelet + Container Runtime Kubelet + Container Runtime
etcd Main Backup SS Main PV
Gardener Dashboard D VPN D
etcd Events SS Events PV R
Gardener API Server D Calico DS
API Server VPN D R
R Kube Proxy DS
Gardener Controller Manager D R
... Scheduler D
Core DNS D
R

Controller Manager D
New Shoot Clusters can be created via the Gardener Optional Addons
dashboard or by uploading a new Shoot resource to Addon Manager D
the Garden Cluster. The Gardener picks it up and starts
a Terraform job to create the necessary IaaS Machine Controller D
components. Then it deploys the Shoot Cluster Control R Actual Workload
Plane into the Seed Cluster and required add-ons into
the Shoot Cluster. Update or delete operations are
Terraformer
Shoot Cluster Control Plane
J
R ...
handled by the Gardener fully automatically as well.
Monitoring VPN
... Logging
R
SCP ... ...
...
IaaS IaaS
Following the Design Principle Gardener
uses…
Workload

Additional Tooling
Add-On Manager Network policies Cert Manager Cluster Prometheus
Helm Calico Cert Broker Autoscaler
EFK Stack

K8S building blocks

Deployments Jobs Stateful Sets Reconciliation
Secrets
PVs PVCs CRDs Controllers Admission Control
Replicasets Load- Config
Balancer Maps Driver API Server Extension RBAC
Pods

Kubernetes as deployment underlay

Where are all these clusters coming from?
Garden clusters are installed on a bootstrap cluster
•in GKE, EKS, AKS
•set up using Gardener’s Kubify
•DR setup with the Gardener Ring (planned)

Seed clusters are created as shoot clusters by the Gardener

Shoot clusters are created by their seed cluster

which is managed by the Gardener
Gardener Demo
Gardener Community Installer
Setting up a Gardener landscape is not trivial,
so we have a community installer:
https://github.com/gardener/landscape-setup

•Many shortcuts to make it simple (Gardener and Seed in a single cluster)

•Do not use productively!
•You can use it as a starter for a productive setup
•Different cluster and different cloud provider accounts recommended
Gardener is Open Source

Long-Term Goal
Become CNCF Project

Gardener Blog
CNCF Presentation
Kubernetes Podcast
Hacker News
Reddit
Thank You!

GitHub https://github.com/gardener
Home Page https://gardener.cloud
Wiki https://github.com/gardener/documentation/wiki
Mailing List https://groups.google.com/forum/?fromgroups#!forum/gardener
Slack Channel https://kubernetes.slack.com/messages/gardener
Community Installer https://github.com/gardener/landscape-setup
Kubernetes Machine Controller Manager
Problem
•Node provisioning and de-provisioning is out of scope of current Kubernetes
•In the beginning we used terraform scripts ➦ unmanageable
•No mechanism
• to smoothly scale clusters
• upgrade cluster nodes for all providers

Machine Controller Manager

•Node custom resources to manage nodes via k8s API
•Plugins enable support for different cloud providers
•Enables cluster auto-scaling and upgrade of cluster nodes
MCM Model

Model for Kubernetes

deployments works great Pod Machine
So why not use it for
machines?
ReplicaSet MachineSet

Deployment MachineDeployment
MCM Custom Resources
AWS-Machine-Class Secret
(Template)
Name: test-secret
Name: v1 Cloudconfig:
Machine Type: t2.large abc….xzy
Disk Size: 50GB AccessKeyId: abc123
Secret: test-secret SecretAccessKey: xyz789
……

Machine Machine-
Machine-Set Deployment
Name: test-
machine Name: test-ms Name: test-md
Replicas: 3 Replicas: 3
MachineClass: v1 UpdateStrategy: Rolling
MachineClass: v1
MachineClass: v1
Working of MCM Kubernetes
Controller
Manager

Cloud 3 VMs
Provider API
Node objects help in
monitoring the machine
status – Health Machine Controller
Manager
Machine
Node Machine
Deployment
Machine
Machine
Class: V1 Node Machine
Class + Secret Machine
Controller
Replicas: 3
V1 Class + Secret
Node Machine

Kubernetes Machine Machine Set

API Server Set Controller
Replicas: 3
kubectl
Machine
Machine Machine Deployment
Class + Secret ETCD Deployment
V1 Controller
(Key-value store)
Class: V1
Replicas: 3
Autoscaling Kubernetes
Controller
Manager

Forked Cluster
Autoscaler Now assume that all the nodes resources are Cloud
nearly consumed and a new pod is created Provider API

Node 4 Machine Machine Controller

Manager
Node 3 Machine
Pod Machine
Node 2 Machine
Image: Nginx Controller
Node: - Node 1 Machine

Kubernetes Machine Set

API Server Controller

kubectl
Machine
Pod Machine Deployment
ETCD Deployment Controller
Image: Nginx (Key-value store)
Node: Node4
Unschedulable Class: V1
Replicas: 4
3
Machine Controller Manager - Components
Create/delete Create/update Update no. of
Machines to maintain Machine-sets to replicas based on
required replicas perform updates load

Machine- Machine- Cluster

Machine
Set Deployment Auto-
Controller
Controller Controller scaler

Responsible for Responsible for Responsible for Scales the number of

Managing Maintaining set of Managing Machine-sets replicas based on
Machines healthy Machine (used for updates) load in the cluster
replicas

Parent-child relationship: Adoption of orphaned children

Controllers cooperate, rather than racing with each other !
Machine Controller
Manager