Puppet at Scale

Sam Kottler
@samkottler

1 Sam Kottler | Puppet at Scale

About me

● Worked on large-scale infra for the web @ Venmo,

Acquia, and Digg
● Rubygems.org infrastructure
● Bundler core
● Fedora developer
● Core committer on the Foreman

2 Sam Kottler | Puppet at Scale

 What we'll cover

1. Some basics
2. Master vs. masterless deployment
3. CA management
4. Clustering
5. Node management
6. Development + deployment practices

3 Sam Kottler | Puppet at Scale

Why we care

● Hyperscale computing
● Massive, multi-DC infrastructure
● Dynamic environments
● The Cloud ™

4 Sam Kottler | Puppet at Scale

 Master vs. masterless

5 Sam Kottler | Puppet at Scale

Provisioning nodes with a master

1. New node comes online

2. A script is run to install packages and configure
/etc/hosts
3. The agent gets run, generates a CSR, and sends it to
the master
4. The cert gets signed based on an autosign rule or
`puppet cert --sign <nodename>`
5. Puppet runs

6 Sam Kottler | Puppet at Scale

Provisioning nodes without a master

1. New node comes online knowing its role

2. A script runs to install packages and retrieve
package/tarball
3. puppet apply

7 Sam Kottler | Puppet at Scale

Certificate authority

● Use by Puppet to authenticate agents

● CSR generated and signed by the CA
● Shared CRL across all CA machines

8 Sam Kottler | Puppet at Scale

Clustering patterns
● CA has lots of state
● Masters should be stateless
● Reduce the number of file shares

9 Sam Kottler | Puppet at Scale

DNS-based clustering

10 Sam Kottler | Puppet at Scale

Load balanced clustering

11 Sam Kottler | Puppet at Scale

Masters across data-centers

● Shared CA vs. per-region

● Deploy in stages across data-centers

12 Sam Kottler | Puppet at Scale

Multi-cluster

13 Sam Kottler | Puppet at Scale

 Node classification

14 Sam Kottler | Puppet at Scale

15 Sam Kottler | Puppet at Scale
External node classifiers

● Output YAML based on external data

● The Foreman, Puppet Enterprise, Puppet
Dashboard
● Your own custom data source
● Key integration source with your own CMDB

16 Sam Kottler | Puppet at Scale

17 Sam Kottler | Puppet at Scale
Packaging for masterless
● Use /etc/puppet/modules (or modulepath)
● Build RPM's/deb's for distribution
● Publish packages to a repo
● Install/update packages on all machines

https://github.com/skottler/librarian-masterless-packaging

18 Sam Kottler | Puppet at Scale

Distributed runs
● Run puppet based on changes in your code
● Mcollective/SSH/cron

19 Sam Kottler | Puppet at Scale

Deployment

● Masters are just another deployment target!

● Build CI pipelines
● One-click deployments to masters
● Lint and test your modules

20 Sam Kottler | Puppet at Scale

 cap puppetmaster deploy DC=london

21 Sam Kottler | Puppet at Scale

Controlled releases

● Separate hosts into groups to do red/black releases

● Build smaller sub-groups of canary hosts
● Monitor your puppet runs

22 Sam Kottler | Puppet at Scale

Thanks!

@samkottler
shk@redhat.com
https://github.com/skottler

23 Sam Kottler | Puppet at Scale