Puppet at SpaceX

How to stay sane, while sending rockets into space

Whos this guy anyway?

!! Linux system admin !! Came from the video game industry !! Started at SpaceX about 9 month ago !! Linux System Engineer

SpaceX
!! !! !! !!
Rocket Company First private company to send a vehicle to LEO AND recover the vehicle Founded in 2002 !! Falcon1, Falcon 9, Dragon

Puppet
!! How it happened
!! Before Scale 10x !! After Scale 10x

Bootstrapping
!! What we have
!! !! !! !! !!
Debian/Ubuntu Kerberos (windows KDC/Active Directory) LDAP (windows LDAP/Active Directory) Apt mirror (with debian squeeze/ubuntu LTS) Tons of scripts!

Deployment
!! Using existing scripts and replacing them with
puppet. !! Install packages from our local repo !! Push configs with puppet !! Using exec (in a couple of places only) to fix a
couple of things.

!! This simplified things a lot for us

Where are we at now?

!! Plug desktop/server in (power/network/keyboard/
video) !! Boot off the network (pxe) !! Pick on pxe menu what you want to install !! We have a number of presets we use !! Install starts and asks the user for hostname !! Machine uses a pre-seed file to setup stuff initially !! Installs puppet as part of that pre-seed from our local
mirror !! Generate and registers with puppet server !! (login on server and sign cert) !! And wait for it! Were DONE!

Cool stuff to come

!! Puppet server infrastructure overview
!! TheForeman
!! Really nice and easy to setup !! Has proxies for handling stuff with DHCP/DNS/PXE
automation (were planning on using this, and its a little more difficult for us, since that part of the infrastructure is windows based)

!! Nagios monitoring
!! Host lists/services/dependencies built by puppet !! HPC !! We have a compute cluster that we are starting to rebuild
using puppet for automation and validation (test cluster with a couple of nodes, validate config, and push to production)