CI & Docker

An experience building
complex CI environments

Patxi Gortázar
francisco.gortazar@urjc.es
@fgortazar
 Bio

Project Coordinator at @elastestio EU project

Devops @ Kurento

Teaching Distributed Systems @ URJC

@fgortazar

https://es.linkedin.com/in/franciscogortazar
Consultancy / Training
Cloud Computing Open source elastic platform
Distributed Systems for end to end testing
Web Technologies
Advanced log management
Extreme Programming
Test orchestration
Testing / Git / Jenkins
Test execution comparison
Software Architectures
Web and Mobile testing
Concurrent Programming
Security testing
IoT testing
Performance testing
http://codeurjc.es http://elastest.io
 Why

A media server for real time video communication

The software under test

WebRTC
Software under test
Software under test
 Software under test
So Kurento is…
• A media server
• APIs for managing connections in a WebRTC
communication
• Tutorials
• Applications
• Testing
– Under many scenarios
– Even more testing in different environments
A soup of technologies
 Some numbers
• 30 code repositories
• ~400 Jenkins jobs
• +1,000 tests
• +20 different environments to test
• +80 artifacts to be deployed at release time
 Feasible infrastructures
Cloud
• Public or private
• 20 environments = 20 different VMs
• Cost increases with each new environment!!
• Effort increases with each new environment to
configure (ops)!!
• Time-to-market increases also!!
 Kurento’s CI Infrastructure

Building clients
Building Debian packages
• 1 node mvn & jdk 7 &
• 2 Ubuntu Trusty VMs node 4
• 4 Ubuntu Xenial VMs • 1 node mvn & jdk 7 &
• 2 Ubuntu latest VMs node 6
• 1 node mvn & jdk 8 &
node 4
• 1 node mvn & jdk 8 &
node 6
 Further problems
• Developers are pushing hard towards ops/devops
to include changes in infrastructure
• Changes can hardly be reverted (possible, but…)
• Hard to test locally
– Works in my machine effect
• Wasted resources
• And at the same time, insufficient resources

Can we do better?
Simplifying CI with Docker
 How
• Leverage Jenkins Pipeline
– CI configuration hosted within code repositories
– Docker support out-of-the-box
– Jenkins jobs launch containers on top of VMs
– VMs have all the same minimum configuration:
Docker
 Current Infrastructure

1 VM Jenkins Master
N VMs Jenkins Workers (running Docker)

2 VM configurations
 Where are my environments?

• The different environments are encapsulated as Docker

images
• Docker images can be used in any of the workers
• Docker images can be used in your laptop
• Docker images are versioned, easy to maintain, easy to build
• Any version can be used
• Several environments can be run in parallel in the same VM:
isolation
 Which environments do I need?

You need images for your builds

You need images for your applications
You need images for your tests
 Images for builds
• Heavy images (> 400Mb)
• One image per tool version: jdk7, jdk8
• Download ahead of time
– As opposed to downloading at build time
– E.g., downloading images in a nightly job
 Images for builds
With Docker it’s easier to tame matrix envs
• Consider building a project using…
– Ubuntu
• Trusty
• Xenial
• Zesty
– Java
• JDK 7
• JDK 8
• JDK 9
 Images for builds
What if we need to mix…
• JDK
• Maven
• Node
• Npm
• Bower
… in the same environment?
 Images for applications
You're ready to ship your application as a
container (optional)
• Choose lightweight containers
– Docker is focusing on lightweight containers
leveraging Alpine Linux
• Include just what your application needs
– Leverage new multi-stage build in Docker
• Think about dependencies (databases, caching...)
 Images for testing
• You can run browsers within a container
– https://github.com/SeleniumHQ/docker-selenium
• Available images
– selenium/hub
– selenium/node-chrome
– selenium/node-firefox
– selenium/node-chrome-debug
– selenium/node-firefox-debug
 Images for testing
• Unfortunately they're a bit out of date
– See https://github.com/SeleniumHQ/docker-
selenium/issues/229#issuecomment-253334224
• Don't worry
– It's easy to build them with the latest browser
– We can test against latest release, beta and canary
• Alternatives
– TestContainers: https://www.testcontainers.org/
 Testing with multiple containers
Orchestration...
• docker compose
– Require to know the number of containers in
advance
• Otherwise, docker run
• Cleaning containers and images with spotify-gc
 Testing with multiple containers
Orchestration...
• End-to-end tests
– 1-5 browsers
– Test scenario is run several times with different
configurations
• Chrome (stable, beta, canary)
• Firefox (stable, beta)
•…
 Testing with multiple containers
Orchestration...
• Tests can include a Docker client for flexibility:
– Start & stop containers
– Container inspection
– Gathering logs
– Managing Volumes
– Managing environment variables
 Testing with multiple containers
Orchestration...
• Network topologies
– Specific network plugins
– Handcrafted networks
• docker run --net=none …  Ugly hack!!
• Better use network plugins
Testing with multiple containers
 Challenges
• Gathering data
– Logs from several containers
– Metrics (CPU, memory, network…)
– Files generated (browser recording)
• Integrating data with test execution
– Comparing some data across several executions
– Data integration
 Challenges
• Reproducing real-world conditions
– Failing nodes
– Failing networks
– Network bandwidth (4G, 5G, DSL)
– Network topologies (NAT traversal, firewalls, …)
– High loads (bursting CPUs)
Consultancy / Training
Cloud Computing Open source elastic platform
Distributed Systems for end to end testing
Web Technologies
Advanced log management
Extreme Programming
Test orchestration
Testing / Git / Jenkins
Test execution comparison
Software Architectures
Web and Mobile testing
Concurrent Programming
Security testing
IoT testing
Performance testing
http://codeurjc.es http://elastest.io
 Bio

Coordinator http://elastest.io/

Devops @ Kurento

Teaching Distributed Systems @ URJC

@fgortazar

https://es.linkedin.com/in/franciscogortazar