Speeding Up CI CD With OCS - Webinar

Optimize your CI/CD pipelines
with container storage

July 7, 2020
Marcel Hergaarden Sagy Volkov
Senior Solutions Architect Storage Performance Instigator
1
Red Hat helps businesses run containers
on any infrastructure
Containers | VMs | Serverless apps
Bare Metal Virtual Private Cloud Public Cloud

2
STORAGE FOR KUBERNETES
Every workload needs state
Application Continuous integration

Development / delivery pipelines
& Deployment
SQL & NoSQL Analytics & AI/ML

databases
3
STORAGE FOR KUBERNETES
Every workload needs state
Application Continuous integration

Development / delivery pipelines
& Deployment
SQL & NoSQL Analytics & AI/ML

databases
4
Agenda
Agenda
● CI / CD trends in cloud native development

● CI / CD tooling
● An example with Jenkins
● Jenkins on Kubernetes using persistent storage
● Results and benefits
● Demo
5
Optimizing CI / CD with persistent storage
CI / CD trends in
cloud native development
6
Creating value depends on your ability
to deliver applications faster
Cloud-native AI & machine Internet of Innovation

Analytics
applications learning Things culture
Containers, Kubernetes, and hybrid cloud are key ingredients.
7
DevOps Standardization on Kubernetes
90%
Cloud native software such as containers,

Kubernetes and serverless is very or
somewhat important
8
Source:
451 Research, “DevOps: The Business and Management of DevOps Voice of the Enterprise 2H 2019 Advisory Report”; Jay Lyman
DevOps prefers open source
90%
Feel open source software to be
47%
Have open source preference or
15%
Have open source aversion or prohibition
somewhat or very important mandate
9 Source:
451 Research, “DevOps: The Business and Management of DevOps Voice of the Enterprise 2H 2019 Advisory Report”; Jay Lyman
CI/CD and it’s tooling
10
Continuous Integration and Continuous Delivery (CI/CD)
11
So at what stage does CI, CD or CD apply ?
12
Speeding up CI / CD with persistent storage
Things can become complex soon
GIT COMMIT RELEASE
Code Run Debug Build Int Test Package Deploy Stage ...
LOCAL DEVELOPMENT CONTINUOUS INTEGRATION CONTINUOUS DELIVERY
13
Polling Question: What are your prefered CI /CD tools that
support Kubernetes?
(Choose all that apply)
A) CircleCI
B) Travis
C) Jenkins
D) CodeFresh
E) Other
F) None yet
14
Polling Question
Let’s take an example
15
Why we use Jenkins for the demo
● Jenkins is an open source automation

server that targets CI/CD
● Jenkins has broad adoption in the market
● Supports most version control tools,

including Git and others
● Can trigger automated builds in various

ways
16
JRebel research
Jenkins features
Easy setup Easy configuration Plugins Extensible Distributed
Self contained Web interface 100s+ plugins Extensible via plugin Easily distributes
Java based Built-in help Integrates with every architecture across multiple
application Built-in error checks tool in the CI/CD Infinite possibilities resources
toolchain
17
Jenkins pipeline
Code Commit Build Test Stage/Release Deploy
Continuous Integration/Delivery
Development Production
Results Report
18
Jenkins pipeline
Results Report
19
Jenkins pipeline
Results Report
20
Jenkins pipeline
Results Report
21
Jenkins pipeline
Results Report
22
Jenkins pipeline
Results Report
23
Jenkins pipeline
Results Report
24
Jenkins architecture
Developer Repository CI Server Build Server Test Server
Creates code Source code Retrieves the Builds code Automated

Submits code repository new code to executable testing
25
Issues and Options
26
Problems at scale
● So what happens if you have 100 jobs,

1000 jobs, 10.000 jobs?
● Jenkins - and others - scale either

vertically, or horizontally with slave
builders
● Dependency managers download the

internet (Maven, Gradle, NPM, RPM etc
etc)
27
Repetitive issues
● Numerous builds
In real life situations, 1000s+ of daily builds can take place
Builds are happening parallel in many situations
● Infrastructure resources impact

Network loads, storage volumes and copies
Repeating similar patterns, over and over again, to Zillion times
28
Mitigation options
● Proxy server
Build servers are known for “downloading the internet”
Use of a proxy server for internet access enables for caching
Huge local network load still remains
● Storage volumes
Mounting shared storage to build servers
Offers local availability of build resources and dependencies
without infrastructure impact
29
Optimizing CI / CD with persistent storage CONFIDENTIAL Designator
Lowering build times - performance results
30
Jenkins on
Kubernetes
31
Jenkins in a container
● Certain issues are often fixed by running business apps in containers

Jenkins in a container remains to have the same issue in a container
Slave Builder
N redundant file shares, that
Central CI/CD need networking & storage
Slave Builder
Server resources to fill and store
Slave Builder (including i/o)
● Jenkins enables for builder scaling by its Master-Slave model

Slave builders run in ephemeral containers by default
Not maintaining state causes repetitive downloads as a result
32
The initial issue will therefore even get worse
Schematic view
Remote Local copy

Packages Packages
Don’t do this !
Slave Builder
N redundant file shares, that
Central CI/CD need networking & storage
Slave Builder
Server resources to fill and store
Slave Builder (including i/o)
N slave builders
* Red Hat OpenShift Container
N of builds per day Storage offers a shared volume,
*
concurrent accessible for all Slave
N packages downloaded
Builders
33
Instance Mount Limitations
● Using Red Hat Openshift Container storage can help overcome mount limit
issues and speed build times by as much as 90%
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/volume_limits.html
34
About
Persistent storage
35
Why aren’t all companies seeing
the promised benefits of containers?
Pros Cons
Containers are not Containers are not

persistent persistent
They have a very light If your container goes down,
footprint your data is lost
They can be created on That’s why you need to
multiple platforms rethink how to do storage
36
Why containerized apps need persistent storage
Container Container Container
Application Application Application
Library & Binaries Library & Binaries Library & Binaries
OS Base Layer OS Base Layer OS Base Layer
Container Platform
Host OS
Infrastructure
37
CONFIDENTIAL Designator
OCS Persistent Storage - Types and Capabilities
Persistent Volume Object Service
Block Shared File System S3
● Primary for DB and ● POSIX-compliant shared file system ● Archiving, Image and video
Transactional workloads ● Interface for legacy workloads repositories
● Low latency ● CI/CD Pipelines ● Object API (S3)
38
Demo Jenkins on top of
by Sagy Volkov
OpenShift Container
Storage
39
Further resources
Take a look at our “Speed Jenkins performance with
container storage” solution brief.
https://www.redhat.com/cms/managed-files/cl-rhocs-for-jenkins-pipelin
es-brief-f21252-202004-en.pdf
Read the blog post

https://blog.openshift.com/deploy-jenkins-pipelines-in-openshift-4-with-
openshift-container-storage-4/
Try our 5-10 minute developer exercises:

https://learn.openshift.com/persistence/
40
Thank you linkedin.com/company/red-hat
youtube.com/user/RedHatVideos
Red Hat is the world’s leading provider of enterprise
open source software solutions. Award-winning

facebook.com/redhatinc
support, training, and consulting services make
Red Hat a trusted adviser to the Fortune 500.

twitter.com/RedHat
41

Speeding Up CI CD With OCS - Webinar

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Speeding Up CI CD With OCS - Webinar

Uploaded by

Copyright:

Available Formats

Optimize your CI/CD pipelines

with container storage

Containers | VMs | Serverless apps

Bare Metal Virtual Private Cloud Public Cloud

Every workload needs state

Application Continuous integration

SQL & NoSQL Analytics & AI/ML

Every workload needs state

Application Continuous integration

SQL & NoSQL Analytics & AI/ML

● CI / CD trends in cloud native development

Cloud-native AI & machine Internet of Innovation

Containers, Kubernetes, and hybrid cloud are key ingredients.

DevOps Standardization on Kubernetes

Cloud native software such as containers,

DevOps prefers open source

CI/CD and it’s tooling

Continuous Integration and Continuous Delivery (CI/CD)

So at what stage does CI, CD or CD apply ?

Things can become complex soon

GIT COMMIT RELEASE

LOCAL DEVELOPMENT CONTINUOUS INTEGRATION CONTINUOUS DELIVERY

Let’s take an example

Why we use Jenkins for the demo

● Jenkins is an open source automation

● Jenkins has broad adoption in the market

● Supports most version control tools,

● Can trigger automated builds in various

Easy setup Easy configuration Plugins Extensible Distributed

Code Commit Build Test Stage/Release Deploy

Code Commit Build Test Stage/Release Deploy

Code Commit Build Test Stage/Release Deploy

Code Commit Build Test Stage/Release Deploy

Code Commit Build Test Stage/Release Deploy

Code Commit Build Test Stage/Release Deploy

Code Commit Build Test Stage/Release Deploy

Developer Repository CI Server Build Server Test Server

Creates code Source code Retrieves the Builds code Automated

Issues and Options

● So what happens if you have 100 jobs,

● Jenkins - and others - scale either

● Dependency managers download the

● Infrastructure resources impact

Lowering build times - performance results

● Certain issues are often fixed by running business apps in containers

● Jenkins enables for builder scaling by its Master-Slave model

Remote Local copy

Instance Mount Limitations

Containers are not Containers are not

Container Container Container

Application Application Application

Library & Binaries Library & Binaries Library & Binaries

OS Base Layer OS Base Layer OS Base Layer

OCS Persistent Storage - Types and Capabilities

Persistent Volume Object Service

Block Shared File System S3

Transactional workloads ● Interface for legacy workloads repositories

● Low latency ● CI/CD Pipelines ● Object API (S3)

Demo Jenkins on top of

Read the blog post

Try our 5-10 minute developer exercises:

open source software solutions. Award-winning

Red Hat a trusted adviser to the Fortune 500.