K3S ? Need Help?

 Get free intro and advanced online training ×

Architecture

Advanced Options and

Configuration
Quick-Start Guide

Installation

This section contains advanced information describing the different ways you can run and manage
Cluster Access
K3s:

Upgrades Certi cate rotation

Auto-deploying manifests
Backup and Restore Using Docker as the container runtime
Con guring containerd
Volumes and Storage Secrets Encryption Con g (Experimental)
Running K3s with RootlessKit (Experimental)

Networking Node labels and taints

Starting the server with the installation script
Additional preparation for Alpine Linux setup
Helm
Running K3d (K3s in Docker) and docker-compose

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 Enabling legacy iptables on Raspbian Buster
Advanced Options and Con guration
Enabling cgroups for Raspbian Buster
SELinux Support
FAQ Additional preparation for (Red Hat/CentOS) Enterprise Linux

Known Issues

Security
Certificate Rotation
By default, certi cates in K3s expire in 12 months.

If the certi cates are expired or have fewer than 90 days remaining before they expire, the certi cates
are rotated when K3s is restarted.

Auto-Deploying Manifests
Any le found in /var/lib/rancher/k3s/server/manifests will automatically be deployed
to Kubernetes in a manner similar to kubectl apply .

For information about deploying Helm charts, refer to the section about Helm.

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 Using Docker as the Container
Runtime
K3s includes and defaults to containerd, an industry-standard container runtime.

To use Docker instead of containerd,

1. Install Docker on the K3s node. One of Rancher’s Docker installation scripts can be used
to install Docker:

curl https://releases.rancher.com/install-docker/19.03.sh | sh

2. Install K3s using the --docker option:

curl -sfL https://get.k3s.io | sh -s - --docker

3. Con rm that the cluster is available:

$ sudo k3s kubectl get pods --all-namespaces

NAMESPACE NAME READY

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 kube-system local-path-provisioner-6d59f47c7-lncxn 1/1
kube-system metrics-server-7566d596c8-9tnck 1/1
kube-system helm-install-traefik-mbkn9 0/1
kube-system coredns-8655855d6-rtbnb 1/1
kube-system svclb-traefik-jbmvl 2/2
kube-system traefik-758cd5fc85-2wz97 1/1

4. Con rm that the Docker containers are running:

$ sudo docker ps
CONTAINER ID IMAGE COMMAND
3e4d34729602 897ce3c5fc8f "entry"
bffdc9d7a65f rancher/klipper-lb "entry"
436b85c5e38d rancher/library-traefik "/traefik --conf
de8fded06188 rancher/pause:3.1 "/pause"
7c6a30aeeb2f rancher/pause:3.1 "/pause"
ae6c58cab4a7 9d12f9848b99 "local-path-prov
be1450e1a11e 9dd718864ce6 "/metrics-server
4454d14e4d3f c4d3d16fe508 "/coredns -conf
c3675b87f96c rancher/pause:3.1 "/pause"
4b1fddbe6ca6 rancher/pause:3.1 "/pause"
64d3517d4a95 rancher/pause:3.1 "/pause"

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 Optional: Use crictl with Docker
crictl provides a CLI for CRI-compatible container runtimes.

If you would like to use crictl after installing K3s with the --docker option, install crictl using the
o cial documentation:

$ VERSION="v1.17.0"
$ curl -L https://github.com/kubernetes-sigs/cri-tools/releases/dow
$ sudo tar zxvf crictl-$VERSION-linux-amd64.tar.gz -C /usr/local/b
crictl

Then start using crictl commands:

$ sudo crictl version

Version: 0.1.0
RuntimeName: docker
RuntimeVersion: 19.03.9
RuntimeApiVersion: 1.40.0
$ sudo crictl images
IMAGE TAG IMAGE ID
rancher/coredns-coredns 1.6.3 c4d3d16fe508b
rancher/klipper-helm v0.2.5 6207e2a3f5225
rancher/klipper-lb v0.1.2 897ce3c5fc8ff
rancher/library-traefik 1.7.19 aa764f7db3051

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 rancher/local-path-provisioner v0.0.11 9d12f9848b99f
rancher/metrics-server v0.3.6 9dd718864ce61
rancher/pause 3.1 da86e6ba6ca19

Configuring containerd
K3s will generate con g.toml for containerd in
/var/lib/rancher/k3s/agent/etc/containerd/config.toml .

For advanced customization for this le you can create another le called config.toml.tmpl in
the same directory and it will be used instead.

The config.toml.tmpl will be treated as a Go template le, and the config.Node structure
is being passed to the template. This template example on how to use the structure to customize the
con guration le.

Secrets Encryption Config

(Experimental)
As of v1.17.4+k3s1, K3s added the experimental feature of enabling secrets encryption at rest by
passing the ag --secrets-encryption on a server, this ag will do the following automatically:

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 Generate an AES-CBC key
Generate an encryption con g le with the generated key

{
"kind": "EncryptionConfiguration",
"apiVersion": "apiserver.config.k8s.io/v1",
"resources": [
{
"resources": [
"secrets"
],
"providers": [
{
"aescbc": {
"keys": [
{
"name": "aescbckey",
"secret": "xxxxxxxxxxxxxxxxxxx"
}
]
}
},
{
"identity": {}
}
]
}
]
}

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 Pass the con g to the KubeAPI as encryption-provider-con g

Once enabled any created secret will be encrypted with this key. Note that if you disable encryption
then any encrypted secrets will not be readable until you enable encryption again.

Running K3s with RootlessKit

(Experimental)
WARNING:
This feature is experimental.

RootlessKit is a kind of Linux-native “fake root” utility, made for mainly running Docker and Kubernetes
as an unprivileged user, so as to protect the real root on the host from potential container-breakout
attacks.

Initial rootless support has been added but there are a series of signi cant usability issues
surrounding it.

We are releasing the initial support for those interested in rootless and hopefully some people can
help to improve the usability. First, ensure you have a proper setup and support for user namespaces.
Refer to the requirements section in RootlessKit for instructions. In short, latest Ubuntu is your best
bet for this to work.

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 Known Issues with RootlessKit
Ports
When running rootless a new network namespace is created. This means that K3s instance is
running with networking fairly detached from the host. The only way to access services run in K3s
from the host is to set up port forwards to the K3s network namespace. We have a controller that
will automatically bind 6443 and service port below 1024 to the host with an offset of 10000.
That means service port 80 will become 10080 on the host, but 8080 will become 8080 without
any offset.
Currently, only LoadBalancer services are automatically bound.
Daemon lifecycle
Once you kill K3s and then start a new instance of K3s it will create a new network namespace, but
it doesn’t kill the old pods. So you are left with a fairly broken setup. This is the main issue at the
moment, how to deal with the network namespace.
The issue is tracked in https://github.com/rootless-containers/rootlesskit/issues/65
Cgroups
Cgroups are not supported.

Running Servers and Agents with Rootless

Just add --rootless ag to either server or agent. So run k3s server --rootless and
then look for the message Wrote kubeconfig [SOME PATH] for where your kubecon g le is.

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 For more information about setting up the kubecon g le, refer to the section about cluster access.

Be careful, if you use -o to write the kubecon g to a different directory it will probably not
work. This is because the K3s instance in running in a different mount namespace.

Node Labels and Taints

K3s agents can be con gured with the options --node-label and --node-taint which adds
a label and taint to the kubelet. The two options only add labels and/or taints at registration time, so
they can only be added once and not changed after that again by running K3s commands.

If you want to change node labels and taints after node registration you should use kubectl . Refer
to the o cial Kubernetes documentation for details on how to add taints and node labels.

Starting the Server with the

Installation Script
The installation script will auto-detect if your OS is using systemd or openrc and start the service.
When running with openrc, logs will be created at /var/log/k3s.log .

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 When running with systemd, logs will be created in /var/log/syslog and viewed using
journalctl -u k3s .

An example of installing and auto-starting with the install script:

curl -sfL https://get.k3s.io | sh -

When running the server manually you should get an output similar to the following:

$ k3s server
INFO[2019-01-22T15:16:19.908493986-07:00] Starting k3s dev
INFO[2019-01-22T15:16:19.908934479-07:00] Running kube-apiserver -
Flag --insecure-port has been deprecated, This flag will be removed
INFO[2019-01-22T15:16:20.196766005-07:00] Running kube-scheduler -
INFO[2019-01-22T15:16:20.196880841-07:00] Running kube-controller-m
Flag --port has been deprecated, see --secure-port instead.
INFO[2019-01-22T15:16:20.273441984-07:00] Listening on :6443
INFO[2019-01-22T15:16:20.278383446-07:00] Writing manifest: /var/l
INFO[2019-01-22T15:16:20.474454524-07:00] Node token is available a
INFO[2019-01-22T15:16:20.474471391-07:00] To join node to cluster:
INFO[2019-01-22T15:16:20.541027133-07:00] Wrote kubeconfig /etc/ra
INFO[2019-01-22T15:16:20.541049100-07:00] Run: k3s kubectl

The output will likely be much longer as the agent will create a lot of logs. By default the server will
register itself as a node (run the agent).

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 Additional Preparation for Alpine
Linux Setup
In order to set up Alpine Linux, you have to go through the following preparation:

Update /etc/update-extlinux.conf by adding:

default_kernel_opts="... cgroup_enable=cpuset cgroup_memory=1 cgro

Then update the con g and reboot:

update-extlinux
reboot

Running K3d (K3s in Docker) and

docker-compose
Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 k3d is a utility designed to easily run K3s in Docker.

It can be installed via the the brew utility on MacOS:

brew install k3d

rancher/k3s images are also available to run the K3s server and agent from Docker.

A docker-compose.yml is in the root of the K3s repo that serves as an example of how to run
K3s from Docker. To run from docker-compose from this repo, run:

docker-compose up --scale agent=3

# kubeconfig is written to current dir

kubectl --kubeconfig kubeconfig.yaml get node

NAME STATUS ROLES AGE VERSION

497278a2d6a2 Ready <none> 11s v1.13.2-k3s2
d54c8b17c055 Ready <none> 11s v1.13.2-k3s2
db7a5a5a5bdd Ready <none> 12s v1.13.2-k3s2

To run the agent only in Docker, use docker-compose up agent .

Alternatively the docker run command can also be used:

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 sudo docker run \
-d --tmpfs /run \
--tmpfs /var/run \
-e K3S_URL=${SERVER_URL} \
-e K3S_TOKEN=${NODE_TOKEN} \
--privileged rancher/k3s:vX.Y.Z

Enabling legacy iptables on

Raspbian Buster
Raspbian Buster defaults to using nftables instead of iptables . K3S networking features
require iptables and do not work with nftables . Follow the steps below to switch con gure
Buster to use legacy iptables :

sudo iptables -F
sudo update-alternatives --set iptables /usr/sbin/iptables-legacy
sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
sudo reboot

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 Enabling cgroups for Raspbian
Buster
Standard Raspbian Buster installations do not start with cgroups enabled. K3S needs cgroups
to start the systemd service. cgroups can be enabled by appending cgroup_memory=1
cgroup_enable=memory to /boot/cmdline.txt .

example of /boot/cmdline.txt

console=serial0,115200 console=tty1 root=PARTUUID=58b06195-02 root

SELinux Support
Supported as of v1.19.4+k3s1. Experimental as of v1.17.4+k3s1.

If you are installing K3s on a system where SELinux is enabled by default (such as CentOS), you must
ensure the proper SELinux policies have been installed.

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 Automatic Installation
Available as of v1.19.3+k3s2

The install script will automatically install the SELinux RPM from the Rancher RPM repository if on a
compatible system if not performing an air-gapped install. Automatic installation can be skipped by
setting INSTALL_K3S_SKIP_SELINUX_RPM=true .

Manual Installation
The necessary policies can be installed with the following commands:

yum install -y container-selinux selinux-policy-base

yum install -y https://rpm.rancher.io/k3s/latest/common/centos/7/no

To force the install script to log a warning rather than fail, you can set the following environment
variable: INSTALL_K3S_SELINUX_WARN=true .

Enabling and Disabling SELinux Enforcement

The way that SELinux enforcement is enabled or disabled depends on the K3s version.

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 K3S V1.19.1+K3S1 K3S BEFORE V1.19.1+K3S1

To leverage SELinux, specify the --selinux ag when starting K3s servers and agents.

This option can also be speci ed in the K3s con guration le:

selinux: true

The --disable-selinux option should not be used. It is deprecated and will be either
ignored or will be unrecognized, resulting in an error, in future minor releases.

Using a custom --data-dir under SELinux is not supported. To customize it, you would
most likely need to write your own custom policy. For guidance, you could refer to the
containers/container-selinux repository, which contains the SELinux policy les for Container
Runtimes, and the rancher/k3s-selinux repository, which contains the SELinux policy for K3s .

Additional preparation for (Red

Hat/CentOS) Enterprise Linux
It is recommended to turn off rewalld:

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 systemctl disable firewalld --now

EDIT THIS PAGE

GET THE LATEST NEWS PRODUCTS COMMUNITY RESOURCES COMPANY

Rancher Events Rancher 2.x Docs About Us

email address
RKE Online Meetups Rancher 1.6 Docs Customers
k3s Training RKE Docs Partners
Read Privacy Policy.
Get Support Tutorials k3s Docs Careers
Pricing Kubernetes Master RancherOS Docs Blog
Classes
Request a Demo eBooks and White Press
Get Certi ed! Papers
CONNECT WITH US Contact Us
Rancher Writing Videos and Podcasts
Program China Site
Getting Started Guide

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 Forums Why Kubernetes Japan Site
Slack

GitHub

© Copyright 2021 Rancher. All Rights Reserved. Privacy Policy Agreements Terms of Service

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD