Setup a K8S on OLE9

configure minio multi tenant on k8s

Mục Lục
I. Phạm vi tài liệu .......................................................................................................... 3
I.1 Nội dung tài liệu ................................................................................................... 3
I.2 Phạm vi tài liệu .................................................................................................... 3
I.3 Giới hạn tài liệu ................................................................................................... 3
I.4 Giới hạn người đọc ............................................................................................. 3
II. Cấu hình cơ bản node K8S trên linux.................................................................... 3
II.1 Thông tin cấu hình cơ bản ................................................................................ 3
II.2 Thông tin file repo trên các node ...................................................................... 3
II.3 Chuẩn bị môi trường cài đặt trên tất cả các node ................................................. 4
II.3.1 Cấu hình disk trên các node .......................................................................... 4
II.3.2 Cấu hình ntp client trên các node .................................................................. 5
II.3.3 Cấu hình file /etc/hosts trên các node ........................................................... 5
II.3.4 Cài đặt các gói phần mềm trên các node ....................................................... 5
II.3.5 Cấu hình cơ bản trên các node ...................................................................... 5
II.3.6 Cài đặt gói rke trên node master 01 .............................................................. 6
II.3.7. Cấu hình passwordless cho ssh .................................................................... 6
II.3.8. Cài đặt helm trên node master 01 ................................................................. 6
II.3.9. Tạo file cluster.yml ....................................................................................... 6
II.3.10. Thực hiện deploy k8s ............................................................................... 11
II.3.11. Kiểm tra node k8s đã cài đặt .................................................................... 11
II.3.12. Cài đặt giao diện rancher cho K8S........................................................... 12
III. Cài đặt, cấu hình minio multi tenant trên K8S ................................................. 12
III.1. Cài đặt krew plugin cho k8s............................................................................. 12
III.2. Cài đặt minio direct pv ..................................................................................... 12
III.3. Cài đặt minio-operator ..................................................................................... 12
III.4. Cấu hình ingress cho minio-operator ............................................................... 13
III.5. Cấu hình minio tenant trên giao diện minio-operator...................................... 14
I. Phạm vi tài liệu
I.1 Nội dung tài liệu
Tài liệu hướng dẫn các bước cài đặt Cụm K8S on Linux phục vụ “…”

I.2 Phạm vi tài liệu

Tài liệu này chỉ áp dụng cho phần mềm K8S on Linux 03 node master và multi node worker

I.3 Giới hạn tài liệu

Tài liệu này không bao gồm các nội dung ngoài việc cài đặt K8S trên Linux

I.4 Giới hạn người đọc

Tài liệu này là tài liệu phục vụ “…”, dành cho người đọc có chuyên môn và phụ trách hệ thống liên
quan đến K8S on Linux 03 node master và multi node worker

II. Cấu hình cơ bản node K8S trên linux

II.1 Thông tin cấu hình cơ bản
STT Tên VM trên vCenter IP sdb, sql data Hostname OS Version
sdc
01 K8S-MASTER-01-163 172.16.123.162 100G /var/lib/docker k8s-master01 OLE 9.3
02 K8S-MASTER-02-165 172.16.123.163 100G /var/lib/docker k8s-master02 OLE 9.3
03 K8S-MASTER-03-167 172.16.123.164 100G /var/lib/docker k8s-master03 OLE 9.3
04 K8S-WORKER-01-162 172.16.123.165 100G /var/lib/docker k8s-worker01 OLE 9.3
500G raw for directpv
05 K8S-WORKER-02-164 172.16.123.166 100G /var/lib/docker k8s-worker02 OLE 9.3
500G raw for directpv
06 K8S-WORKER-03-166 172.16.123.167 100G /var/lib/docker k8s-worker03 OLE 9.3
500G raw for directpv
07 K8S-WORKER-04-167 172.16.123.168 100G /var/lib/docker k8s-worker04 OLE 9.3
500G raw for directpv
08 K8S-WORKER-05-168 172.16.123.169 100G /var/lib/docker k8s-worker05 OLE 9.3
500G raw for directpv
09 K8S-WORKER-06-169 172.16.123.170 100G /var/lib/docker k8s-worker06 OLE 9.3
500G raw for directpv
10 K8S-WORKER-07-170 172.16.123.171 100G /var/lib/docker k8s-worker07 OLE 9.3
500G raw for directpv
11 K8S-WORKER-08-171 172.16.123.172 100G /var/lib/docker k8s-worker08 OLE 9.3
500G raw for directpv

Các node sẽ tắt selinux và firewalld để thực hiện cài đặt như dưới đây

II.2 Thông tin file repo trên các node

https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-
kubeadm/
sudo setenforce 0
sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config
cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.29/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.29/rpm/repodata/repomd.xml.key
exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni
EOF

sudo yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes

sudo systemctl enable --now kubelet

echo -e
'br_netfilter\nip6_udp_tunnel\nip_set\nip_set_hash_ip\nip_set_hash_net\niptable_f
ilter\niptable_nat\niptable_mangle\niptable_raw\nnf_conntrack_netlink\nnf_conntra
ck\nnf_conntrack_ipv4\nnf_defrag_ipv4\nnf_nat\nn
f_nat_ipv4\nnf_nat_masquerade_ipv4\nnfnetlink\nudp_tunnel\nveth\nvxlan
\nx_tables\nxt_addrtype\nxt_conntrack\nxt_comment\nxt_mark\nxt_multiport\nxt_nat\
nxt_recent\nxt_set\nxt_statistic\nxt_tcpudp' > /etc/modules-load.d/rke.conf [

echo "net.bridge.bridge-nf-call-iptables = 1" >> /etc/sysctl.d/rke.conf

echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.d/rke.conf
sysctl -p /etc/sysctl.d/rke.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-iptables=1
NOTE: tham chiếu tài liệu https://medium.com/@osmarrleao/installing-rke-cluster-on-enterprise-
linux-d4cded73fa84
II.3 Chuẩn bị môi trường cài đặt trên tất cả các node
II.3.1 Cấu hình disk trên các node
Kiểm tra disk trên hệ thống bằng

# lsblk

# lvmdiskscan

Thực hiện tạo lvm cho vùng chứa docker, mục đích tách riêng vùng disk sinh nhiều dữ liệu và os

# pvcreate /dev/sdb

# vgcreate vg0 /dev/sdb

# lvcreate -l 100%FREE -n docker vg0

# mkfs.ext4 /dev/mapper/vg0-docker

# echo "/dev/mapper/vg0-docker /var/lib/docker ext4 defaults 1 2" >> /etc/fstab

# mkdir -p /var/lib/docker

# systemctl daemon-reload

# mount -a

Kiểm tra disk sau khi cấu hình

# df -h
II.3.2 Cấu hình ntp client trên các node
Thực hiện cài đặt các gói cần thiết trên các node, hệ thống này mục tiêu cài minio
# yum install -y kubelet kubeadm kubectl wget net-tools chrony git

# systemctl start chronyd

# systemctl enable chronyd

# timedatectl

II.3.3 Cấu hình file /etc/hosts trên các node

# cat /etc/hosts

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4

#::1 localhost localhost.localdomain localhost6 localhost6.localdomain6

172.16.123.162 k8s-master01

172.16.123.163 k8s-master02

172.16.123.164 k8s-master03

172.16.123.165 k8s-worker01

172.16.123.166 k8s-worker02

172.16.123.167 k8s-worker03

172.16.123.168 k8s-worker04

172.16.123.169 k8s-worker05

172.16.123.170 k8s-worker06

172.16.123.171 k8s-worker07

172.16.123.172 k8s-worker08

II.3.4 Cài đặt các gói phần mềm trên các node
# curl https://releases.rancher.com/install-docker/23.0.sh | sh

# curl https://releases.rancher.com/install-docker/24.0.sh | sh

INFO: Searching repository for VERSION '24.0.7'

INFO: yum list --showduplicates docker-ce | grep '24.0.7.*el' | tail -1 | awk '{print $2}'

+ sh -c 'yum install -y -q docker-ce-24.0.7-1.el9 docker-ce-cli-24.0.7-1.el9 containerd.io docker-compose-plugin docker-

ce-rootless-extras-24.0.7-1.el9 docker-buildx-plugin'

# yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes

II.3.5 Cấu hình cơ bản trên các node

# systemctl start docker

# systemctl enable docker

II.3.6 Cài đặt gói rke trên node master 01
Tạo thư mục /opt/rke

# mkdir -p /opt/rke

# wget https://github.com/rancher/rke/releases/download/v1.4.11/rke_linux-amd64

# chmod +x rke_linux-amd64

# mv rke_linux-amd64 /usr/local/bin/rke

# rke –version

II.3.7. Cấu hình passwordless cho ssh

# ssh-keygen -t rsa

# chmod 400 ~/.ssh/id_rsa.pub

Thực hiện copy ssh key sang

# ssh-copy-id k8s-master01

Kiểm tra ssh passwordless bằng

# ssh k8s-master01

Các node khác làm tương tự

# ssh-copy-id k8s-master02

# ssh-copy-id k8s-master03

# ssh-copy-id k8s-worker01

# ssh-copy-id k8s-worker02

# ssh-copy-id k8s-worker03

II.3.8. Cài đặt helm trên node master 01

# curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3

# chmod 700 get_helm.sh

./get_helm.sh

II.3.9. Tạo file cluster.yml

Thực hiện lệnh
# rke config
output console
[root@k8s-master01 ~]# rke config

[+] Cluster Level SSH Private Key Path [~/.ssh/id_rsa]:

[+] Number of Hosts [1]: 2 #theo số lượng server cần cài đặt

[+] SSH Address of host (1) [none]: 172.16.123.163 # ip master, server làm lần lượt theo form 1 sau đó mới chuyển sang
form 2 cho worker

[+] SSH Port of host (1) [22]:

[+] SSH Private Key Path of host (172.16.123.163) [none]:

[-] You have entered empty SSH key path, trying fetch from SSH key parameter
[+] SSH Private Key of host (172.16.123.163) [none]:

[-] You have entered empty SSH key, defaulting to cluster level SSH key: ~/.ssh/id_rsa

[+] SSH User of host (172.16.123.163) [ubuntu]: root

[+] Is host (172.16.123.163) a Control Plane host (y/n)? [y]: y

[+] Is host (172.16.123.163) a Worker host (y/n)? [n]: n

[+] Is host (172.16.123.163) an etcd host (y/n)? [n]: y

[+] Override Hostname of host (172.16.123.163) [none]: k8s-master01 #set theo hostname server

[+] Internal IP of host (172.16.123.163) [none]:

[+] Docker socket path on host (172.16.123.163) [/var/run/docker.sock]:

[+] SSH Address of host (2) [none]: 172.16.123.162 #Ip server worker

[+] SSH Port of host (2) [22]:

[+] SSH Private Key Path of host (172.16.123.162) [none]:

[-] You have entered empty SSH key path, trying fetch from SSH key parameter

[+] SSH Private Key of host (172.16.123.162) [none]:

[-] You have entered empty SSH key, defaulting to cluster level SSH key: ~/.ssh/id_rsa

[+] SSH User of host (172.16.123.162) [ubuntu]: root

[+] Is host (172.16.123.162) a Control Plane host (y/n)? [y]: n

[+] Is host (172.16.123.162) a Worker host (y/n)? [n]: y

[+] Is host (172.16.123.162) an etcd host (y/n)? [n]: n

[+] Override Hostname of host (172.16.123.162) [none]: k8s-worker01

[+] Internal IP of host (172.16.123.162) [none]:

[+] Docker socket path on host (172.16.123.162) [/var/run/docker.sock]:

[+] Network Plugin Type (flannel, calico, weave, canal, aci) [canal]: calico

[+] Authentication Strategy [x509]:

[+] Authorization Mode (rbac, none) [rbac]:

[+] Kubernetes Docker image [rancher/hyperkube:v1.26.9-rancher1]:

[+] Cluster domain [cluster.local]: cluster.local #set theo domain tuỳ chọn

[+] Service Cluster IP Range [10.43.0.0/16]:

[+] Enable PodSecurityPolicy [n]:

[+] Cluster Network CIDR [10.42.0.0/16]:

[+] Cluster DNS Service IP [10.43.0.10]:

[+] Add addon manifest URLs or YAML files [no]:

Tiếp đến sửa config theo from mẫu sau

# If you intended to deploy Kubernetes in an air-gapped environment,

# please consult the documentation on how to configure custom RKE images.
nodes:
- address: 172.16.123.162
 port: "22"
internal_address: 172.16.124.162
role:
- controlplane
- etcd
hostname_override: ""
user: root
docker_socket: /var/run/docker.sock
ssh_key: ""
ssh_key_path: ~/.ssh/id_rsa
ssh_cert: ""
ssh_cert_path: ""
labels: {}
- address: 172.16.123.165
port: "22"
internal_address: 172.16.124.165
role:
- worker
hostname_override: ""
user: root
docker_socket: /var/run/docker.sock
ssh_key: ""
ssh_key_path: ~/.ssh/id_rsa
ssh_cert: ""
ssh_cert_path: ""
labels: {}
taints: []
services:
etcd:
image: ""
extra_args: {}
extra_args_array: {}
extra_binds: []
extra_env: []
win_extra_args: {}
win_extra_args_array: {}
win_extra_binds: []
win_extra_env: []
external_urls: []
ca_cert: ""
cert: ""
key: ""
path: ""
uid: 0
gid: 0
snapshot: null
retention: ""
creation: ""
backup_config: null
kube-api:
image: ""
extra_args: {}
extra_args_array: {}
extra_binds: []
extra_env: []
win_extra_args: {}
win_extra_args_array: {}
win_extra_binds: []
win_extra_env: []
service_cluster_ip_range: 10.43.0.0/16
service_node_port_range: ""
pod_security_policy: false
 pod_security_configuration: ""
always_pull_images: false
secrets_encryption_config: null
audit_log: null
admission_configuration: null
event_rate_limit: null
kube-controller:
image: ""
extra_args: {}
extra_args_array: {}
extra_binds: []
extra_env: []
win_extra_args: {}
win_extra_args_array: {}
win_extra_binds: []
win_extra_env: []
cluster_cidr: 10.42.0.0/16
service_cluster_ip_range: 10.43.0.0/16
scheduler:
image: ""
extra_args: {}
extra_args_array: {}
extra_binds: []
extra_env: []
win_extra_args: {}
win_extra_args_array: {}
win_extra_binds: []
win_extra_env: []
kubelet:
image: ""
extra_args:
max-pods: 500
extra_args_array: {}
extra_binds: []
extra_env: []
win_extra_args: {}
win_extra_args_array: {}
win_extra_binds: []
win_extra_env: []
cluster_domain: cluster.local
infra_container_image: ""
cluster_dns_server: 10.43.0.10
fail_swap_on: false
generate_serving_certificate: false
kubeproxy:
image: ""
extra_args: {}
extra_args_array: {}
extra_binds: []
extra_env: []
win_extra_args: {}
win_extra_args_array: {}
win_extra_binds: []
win_extra_env: []
network:
plugin: calico
options: {}
mtu: 0
node_selector: {}
update_strategy: null
tolerations: []
authentication:
 strategy: x509
sans: []
webhook: null
addons: ""
addons_include: []
system_images:
etcd: rancher/mirrored-coreos-etcd:v3.5.6
alpine: rancher/rke-tools:v0.1.96
nginx_proxy: rancher/rke-tools:v0.1.96
cert_downloader: rancher/rke-tools:v0.1.96
kubernetes_services_sidecar: rancher/rke-tools:v0.1.96
kubedns: rancher/mirrored-k8s-dns-kube-dns:1.22.20
dnsmasq: rancher/mirrored-k8s-dns-dnsmasq-nanny:1.22.20
kubedns_sidecar: rancher/mirrored-k8s-dns-sidecar:1.22.20
kubedns_autoscaler: rancher/mirrored-cluster-proportional-autoscaler:1.8.6
coredns: rancher/mirrored-coredns-coredns:1.9.4
coredns_autoscaler: rancher/mirrored-cluster-proportional-autoscaler:1.8.6
nodelocal: rancher/mirrored-k8s-dns-node-cache:1.22.20
kubernetes: rancher/hyperkube:v1.26.9-rancher1
flannel: rancher/mirrored-flannel-flannel:v0.21.4
flannel_cni: rancher/flannel-cni:v0.3.0-rancher8
calico_node: rancher/mirrored-calico-node:v3.25.0
calico_cni: rancher/calico-cni:v3.25.0-rancher1
calico_controllers: rancher/mirrored-calico-kube-controllers:v3.25.0
calico_ctl: rancher/mirrored-calico-ctl:v3.25.0
calico_flexvol: rancher/mirrored-calico-pod2daemon-flexvol:v3.25.0
canal_node: rancher/mirrored-calico-node:v3.25.0
canal_cni: rancher/calico-cni:v3.25.0-rancher1
canal_controllers: rancher/mirrored-calico-kube-controllers:v3.25.0
canal_flannel: rancher/mirrored-flannel-flannel:v0.21.4
canal_flexvol: rancher/mirrored-calico-pod2daemon-flexvol:v3.25.0
weave_node: weaveworks/weave-kube:2.8.1
weave_cni: weaveworks/weave-npc:2.8.1
pod_infra_container: rancher/mirrored-pause:3.7
ingress: rancher/nginx-ingress-controller:nginx-1.7.0-rancher1
ingress_backend: rancher/mirrored-nginx-ingress-controller-defaultbackend:1.5-rancher1
ingress_webhook: rancher/mirrored-ingress-nginx-kube-webhook-certgen:v20230312-helm-
chart-4.5.2-28-g66a760794
metrics_server: rancher/mirrored-metrics-server:v0.6.3
windows_pod_infra_container: rancher/mirrored-pause:3.7
aci_cni_deploy_container: noiro/cnideploy:6.0.3.1.81c2369
aci_host_container: noiro/aci-containers-host:6.0.3.1.81c2369
aci_opflex_container: noiro/opflex:6.0.3.1.81c2369
aci_mcast_container: noiro/opflex:6.0.3.1.81c2369
aci_ovs_container: noiro/openvswitch:6.0.3.1.81c2369
aci_controller_container: noiro/aci-containers-controller:6.0.3.1.81c2369
aci_gbp_server_container: ""
aci_opflex_server_container: ""
ssh_key_path: ~/.ssh/id_rsa
ssh_cert_path: ""
ssh_agent_auth: false
authorization:
mode: rbac
options: {}
ignore_docker_version: null
enable_cri_dockerd: null
kubernetes_version: ""
private_registries: []
ingress:
provider: ""
options: {}
node_selector: {}
 extra_args: {}
dns_policy: ""
extra_envs: []
extra_volumes: []
extra_volume_mounts: []
update_strategy: null
http_port: 0
https_port: 0
network_mode: ""
tolerations: []
default_backend: null
default_http_backend_priority_class_name: ""
nginx_ingress_controller_priority_class_name: ""
default_ingress_class: null
cluster_name: ""
cloud_provider:
name: ""
prefix_path: ""
win_prefix_path: ""
addon_job_timeout: 0
bastion_host:
address: ""
port: ""
user: ""
ssh_key: ""
ssh_key_path: ""
ssh_cert: ""
ssh_cert_path: ""
ignore_proxy_env_vars: false
monitoring:
provider: ""
options: {}
node_selector: {}
update_strategy: null
replicas: null
tolerations: []
metrics_server_priority_class_name: ""
restore:
restore: false
snapshot_name: ""
rotate_encryption_key: false
dns: null
Thực hiện lưu file config
II.3.10. Thực hiện deploy k8s
# rke up

# mv kube_config_cluster.yml ~/.kube/config

II.3.11. Kiểm tra node k8s đã cài đặt

# kubectl get node
NAME STATUS ROLES AGE VERSION

172.16.123.162 Ready worker 13d v1.26.9

172.16.123.163 Ready controlplane,etcd 13d v1.26.9

172.16.123.164 Ready worker 13d v1.26.9

172.16.123.165 Ready controlplane,etcd 13d v1.26.9

172.16.123.166 Ready worker 13d v1.26.9

172.16.123.167 Ready controlplane,etcd 13d v1.26.9

II.3.12. Cài đặt giao diện rancher cho K8S

add repo charts
# helm repo add rancher-latest https://releases.rancher.com/server-charts/latest

# helm repo add jetstack https://charts.jetstack.io

add cert-manager CRD

# kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.12.2/cert-manager.crds.yaml

# helm upgrade -i cert-manager jetstack/cert-manager --namespace cert-manager --create-namespace

# helm upgrade -i rancher rancher-latest/rancher \

--create-namespace --namespace cattle-system \

--set hostname="rancher.cluster.local" \

--set bootstrapPassword="admin@123" \

--set replicas=1

III. Cài đặt, cấu hình minio multi tenant trên K8S
III.1. Cài đặt krew plugin cho k8s
NOTE: Cài trên 1 master, toàn bộ worker để chạy minio

Chuẩn bị môi trường bằng cách cài các gói phần mềm

# yum install -y kubelet kubeadm kubectl wget net-tools wget chrony git

https://krew.sigs.k8s.io/docs/user-guide/setup/install/

III.2. Cài đặt minio direct pv

NOTE: Cài trên 1 master, toàn bộ worker để chạy minio

NOTE: Tài liệu gốc online tham chiếu

https://github.com/minio/directpv?tab=readme-ov-file

[root@worker1 ~]# kubectl krew update

[root@worker1 ~]# kubectl krew install directpv

[root@worker1 ~]# kubectl directpv install

[root@worker1 ~]# kubectl directpv discover

[root@worker1 ~]# kubectl directpv init /root/drives.yaml --dangerous

III.3. Cài đặt minio-operator

NOTE: Tài liệu gốc online tham chiếu

https://min.io/docs/minio/kubernetes/upstream/operations/installation.html

kubectl krew update

kubectl krew install minio
kubectl minio init
helm repo add minio-operator https://operator.min.io
helm search repo minio-operator
helm install \
--namespace minio-operator \
--create-namespace \
operator minio-operator/operator
kubectl get all -n minio-operator
III.4. Cấu hình ingress cho minio-operator
Thiết lập file hosts với tên miền minio-testlab.local tương ứng với ip worker
thực hiện cấu hình trên web quản trị rancher k8s như dưới
III.5. Cấu hình minio tenant trên giao diện minio-operator
login vào giao diện web của minio cần jwt bằng lệnh

# kubectl minio proxy

đăng nhập vào