OpenShift User Provisioned Method

Machine requirements for a cluster with user-provisioned infrastructure

For a cluster that contains user-provisioned infrastructure, you must deploy all of the
required machines.
Required machines

The smallest OpenShift Container Platform clusters require the following hosts:

One temporary bootstrap machine

Three control plane, or master, machines

At least two compute machines, which are also known as worker machines. If you are
running a three-node cluster, running zero compute machines is supported. Running one
compute machine is not supported.
The bootstrap and control plane machines must use Red Hat Enterprise Linux CoreOS
(RHCOS) as the operating system.

Network connectivity requirements

All the Red Hat Enterprise Linux CoreOS (RHCOS) machines require network in initramfs
during boot to fetch Ignition config files from the Machine Config Server. During the initial
boot, the machines require either a DHCP server or that static IP addresses be set in order
to establish a network connection to download their Ignition config files. Additionally, each
OpenShift Container Platform node in the cluster must have access to a Network Time
Protocol (NTP) server. If a DHCP server provides NTP servers information, the chrony
time service on the Red Hat Enterprise Linux CoreOS (RHCOS) machines read the
information and can sync the clock with the NTP servers.

Minimum resource requirements

Each cluster machine must meet the following minimum requirements:

Storag
Machine Operating System vCPU [1] Virtual RAM
e
Bootstrap RHCOS 4 16 GB 120 GB

Control
RHCOS 4 16 GB 120 GB
plane

RHCOS or RHEL RHCOS or RHEL

Compute 2 8 GB
7.9 7.9

1 1 vCPU is equivalent to 1 physical core when simultaneous multithreading (SMT),

or hyperthreading, is not enabled. When enabled, use the following formula to
calculate the corresponding ratio: (threads per core × cores) × sockets = vCPUs.

Download Software
 Login to RedHat OpenShift Cluster Manager
 Select 'Create Cluster' from the 'Clusters' navigation menu
 Select 'RedHat OpenShift Container Platform'
 Select 'Run on Bare Metal'
 Download the following files:
 Openshift Installer for Linux
 Pull secret
 Command Line Interface for Linux and your workstations OS
 Red Hat Enterprise Linux CoreOS (RHCOS)
 rhcos-X.X.X-x86_64-metal.x86_64.raw.gz
 rhcos-X.X.X-x86_64-installer.x86_64.iso

Compute node setup

1 Use Red Hat Subscription Manager to register the provisioner node.

$ sudo subscription-manager register --username=<user> --password=<pass> --auto-
attach
$ sudo subscription-manager repos --enable=rhel-8-for-x86_64-appstream-rpms --
enable=rhel-8-for-x86_64-baseos-rpms

2 Configure networking.
$ export PUB_CONN=<baremetal_nic_name>
$ export PROV_CONN=<prov_nic_name>
$ sudo nohup bash -c "
nmcli con down \"$PROV_CONN\"
nmcli con down \"$PUB_CONN\"
nmcli con delete \"$PROV_CONN\"
nmcli con delete \"$PUB_CONN\"
# RHEL 8.1 appends the word \"System\" in front of the connection, delete in case it
exists
nmcli con down \"System $PUB_CONN\"
nmcli con delete \"System $PUB_CONN\"
nmcli connection add ifname provisioning type bridge con-name provisioning
nmcli con add type bridge-slave ifname \"$PROV_CONN\" master provisioning
nmcli connection add ifname baremetal type bridge con-name baremetal
nmcli con add type bridge-slave ifname \"$PUB_CONN\" master baremetal
pkill dhclient;dhclient baremetal
nmcli connection modify provisioning ipv6.addresses fd00:1101::1/64 ipv6.method
manual
nmcli con down provisioning
nmcli con up provisioning
"

Note: The ssh connection might disconnect after executing this step.
The IPv6 address can be any address as long as it is not routable via the baremetal
network.
Ensure that UEFI is enabled and UEFI PXE settings are set to the IPv6 protocol when
using IPv6 addressing.

3 Verify the connection bridges have been properly created.

$ sudo nmcli con show

NAME UUID TYPE DEVICE

baremetal 4d5133a5-8351-4bb9-bfd4-3af264801530 bridge baremetal
provisioning 43942805-017f-4d7d-a2c2-7cb3324482ed bridge provisioning
virbr0 d9bca40f-eee1-410b-8879-a2d4bb0465e7 bridge virbr0
bridge-slave-eno1 76a8ed50-c7e5-4999-b4f6-6d9014dd0812 ethernet eno1
bridge-slave-eno2 f31c3353-54b7-48de-893a-02d2b34c4736 ethernet eno2

4 Move the files downloaded from the RedHat Cluster Manager site to the
compute
scp ~/Downloads/openshift-install-linux.tar.gz ~/Downloads/openshift-client-linux.tar.gz
~/Downloads/rhcos-x.x.x-x86_64-installer.x86_64.iso root@{ocp4.imss_IP_address}:/root/

5 SSH to the compute node

ssh root@{ocp4.imss_IP_address}

6 Extract Client tools and copy them to /usr/local/bin

tar xvf openshift-client-linux.tar.gz

mv oc kubectl /usr/local/bin

7 Confirm Client Tools are working

kubectl version

oc version

8 Extract the OpenShift Installer

tar xvf openshift-install-linux.tar.gz

9 Update CentOS so we get the latest packages for each of the services we are
about to install

dnf update

10 OPTIONAL: Create a file '~/.vimrc' and paste the following (this helps with
editing in vim, particularly yaml files):

cat <<EOT >> ~/.vimrc

syntax on
set nu et ai sts=0 ts=2 sw=2 list hls
EOT

Update the preferred editor

export OC_EDITOR="vim"

export KUBE_EDITOR="vim"

11 Set a Static IP for OCP network interface nmtui-edit ens224 or edit

/etc/sysconfig/network-scripts/ifcfg-ens224

 Address: 172.16.1.185
 DNS Server: 127.0.0.1
 Search domain: imss.com
 Never use this network for default route
 Automatically connect
If changes arent applied automatically you can bounce the NIC with nmcli
connection down ens224 and nmcli connection up ens224

12 Setup firewalld

Create libvirt and public zones

nmcli connection modify ens224 connection.zone libvirt

nmcli connection modify ens192 connection.zone public

View zones:
firewall-cmd --get-active-zones

Set masquerading (source-nat) on the both zones.

So to give a quick example of source-nat - for packets leaving the external interface, which
in this case is ens192 - after they have been routed they will have their source address
altered to the interface address of ens192 so that return packets can find their way back to
this interface where the reverse will happen.
firewall-cmd --zone=public --add-masquerade --permanent
firewall-cmd --zone=libvirt --add-masquerade --permanent

Reload firewall config

firewall-cmd --reload

Check the current settings of each zone

firewall-cmd --list-all --zone=libvirt
firewall-cmd --list-all --zone=public

When masquerading is enabled so is ip forwarding which basically makes this host a

router. Check:
cat /proc/sys/net/ipv4/ip_forward

13 Install and configure BIND DNS

Install dns package

dnf install bind bind-utils -y

Apply configuration
copy named.conf file from folder
copy zone files from folder

Configure the firewall for DNS

firewall-cmd --add-port=53/udp --zone=libvirt --permanent
firewall-cmd --reload

Enable and start the service

systemctl enable named
systemctl start named
systemctl status named

At the moment DNS will still be pointing to the LAN DNS server. You can see
this by testing with dig imss.com.

Change the LAN nic (ens192) to use 127.0.0.1 for DNS AND ensure Ignore automatically
Obtained DNS parameters is ticked
nmtui-edit ens192

Restart Network Manager

systemctl restart NetworkManager

Confirm dig now sees the correct DNS results by using the DNS Server running locally
dig imss.com
# The following should return the answer ocp-bootstrap.lab.ocp.lan from the local server
dig -x 172.22.0.1

curl -v https://quay.io/v2/
vim /etc/sysconfig/named

OPTIONS="-4"

14 Install & configure DHCP

Install the DHCP Server

dnf install dhcp-server -y

Edit dhcpd.conf from the cloned git repo to have the correct mac address for each host
and copy the conf file to the correct location for the DHCP service to use

copy dhcpd.conf file from folder

Configure the Firewall

firewall-cmd --add-service=dhcp --zone=internal --permanent

firewall-cmd --reload

Enable and start the service

systemctl enable dhcpd
systemctl start dhcpd
systemctl status dhcpd

15 Install & configure Apache Web Server

Install Apache
dnf install httpd -y

Change default listen port to 8080 in httpd.conf

sed -i 's/Listen 80/Listen 0.0.0.0:8080/' /etc/httpd/conf/httpd.conf

Configure the firewall for Web Server traffic

firewall-cmd --add-port=8080/tcp --zone=libvirt --permanent
firewall-cmd --reload

Enable and start the service

systemctl enable httpd
systemctl start httpd
systemctl status httpd
Making a GET request to localhost on port 8080 should now return the default Apache
webpage
curl localhost:8080

16 Install & configure HAProxy

Install HAProxy
dnf install haproxy -y

Copy HAProxy config

copy haproxy.cfg file from folder to /etc/haproxy/haproxy.cfg

17 Configure the Firewall

Note: Opening port 9000 in the external zone allows access to HAProxy stats
that are useful for monitoring and troubleshooting. The UI can be accessed at:
http://{ocp-imss_IP_address}:9000/stats

firewall-cmd --add-port=6443/tcp --zone=libvirt --permanent # kube-api-server on control

plane nodes
firewall-cmd --add-port=6443/tcp --zone=public --permanent # kube-api-server on control
plane nodes
firewall-cmd --add-port=22623/tcp --zone=libvirt --permanent # machine-config server
firewall-cmd --add-service=http --zone=libvirt --permanent # web services hosted on
worker nodes
firewall-cmd --add-service=http --zone=public --permanent # web services hosted on
worker nodes
firewall-cmd --add-service=https --zone=libvirt --permanent # web services hosted on
worker nodes
firewall-cmd --add-service=https --zone=public --permanent # web services hosted on
worker nodes
firewall-cmd --add-port=9000/tcp --zone=public --permanent # HAProxy Stats
firewall-cmd --reload

Enable and start the service

systemctl enable haproxy

systemctl start haproxy
systemctl status haproxy
 18 Install and configure NFS for the OpenShift Registry. It is a requirement
to provide storage for the Registry, emptyDir can be specified if
necessary (Optional steps- openshift version 4.7 or below versions
need to follow)

Install NFS Server

dnf install nfs-utils -y

Create the Share

Check available disk space and its location df -h
mkdir -p /shares/registry
chown -R nobody:nobody /shares/registry
chmod -R 777 /shares/registry

Export the Share

echo "/shares/registry 172.22.0.0/24(rw,sync,root_squash,no_subtree_check,no_wdelay)"
> /etc/exports

exportfs -rv

Set Firewall rules:

firewall-cmd --zone=libvirt --add-service mountd --permanent

firewall-cmd --zone=libvirt --add-service rpc-bind --permanent
firewall-cmd --zone=libvirt --add-service nfs --permanent
firewall-cmd --reload

Enable and start the NFS related services

systemctl enable nfs-server rpcbind
systemctl start nfs-server rpcbind nfs-mountd

Generate and host install files

1 Generate an SSH key pair keeping all default options

ssh-keygen -t ed25519 -N '' -f ~/.ssh/id_rsa

Start the ssh-agent process as a background task:

$ eval "$(ssh-agent -s)"

Example output
Agent pid 31874

Add your SSH private key to the ssh-agent:

 $ ssh-add <path>/<file_name>

Example output
Identity added: /home/<you>/<path>/<file_name> (<computer_name>)

Specify the path and file name for your SSH private key, such as
~/.ssh/id_rsa

2 Create an install directory

mkdir ~/clusterconfig

3. Copy the install-config.yaml included in the clones repository to the install directory

copy install-config.yaml file from folder to ~/clusterconfig

4. Update the install-config.yaml with your own pull-secret and ssh key.

 Line 23 should contain the contents of your pull-secret.txt

 Line 24 should contain the contents of your '~/.ssh/id_rsa.pub'
vim ~/clusterconfig/install-config.yaml

5. Generate Kubernetes manifest files

~/openshift-install create manifests --dir ~/clusterconfig

Note:- A warning is shown about making the control plane nodes schedulable. It
is up to you if you want to run workloads on the Control Plane nodes. If you dont
want to you can disable this with:

sed -i 's/mastersSchedulable: true/mastersSchedulable: false/'

~/clusterconfig/manifests/cluster-scheduler-02-config.yml.

Make any other custom changes you like to the core Kubernetes manifest files.

Remove the Kubernetes manifest files that define the control plane machines
and compute machine sets:

$ rm -f openshift/99_openshift-cluster-api_master-machines-*.yaml
openshift/99_openshift-cluster-api_worker-machineset-*.yaml

Generate the Ignition config and Kubernetes auth files

~/openshift-install create ignition-configs --dir ~/clusterconfig/

The following files are generated in the directory:

.
 ├── auth
│ ├── kubeadmin-password
│ └── kubeconfig
├── bootstrap.ign
├── master.ign
├── metadata.json
└── worker.ign

6. Create a hosting directory to serve the configuration files for the OpenShift
booting process

mkdir /var/www/html/ocp4

7. Copy all generated install files to the new web server directory
cp -R ~/clusterconfig/* /var/www/html/ocp4

8. Move the Core OS image to the web server directory (later you need to type this
path multiple times so it is a good idea to shorten the name)
mv ~/rhcos-metal.x86_64.raw.gz /var/www/html/ocp4/rhcos.raw.gz //not needed

9. Change ownership and permissions of the web server directory

chown -R apache: /var/www/html/ocp4/
chmod 755 /var/www/html/ocp4/

10. Confirm you can see all files added to the /var/www/html/ocp4/ dir through
Apache
curl localhost:8080/ocp4/

Deploy OpenShift

Power on the ocp-bootstrap host and ocp-cp-# hosts and select 'Tab' to enter boot
configuration. Enter the following configuration:

$ sudo coreos-installer install devsda –ignition-url

http://172.16.3.36:8080/ocp4/bootstrap.ign --copy-network --insecure-ignition

ip=172.22.0.96::172.22.0.1:255.255.255.0:bootstrap.ocp4.imss.com:enp1s0:none
nameserver=172.22.0.1 coreos.inst.install_dev=vda
coreos.inst.ignition_url=http://172.22.0.1:8080/ocp4/bootstrap.ign coreos.inst.insecure=yes
coreos.inst.image_url=http://172.22.0.1:8080/ocp4/rhcos
Monitor the Bootstrap Process

1 You can monitor the bootstrap process from the ocp-svc host at different log levels
(debug, error, info)
run below command in bationhost

~/openshift-install --dir ~/clusterconfig wait-for bootstrap-complete --log-level=debug

2.Once bootstrapping is completed add the master nodes

check bootstracp process

sudo crictl images

sudo crictl ps

For Master node

ip=172.22.0.97::172.22.0.1:255.255.255.0:master0.ocp4.imss.com:enp1s0:none
nameserver=172.22.0.1 coreos.inst.install_dev=vda
coreos.inst.ignition_url=http://172.22.0.1:8080/ocp4/master.ign coreos.inst.insecure=yes
coreos.inst.image_url=http://172.22.0.1:8080/ocp4/rhcos.raw.gz

Once bootstrapping is complete the ocp-boostrap node can be removed

Remove the Bootstrap Node

1 Remove all references to the ocp-bootstrap host from the /etc/haproxy/haproxy.cfg

file
# Two entries
vim /etc/haproxy/haproxy.cfg
# Restart HAProxy - If you are still watching HAProxy stats console you will see that
the ocp-boostrap host has been removed from the backends.
systemctl reload haproxy

2 The ocp-bootstrap host can now be safely shutdown and deleted from the libvirt
Console, the host is no longer required
Wait for installation to complete

IMPORTANT: if you set mastersSchedulable to false the worker nodes will need
to be joined to the cluster to complete the installation. This is because the
OpenShift Router will need to be scheduled on the worker nodes and it is a
dependency for cluster operators such as ingress, console and authentication.

1 Collect the OpenShift Console address and kubeadmin credentials from the output
of the install-complete event
~/openshift-install --dir ~/clusterconfig wait-for install-complete

2 Continue to join the worker nodes to the cluster in a new tab whilst waiting for the
above command to complete

For Worker node

ip=172.22.0.11::172.22.0.1:255.255.255.0:worker0.ocp4.imss.com:enp1s0:none
nameserver=172.22.0.1 coreos.inst.install_dev=vda
coreos.inst.ignition_url=http://172.22.0.1:8080/ocp4/worker.ign
coreos.inst.insecure=yes
coreos.inst.image_url=http://172.22.0.1:8080/ocp4/rhcos.raw.gz

Join Worker Nodes

1 Setup 'oc' and 'kubectl' clients on the ocp-svc machine

export KUBECONFIG=~/clusterconfig/auth/kubeconfig

# Test auth by viewing cluster nodes

oc get nodes

2 View and approve pending CSRs

Note: Once you approve the first set of CSRs additional 'kubelet-serving'
CSRs will be created. These must be approved too. If you do not see
pending requests wait until you do.

# View CSRs

oc get csr

# Approve all pending CSRs

oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}

{{end}}{{end}}' | xargs oc adm certificate approve

# Wait for kubelet-serving CSRs and approve them too with the same command
 oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}
{{end}}{{end}}' | xargs oc adm certificate approve

3 Watch and wait for the Worker Nodes to join the cluster and enter a 'Ready' status
This can take 5-10 minutes

watch -n5 oc get nodes

Note: Rescheduling ingress pods to worker nodes- unscheduling master nodes

1. Configure control plane nodes (also known as the master nodes) to be unschedulable
(which is recommended) using the following procedure[1]:

oc edit schedulers.config.openshift.io cluster

~~~
mastersSchedulable: false
~~~

2. Delete the ingress pods to be rescheduled on the worker node:

~~~
$ oc delete pods --all -n openshift-ingress
~~~

3. Check on which nodes the ingress pod is scheduled:

~~~
$ oc get pod -n openshift-ingress -o wide

Scale to ingress pod to 3 so that route pods will be available on all three workers[2]:

~~~
$ oc patch -n openshift-ingress-operator ingresscontroller/default --patch
'{"spec":{"replicas": 3}}' --type=merge
~~~

Wait for few minutes and check back the status of the operator.

~~~
$ oc get co
~~~

Feel free to update the case at any time.

[1] https://docs.openshift.com/container-platform/4.8/nodes/nodes/nodes-nodes-
working.html#nodes-nodes-working-master-schedulable_nodes-nodes-working

[2] https://docs.openshift.com/container-platform/4.8/networking/ingress-
operator.html#nw-ingress-controller-configuration_configuring-ingress
Configure storage for the Image Registry

A Bare Metal cluster does not by default provide storage so the Image Registry
Operator bootstraps itself as 'Removed' so the installer can complete. As the
installation has now completed storage can be added for the Registry and the
operator updated to a 'Managed' state.

1 Create the 'image-registry-storage' PVC by updating the Image Registry operator

config by updating the management state to 'Managed' and adding 'pvc' and 'claim'
keys in the storage key:
oc edit configs.imageregistry.operator.openshift.io

managementState: Managed

storage:
pvc:
claim: # leave the claim blank

2 Confirm the 'image-registry-storage' pvc has been created and is currently in a

'Pending' state
oc get pvc -n openshift-image-registry

3 Create the persistent volume for the 'image-registry-storage' pvc to bind to

oc create -f ~/path/manifest/registry-pv.yaml
 4 After a short wait the 'image-registry-storage' pvc should now be bound
oc get pvc -n openshift-image-registry

Create the first Admin user

1 Apply the oauth-htpasswd.yaml file to the cluster

This will create a user 'admin' with the password 'password'. To set a
different username and password substitue the htpasswd key in the
'~/ocp4-metal-install/manifest/oauth-htpasswd.yaml' file with the output of
htpasswd -n -B -b <username> <password>

oc apply -f ~/path/manifest/oauth-htpasswd.yaml

2 Assign the new user (admin) admin permissions

oc adm policy add-cluster-role-to-user cluster-admin admin

Access the OpenShift Console

1 Wait for the 'console' Cluster Operator to become available
oc get co

2 Append the following to your local workstations /etc/hosts file:

From your local workstation If you do not want to add an entry for each
new service made available on OpenShift you can configure the ocp-svc
DNS server to serve externally and create a wildcard entry for
*.apps.lab.ocp.lan

# Open the hosts file

sudo vi /etc/hosts

# Append the following entries:

172.16.1.185 ocp4.imss.com api.ocp4.imss.com console-openshift-
console.apps.ocp4.imss.com oauth-openshift.apps.ocp4.imss.com downloads-
openshift-console.apps.ocp4.imss.com alertmanager-main-openshift-
monitoring.apps.ocp4.imss.com grafana-openshift-monitoring.apps.ocp4.imss.com
prometheus-k8s-openshift-monitoring.apps.ocp4.imss.com thanos-querier-
openshift-monitoring.apps.ocp4.imss.com

3 Navigate to the OpenShift Console URL and log in as the 'admin' user
You will get self signed certificate warnings that you can ignore If you
need to login as kubeadmin and need to the password again you can
retrieve it with: cat ~/authconfig/auth/kubeadmin-password
Changing worker tags to infra:-

Below command to add infra tag

oc label nodes vlnfedevinfra1.nfeokddev.accelya.local

vlnfedevinfra2.nfeokddev.accelya.local vlnfedevinfra3.nfeokddev.accelya.local node-
role.kubernetes.io/infra=""

Below command to remove worker tag

oc label nodes vlnfedevinfra1.nfeokddev.accelya.local

vlnfedevinfra2.nfeokddev.accelya.local vlnfedevinfra3.nfeokddev.accelya.local node-
role.kubernetes.io/worker-

Below command to check the tag changes

oc get nodes

Below command to check the mcp status

oc get mcp

Reschedule ingrace to load pods on infra nodes

Edit the below yaml

oc get ingresscontroller default -n openshift-ingress-operator -o yaml

Add the below conntents in the yaml

nodePlacement:

nodeSelector:

matchLabels:

node-role.kubernetes.io/infra: ""

998 htpasswd -Bb $HOME/htpasswd kartheek kartheek@123

999 oc create secret generic htpasswd --from-file=htpasswd=htpasswd --dry-run

=client -o yaml -n openshift-config | oc replace -f -

1000 oc adm policy add-cluster-role-to-user cluster-admin kartheek

To manually approve certificate individually
oc adm certificate approve <csr_name>

To approve certificate all at once

oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs --no-
run-if-empty oc adm certificate approve