OpenShift 4.

x Troubleshooting Quick
Reference
Updated July 11 2020 at 8:59 AM - 
English 

This article is not meant to be a clone of the kubernetes cheat sheet. You should use
this in conjunction with that cheat sheet, however, you are likely to find this document
more concise.

Hosts
You will use these commands on a host, in the cluster often on a master. To run them
you will need SSH access to the host, and root (sudo) permissions.

 If you have a functioning api you can get system logs via a shim

Raw

$ oc adm node-logs --role=<master|worker> -u <systemd_unit>

$ oc adm node-logs --role=master --path=<path_under_/var/log>

 Note: All commands listed require ssh access to the host.

Raw

$ grep OSTREE_VERSION /etc/os-release

$ rpm-ostree version

$ sudo journalctl -b -f -ukubelet.service -u crio.service

$ sudo crictl pods
$ sudo crictl logs <pod>
$ sudo tail -f /var/log/containers/*

 Commands below, are only relevant at install time (on the bootstrap host)

Raw

$ sudo journalctl -b -f -u bootkube.service

$ sudo podman logs <pod>
$ sudo tail -f
/var/lib/containers/storage/overlay-containers/*/userdata/ctr.log
General Cluster Health and Version Information
These commands can be run from anywhere, so long as you have oc (and in some
cases using kubectl may work), provided you have permissions on the cluster to read
the data you are requesting.

 Use oc adm must-gather to collect information about the state of your cluster.
o This command (and all oc adm commands) requires admin privileges to
run.

Raw
$ oc get clusterversion
$ oc get clusteroperators
$ oc adm release info $(oc get clusterversion -o
jsonpath='{.items[].status.current.payload}{"\n"}')

Applications
These commands can be run from anywhere, so long as you have oc (and in some
cases using kubectl may work), provided you have permissions on the cluster to read
the data you are requesting.

Raw
$ oc api-resources --verbs=list --namespaced -o name
$ oc get deployments,daemonsets,pods,configmaps,services -n <namespace> -o yaml
$ oc describe <object> -n <namespace>
$ oc logs <pod> -n <namespace>

 Use component endpoint debugging, to open a port to these critical endpoints.

Raw
$ oc get --insecure-skip-tls-verify --server=https://localhost:<port> --raw=/version
$ oc get --insecure-skip-tls-verify --server=https://localhost:<port> --raw=/healthz
$ oc get --insecure-skip-tls-verify --server=https://localhost:<port> --raw=/metrics