CBIS Resources not working Properly /

Dentry Cache
(CBIS release 19/19A)

Customer: SETAR

SF: 02838221

pg. 1
Difficulty Level: Easy

Approximate Completion Time: 60 minutes

Version: 1.0

Contents
Summary ..................................................................................................................................... 3
A. Pre-Health check .................................................................................................................. 3
B. Cleanup PCS resources .......................................................................................................... 4
C. Barbican API Healthcheck ..................................................................................................... 4
D. Limit Negative Dentry Size .................................................................................................... 4
E. Rollback ............................................................................................................................... 5
F. Post-Health check ................................................................................................................. 5

pg. 2
Summary
This MOP covers these procedural actions for CBIS 19:
• Modification of /openstack/healthcheck on each controller
• dropping inode/dentry cache on each controller

A. Pre-Health check
1. Hardware status of compute and controller. Make sure Power State should be on for all,
Provisioning State should be active and Maintenance mode should false

[stack@undercloud ~]$ openstack baremetal node list

2. Openstack Status: Undercloud ----> There should not be any failed Service.

[stack@undercloud ~]$ openstack-status

3. Nova Compute Services: Undercloud ----> Status should be enable and State should be UP

[stack@undercloud ~]$ source overcloudrc

[stack@undercloud ~]$ nova service-list

4. Tenant Status: Undercloud ----> All Tenants should be in ACTIVE State (Leave VM named Test)

[stack@undercloud ~]$ source overcloudrc

[stack@undercloud ~]$ nova list --all-tenants --field
name,instance_name,host,status

5. Cluster Status: Undercloud ----> All Cluster should be in synced status.

[stack@undercloud (stackrc) ~]$ salt 'controller*' cmd.run 'sudo docker exec

$(docker ps --filter name=galera-bundle --quiet) clustercheck'

6. PCS Status ----> Controller (Any) ----> No Service should be in stopped status.

[stack@undercloud ~]$ o 0
[cbis-admin@overcloud-controller-0 ~]$ sudo pcs status

7. Ceph Status and CEPH Health Check: Undercloud ----> This status will vary at Pre-Check. At
Post Health Check, it should be active and all the OSDs should be up.

[stack@undercloud ~]$ salt '*ontroller-0*' cmd.run 'ceph -s'

[stack@undercloud ~]$ salt '*ontroller-0' cmd.run 'ceph osd tree'

10. Zabbix: Check for Last 20 Issues on Zabbix Dashboard.

pg. 3
B. Cleanup PCS resources
Check the resources and perform a clan up if needed
pcs status

Clean up PCS resources that might be in “Failed” or “Stopped” state

pcs resource cleanup

Validate resources have recovered

pcs status

C. Barbican API Healthcheck

Do the below steps on each controller

1. On each controller as root, modify the /openstack/healthcheck by adding "export

NSS_SDB_USE_CACHE=no" before the curl command in the script.

[root@overcloud-controller-0 ~]# docker exec -it barbican_api bash

()[root@overcloud-controller-0 /]# vi /openstack/healthcheck

Add export NSS_SDB_USE_CACHE=no before the curl command in the script

2. Save and quite the vi with “:wq”
3. After the change, the script should look like this:
()[root@overcloud-controller-0 /]# cat /openstack/healthcheck
#!/bin/sh

. ${HEALTHCHECK_SCRIPTS:-/usr/share/openstack-tripleo-common/healthcheck}/common.sh

check_url=$(get_url_from_vhost /etc/httpd/conf.d/10-barbican_wsgi_main.conf)
export NSS_SDB_USE_CACHE=no
curl ${check_url} --insecure --cacert /etc/pki/ca-trust/source/anchors/ca-internal.crt.pem
--key /etc/barbican/barbican-server.key --cert /etc/barbican/barbican-server.pem
==============================================================================

4. Exit from the docker exec opened in step1

()[root@overcloud-controller-0 /]# exit

D. Limit Negative Dentry Size

Do the below steps on each controller one by one.

1. We can reclaim memory manually, by dropping inode/dentry cache on each controller (this
command my take few minutes during the first execution):

pg. 4
 [root@overcloud-controller-0 ~]# sysctl -w vm.drop_caches=2

2. Check the status of the cluster and perform a cleanup if needed

# pcs status
# pcs resource cleanup

3. Add the command in the root crontab of the controller :

[root@overcloud-controller-0 ~]# crontab -e

Add the below lines :
# free reclaimable slab objects (includes dentries and inodes)
30 23 * * * /sbin/sysctl -w vm.drop_caches=2

Save and quite using “:wq”

Check the with crontab -l

4. Check the status of the cluster

pcs status

5. Check the status of the openstack services :

docker ps -a | grep -i nova_api
docker ps -a | grep -i rabbit
docker ps -a | grep -i cinder
docker ps -a | grep -i zabbix
docker ps -a | grep -i galera

E. Rollback
Below the steps for rollback :

1. Barbican API Healthcheck

Execute the same steps described in chapter C and remove the added line export
NSS_SDB_USE_CACHE=no

2. Limit Negative Dentry Size

Execute the step 3 described in chapter D and remove the added lines in root crontab on each
controller.

F. Post-Health check
Hardware status of compute and controller. Make sure Power State should be on for all, Provisioning
State should be active and Maintenance mode should false

pg. 5
[stack@undercloud ~]$ openstack baremetal node list

2. Openstack Status : Undercloud ----> There should not be any failed Service.

[stack@undercloud ~]$ openstack-status

3. Nova Compute Services: Undercloud ----> Status should be enable and State should be UP

[stack@undercloud ~]$ source overcloudrc

[stack@undercloud ~]$ nova service-list

4. Tenant Status: Undercloud ----> All Tenants should be in ACTIVE State (Leave VM named
Test)

[stack@undercloud ~]$ source overcloudrc

[stack@undercloud ~]$ nova list --all-tenants --field
name,instance_name,host,status

5. Cluster Status : Undercloud ----> All Cluster should be in synced status.

[stack@undercloud (stackrc) ~]$ salt 'controller*' cmd.run 'sudo docker exec

$(docker ps --filter name=galera-bundle --quiet) clustercheck'

6. PCS Status ----> Controller (Any) ----> No Service should be in stopped status.

[stack@undercloud ~]$ o 0
[cbis-admin@overcloud-controller-0 ~]$ sudo pcs status

7. Ceph Status and CEPH Health Check: Undercloud ----> This status will vary at Pre-Check. At
Post Health Check, it should be active and all the OSDs should be up.

[stack@undercloud ~]$ salt '*ontroller-0*' cmd.run 'ceph -s'

[stack@undercloud ~]$ salt '*ontroller-0' cmd.run 'ceph osd tree'

8. Zabbix : Check for Last 20 Issues on Zabbix Dashboard.

pg. 6