CBIS - Acceptance Test Procedures - 12-2021

CloudBand Infrastructure
CBIS 20.xxx
CBIS Acceptance Test Procedures

(ATP) - R20GA, R20SP1 and R20SP2
DN09271553
Issue: 1-0
The information in this document applies solely to the hardware/software product (“Product”) specified
herein, and only as specified herein. Reference to “Nokia” later in this document shall mean the respective
company within Nokia Group of Companies with whom you have entered into the Agreement (as defined
below).
This document is intended for use by Nokia's customers (“You”) only, and it may not be used except for
the purposes defined in the agreement between You and Nokia (“Agreement”) under which this document
is distributed. No part of this document may be used, copied, reproduced, modified or transmitted in any
form or means without the prior written permission of Nokia. If you have not entered into an Agreement
applicable to the Product, or if that Agreement has expired or has been terminated, You may not use
this document in any manner and You are obliged to return it to Nokia and destroy or delete any copies
thereof.
The document has been prepared to be used by professional and properly trained personnel, and You
assume full responsibility when using it. Nokia welcomes your comments as part of the process of
continuous development and improvement of the documentation.
This document and its contents are provided as a convenience to You. Any information or statements
concerning the suitability, capacity, fitness for purpose or performance of the Product are given solely
on an “as is” and “as available” basis in this document, and Nokia reserves the right to change any such
information and statements without notice. Nokia has made all reasonable efforts to ensure that the
content of this document is adequate and free of material errors and omissions, and Nokia will correct
errors that You identify in this document. Nokia's total liability for any errors in the document is strictly
limited to the correction of such error(s). Nokia does not warrant that the use of the software in the Product
will be uninterrupted or error-free.
NO WA RR AN TY O F AN Y K I ND , EI T H ER EX PR E S S O R I M P L I E D , I N C L U D I N G B U T
NOT LI M I T ED T O AN Y W AR RA NTY O F AV AI L A B I L I T Y , A C C U R A C Y , R E L I A B I L I T Y ,
TITL E, NO N- I NFR I NG E M EN T, M ER CH AN T AB I L I T Y O R F I T N E S S F O R A P A R T I C U L A R
PURPO S E, I S M AD E I N RE LATI O N T O T HE CO N T E N T O F T H I S D O C U M E N T . I N N O
EVENT W I LL NO KI A BE LI AB LE FO R AN Y D AM A G E S , I N C L U D I N G B U T N O T L I M I T E D
TO SPECI AL, D I RE CT, I N DI R EC T, I NC I DE NT A L O R C O N S E Q U E N T I A L O R A N Y
L OSSES, SUCH AS BU T NO T LI M I T E D T O L O S S O F P R O F I T , R E V E N U E , B U S I N E S S
INTERRUPT I O N, B US I NE SS O PP O RT U NI T Y OR D A T A T H A T M A Y A R I S E F R O M T H E
USE O F TH I S D O CU M EN T O R TH E I N F O RM A T I O N I N I T , E V E N I N T H E C A S E O F
ERRO RS I N O R O M I SS I O NS FRO M T H I S D O CU M E N T O R I T S C O N T E N T .
This document is Nokia proprietary and confidential information, which may not be distributed or disclosed
to any third parties without the prior written consent of Nokia.
Nokia is a registered trademark of Nokia Corporation. Other product names mentioned in this document
may be trademarks of their respective owners.
Copyright © 2021 Nokia. Nokia confidential.
Important Notice on Product Safety
This product may present safety risks due to laser, electricity, heat, and other sources of danger.
Only trained and qualified personnel may install, operate, maintain or otherwise handle this product
and only after having carefully read the safety information applicable to this product.
The safety information is provided in the Safety Information section in the “Legal, Safety and
Environmental Information” part of this document or documentation set.
Nokia is continually striving to reduce the adverse environmental effects of its products and services. We
would like to encourage you as our customers and users to join us in working towards a cleaner, safer
environment. Please recycle product packaging and follow the recommendations for power use and proper
disposal of our products and their components.
If you should have questions regarding our Environmental Policy or any of the environmental services we
offer, please contact us at Nokia for any additional information.
CBIS Acceptance Test Procedures (ATP) - R20GA, R20SP1 and R20SP2
Contents
1 Summary of Changes..................................................................................................................................... 6
2 About This Document.................................................................................................................................... 7

2.1 Intended Audience....................................................................................................................................7
2.2 Technical Support.....................................................................................................................................7
2.3 Related Information.................................................................................................................................. 7
2.4 Supported Hardware.................................................................................................................................8
2.5 Important Note About Copy and Paste.................................................................................................... 8
3 Scope of Document........................................................................................................................................ 9
3.1 Overview................................................................................................................................................... 9
3.2 Assumptions..............................................................................................................................................9
3.3 10 Test Execution Tips.............................................................................................................................9
3.4 Prerequisites........................................................................................................................................... 10
3.5 Test Outcomes....................................................................................................................................... 10
3.6 CLI Deprecation - Important information................................................................................................ 10
4 CBIS Acceptance Test Cases......................................................................................................................12

4.1 CBIS Functional Test Cases.................................................................................................................. 12
4.1.1 CBIS Automated Installation and Configuration (with/without Nuage SDN)...................................12
4.1.2 Verify CBIS Health......................................................................................................................... 13
4.1.3 CRUD (Create, Update and Delete) Virtual Resources in OpenStack (Tenants, Images,
Networks, VMs, Stacks and vRouters)............................................................................................. 15
4.1.4 CBIS Storage Services.................................................................................................................. 18
4.1.4.1 CRUD Storage on HCI-Ceph (Create, Attach, Detach, Delete).............................................18
4.1.4.2 CRUD Storage on Storage Node - Ceph (Create, Attach, Detach, Delete)...........................21
4.1.4.3 Ceph Fast Pools.................................................................................................................... 24
4.1.4.4 Multiple Ceph Pools Support................................................................................................. 28
4.1.4.5 Ceph File Storage (FS)......................................................................................................... 32
4.1.4.6 CRUD External Storage on NetApp (Create, Attach, Detach, Delete)...................................34
4.1.5 CBIS Networking............................................................................................................................ 36
4.1.5.1 IPv4, IPv6 and Dual Stack Virtual Networking Support.........................................................36
4.1.5.2 Support SR-IOV on AirFrame HD Infra NIC..........................................................................41
4.1.5.3 Trust on VFs enabled on AirFrame HD.................................................................................43
4.1.5.4 Unmanaged Virtual Networks Support (no IPAM - IP Address Management)....................... 47
4.1.5.5 VxLAN Overlay for East-West Traffic.................................................................................... 54
4.1.5.6 VGT+ (VLAN Guest Tagging)................................................................................................55
4.1.5.7 L3 Virtual Router with HA......................................................................................................57
4.1.5.8 ARP Responder..................................................................................................................... 57
4.1.5.9 DPDK- Active/Active Bonding Test........................................................................................58
4.1.6 Network Connectivity and Performance.........................................................................................60
4.1.6.1 Basic VXLAN Connectivity.....................................................................................................62
4.1.6.2 Basic VLAN and SR-IOV Connectivity.................................................................................. 63
3
4.1.6.3 Basic VXLAN Connectivity Through Router.......................................................................... 64

4.1.6.4 Basic VXLAN Connectivity and Dual Stack Through Router.................................................64
4.1.6.5 Basic Floating IPs Connectivity............................................................................................. 65
4.1.6.6 Live Migrate VMs with VXLAN Networks.............................................................................. 66
4.1.6.7 Live Migrate VMs with VXLAN Networks and Floating IP..................................................... 67
4.1.6.8 Live Migrate VMs with OVS-DPDK Networking.....................................................................68
4.1.6.9 Basic Connectivity Test - Infra NIC Bonding........................................................................ 69
4.1.6.10 Basic Connectivity Test – tenant NIC Bonding....................................................................72
4.1.6.11 Basic Connectivity Test - OVS L2 Population..................................................................... 73
4.1.6.12 HugePages...........................................................................................................................74
4.1.6.13 NUMA Awareness................................................................................................................76
4.1.6.14 OVS-DPDK...........................................................................................................................81
4.1.6.15 CPU Pinning........................................................................................................................ 82
4.1.6.16 CPU Allocation Ratio per Host Group................................................................................. 83
4.1.7 Software RAID 1............................................................................................................................ 86
4.2 CBIS Non-Functional Test Cases.......................................................................................................... 88
4.2.1 CBIS High Availability.................................................................................................................... 88
4.2.1.1 Neutron Router HA Test (3 Controllers Setup)..................................................................... 88
4.2.1.2 Controller Nodes HA (Server Failovers and CBIS Monitoring).............................................. 91
4.2.1.3 Storage Volume Recovery (NetApp, EMC, HCI-Ceph, SN-Ceph failovers and CBIS
monitoring)....................................................................................................................................95
4.2.2 CBIS Monitoring (Alarms and Logging)......................................................................................... 97
4.2.2.1 Zabbix Monitoring (System Health)....................................................................................... 97
4.2.2.2 Zabbix Metrics Exporter.........................................................................................................99
4.2.2.3 HW Alarms Northbound Forwarding....................................................................................101
4.2.2.4 Ceph Monitoring...................................................................................................................107
4.2.2.5 Zabbix to monitor the Undercloud VM.................................................................................108
4.2.2.6 Alarm Manager (SNMP v2c, V3)......................................................................................... 109
4.2.2.7 ELK.......................................................................................................................................110
4.2.2.8 Log Auditing (Rsyslog).........................................................................................................114
4.2.2.9 User Action Auditing (Horizon)............................................................................................ 116
4.2.2.10 Instance Monitoring............................................................................................................117
4.2.3 CBIS Security............................................................................................................................... 121
4.2.3.1 Customer Certificate............................................................................................................ 121
4.2.3.2 CBIS TLS for External Services (Horizon/Keystone Zabbix, Kibana).................................. 123
4.2.3.3 Host and Operating System Hardening Check Mode..........................................................123
4.2.3.4 Host and Operating System Hardening...............................................................................127
4.2.3.5 Host and Operating System Hardening Rollback................................................................ 131
4.2.3.6 Multi-Tenancy (OpenStack Tenants)................................................................................... 134
4.2.3.7 External LDAPS server........................................................................................................135
4.2.4 CBIS Operations.......................................................................................................................... 140
4.2.4.1 Undercloud VM Backup....................................................................................................... 140
4.2.4.2 Undercloud VM Restore...................................................................................................... 141
4.2.4.3 Overcloud Database Backup............................................................................................... 143
4.2.4.4 Overcloud Database Restore...............................................................................................145
4.2.4.5 Scale In Compute (HCI or none-HCI compute)...................................................................147
4
4.2.4.6 Scale Out Compute (HCI or none-HCI compute)................................................................ 149

4.2.4.7 Scale In Storage Node........................................................................................................ 151
4.2.4.8 Scale Out Storage Node......................................................................................................152
4.2.4.9 Set to Maintenance Compute.............................................................................................. 154
4.2.4.10 Reboot Server, Host Group............................................................................................... 156
4.2.4.11 Replace Controller............................................................................................................. 158
4.2.4.12 CBIS Cluster Shutdown/Power Up.................................................................................... 159
4.2.4.13 Patch Management............................................................................................................ 161
4.2.4.14 Patch Distribution............................................................................................................... 162
5 Appendix 1: Basic CLI................................................................................................................................164

5.1 Connecting with CLI............................................................................................................................. 164
5.2 Source Undercloud/Overcloud Credentials.......................................................................................... 164
5.3 VM/Instances and Hosts Listing........................................................................................................... 164
5.4 Availability Zone and Host Aggregate.................................................................................................. 165
5.5 Flavors.................................................................................................................................................. 165
5.6 Images.................................................................................................................................................. 166
5.7 Volumes................................................................................................................................................ 166
5.8 Security Group......................................................................................................................................167
5.9 VM/Instance Creation........................................................................................................................... 167
5.10 Stack Management............................................................................................................................. 168
5.11 Tenants Management......................................................................................................................... 169
5.12 Networks............................................................................................................................................. 169
6 Appendix 2: Artifact Files for ATP Tests................................................................................................. 174

6.1 Images.................................................................................................................................................. 174
6.2 Stack Files............................................................................................................................................ 174
6.3 Patch Files............................................................................................................................................ 174
6.4 Scripts................................................................................................................................................... 174
5
CBIS Acceptance Test Procedures (ATP) - R20GA, Summary of Changes
R20SP1 and R20SP2
1 Summary of Changes
The following tables list the issues and dates of the last publications of the CBIS documentation set.
Discovery Center: https://doc.networks.nokia.com/default Portfolio > Select a product > Products

> CloudBand Infrastructure. Click the >icon. Select the relevant CBIS release. For a full list and
description of the resolved tickets and known issues, please refer to the relevant CBIS Release Notes.
Disovery Center Documentation:
Discovery Center
Update
Repository Document Version Date
Frequency
Set
Discovery Issue 1-0 24 June 2021

Center
DN09271553 1-0 © 2021 Nokia 6

CBIS Acceptance Test Procedures (ATP) - R20GA, About This Document
R20SP1 and R20SP2
2 About This Document

This document contains information relating to CBIS (CloudBand Infrastructure Software). It is a
combination of the following releases:
• R20 GA - Scratch and Upgrade installation

• R20 SP1 - Scratch installation
• R20 SP2 - Upgrade installation (NO SCRATCH INSTALLATION)
Please ensure that the correct and relevant information/procedure is used according to the specific
release.
2.1 Intended Audience

This document is intended for Node personnel.
2.2 Technical Support

Click this link for technical support: https://customer.nokia.com/support/s/.
2.3 Related Information

The following manuals are available in the release library:
No. Document title DN
1 CBIS Guide to Documentation DN1000020022
2 CBIS Glossary DN1000020033
3 CBIS Release Notes DN09260007
4 CBIS Privacy Considerations DN09281088
5 CBIS Product Description DN09259905
6 CBIS Upgrade From R19A MP3 to CBIS R20 SP2 DN1000047791
7 CBIS Manager DN09259917
8 CBIS Installation Manual DN1000036376
9 CBIS Operations Manual DN09259995
10 CBIS Security Hardening Manual DN09260019
11 CBIS Legal Notices relating to FOSS DN09260022
DN09271553 1-0 © 2021 Nokia 7

CBIS Acceptance Test Procedures (ATP) - R20GA, About This Document
R20SP1 and R20SP2
No. Document title DN
12 CBIS Acceptance Test Procedure (ATP) DN09271553
13 CBIS New Hardware Certification Tests DN09277365
14 CBIS Guide to Networking and Storage Performance DN1000036009
15 CBIS Networking and Storage Performance Benchmarking DN1000008298

Test Report
16 CBIS FOSS Data - extracted from TALKO site DN1000047880
17 CBIS FOSS Data (zip) - extracted from TALKO site
18 CBIS Mixed Hardware Detailed Tests DN1000047235
19 CBIS Zabbix Metrics KPIs - excel file DN09273684
2.4 Supported Hardware

The supported hardware for this release is detailed in the CloudBand Compatibility Matrix:
For additional information please contact the CBIS Hardware Certification team.
2.5 Important Note About Copy and Paste

Remember: Code lines can be corrupted in PDF and Word output. The most accurate
output of code is available in online help (HTML). Where possible, copy code from online
help into your application. Do not blindly copy or paste code sections from this manual (PDF)
into your application. However, for code lines that must be pasted from the document, we
recommend that you copy the code to Notepad first, check that the syntax is correct, and
then copy from the Notepad to the application. Code lines can also be viewed in the CBIS
Online Help.
DN09271553 1-0 © 2021 Nokia 8

CBIS Acceptance Test Procedures (ATP) - R20GA, Scope of Document
R20SP1 and R20SP2
3 Scope of Document
3.1 Overview
These are generic test cases that are used by delivery teams for CBIS acceptance at customer sites.
The test cases are optional. In other words, customize these ATPs per customer. There is no need
to perform all of them. In addition, the appendix (the CLI - included with this document) contains all
relevant command examples.
Note: Code shown in this manual that contains several lines cannot be pasted from a PDF
file because it will split the code into 2 or more lines causing the pasted code to fail when
applied. Users are recommended in these instances to copy and paste the code to notepad,
fix the code and then copy and paste from notepad to the server.
3.2 Assumptions
The delivery teams have experience with OpenStack and are trained with the CBIS product.
3.3 10 Test Execution Tips

In order to enable users to better prepare to run the ATP tests set out here, preparations are key. Here
are some tips.
1. Time saving:
• Import of an Installation Deploy JSON prepared in advance.

• Run the long installation activity over a previous night.
2. Formal scripts – prepare formal scripts to run. If you can re-use any of QA’s scripts, all the better.
At a minimum, use your use cases to build test scripts. As an added bonus, these scripts will serve
as training to users on how to use the system after deployment. We suggest you have scripts for
testing both functionally and migrated data.
3. Informal scripts – prepare informal, unstructured scripts to run as well. They will also help to pull
out defects that will assist in identifying possible related issues in the system, such as verifying
Ceph and services status.
4. Use a tool – we strongly encourage you to put your scripts in a tool.
5. Primary data – create primary data that can be used for testing. This includes logins and
passwords and any data they must look at and/or consume in the tool. You should also organize
this primary data into a format such as a spreadsheet by test case, so they can quickly reference
what data they should use in each script.
DN09271553 1-0 © 2021 Nokia 9

R20SP1 and R20SP2
6. ATP Kick-off deck – Create a slide deck to kick off the ATP window off. This kick-off should
include the scope of testing, a reminder about the value of the system, a reminder that it is a
testing phase and they will find defects in the system, and instructions on how to perform the ATP.
7. ATP User Manual – This book.
8. Pre-run scripts – Ideally you should pre-run the scripts before users try to execute them. You are
familiar with the system, so your eyes on the scripts will be looking for things that are not obvious
or incorrect steps. This will help ensure a much more smoothly run ATP.
9. Report and Track Defects – Ensure that users report defects into a defect tracking including
(logins, urls, steps to recreate) and how to set severity and priority values if appropriate.
10. Coordinate schedule – Ensure that all has been coordinated with the ATP testing schedule after
initial installation and that the setup is running normally and is configured correctly.
3.4 Prerequisites
Successful CBIS installation per the high and low level design agreed upon with the customer. In
detail:
• Hardware and Networking equipment installation and configuration, according to the CBIS
Installation Manual.
• (Hypervisor) Undercloud Physical Server installation.
• CBIS Manager installation over the Hypervisor using the CBIS Manager manual.
• Image to be used for testing.
• All Stacks.
• Appendix 2: Artifact Files for ATP Tests on page 174 in the sub-section titled Patch Files on
page 174
3.5 Test Outcomes

These are the potential test outcomes:
• Pass – Passed OK
• Partially passed – Passed with comments (can be corrected in a later release or in a roadmap
commitment)
• Failed – The test case was not executed OK
• Skipped – The test was passed over and not run
3.6 CLI Deprecation - Important information

When running certain commands, on CLI, in this release, the following message will appear CLI is
deprecated and will be removed in the future. Use openstack CLI instead.
DN09271553 1-0 © 2021 Nokia 10

R20SP1 and R20SP2
DN09271553 1-0 © 2021 Nokia 11

CBIS Acceptance Test Procedures (ATP) - R20GA, CBIS Acceptance Test Cases
R20SP1 and R20SP2
4 CBIS Acceptance Test Cases
4.1 CBIS Functional Test Cases
4.1.1 CBIS Automated Installation and Configuration (with/without Nuage SDN)
Description Install the CBIS Undercloud Physical Server, Undercloud VM, and Overcloud.
Test Case ID
Objective Verify that the CBIS Undercloud and Overcloud have installed successfully.
Estimated ~ 8 hours, depending on the setup scale and selected options and preferably run during the
Duration previous night.
Supported All
From Version
Prerequisite/s
Test Execution 1. Before the installation:
a. Follow the relevant CIQ (Customer Information Questionnaire) and HLD (High Level
Design).
b. Follow the hardware configuration sections in the CBIS Installation manual (per
hardware type).
2. Use the CBIS Manager manual and install the CBIS Undercloud Physical Server.
3. Use the CBIS Manager and deploy the CBIS Undercloud VM and Overcloud.
Note: The above 2 steps are normally executed within a single installation
deploy activity. Desired settings can be prepared in advance on another setup
and importing a saved JSON file or on-line based on relevant platform default
template, while consulting the CBIS Manager manual.
Note: When CBIS is installed with Nuage, the ION delivery/NPI should be
involved as Nuage binaries and licenses are provided by ION teams. To support
TLS for CBIS external services (Horizon/Keystone, Zabbix, Kibana), follow the
Overcloud TLS manual configuration steps.
Note: If a secured communication setup is required, the user should run the
appropriate security hardening sections after deployment and execute the ATP
with no additional steps.
DN09271553 1-0 © 2021 Nokia 12

R20SP1 and R20SP2
Note: Testing IPv4 or IPv6 addresses in the rest of the test cases depends on the
type of installation in the current test case (either IPv4 or IPv6).
Expected 1. The CBIS Undercloud Physical Server has been deployed successfully.
Result 2. The CBIS Undercloud and Overcloud are successfully installed and all steps are marked
successful (Green).
3. CBIS automated verification steps have completed successfully.
• Verification steps are executed automatically by the NOVL tool (Node Validation
Tool).
• NOVL Log can be found in Undercloud VM under /var/log/cbis/cbis_novl_
res.log.
• Check that the CBIS controllers and computes are configured per the high and low
level design (NOVL will show this).
• Undercloud VM is accessible via SSH from the customer network.
• Controllers are available via SSH from the customer network.
• All password in the system are set per the user_config.yaml file.
Status OK | not OK
Comments
4.1.2 Verify CBIS Health
Description CBIS Health Check.
Test Case ID
Objective Verify that CBIS is healthy.
Estimated 0.5 Hour

Duration
Supported All
From Version
Prerequisite/s The CBIS Cluster is installed successfully.
Test Execution 1. Open the OpenStack Horizon Dashboard and check various windows.
2. Open a CLI session to the Undercloud Physical Server (HV). Open tmux. SSH to the
Undercloud using:
ssh stack@uc
3. Check the Nova services status by running the command:
source ~/overcloudrc && openstack compute service list
DN09271553 1-0 © 2021 Nokia 13

R20SP1 and R20SP2
4. Check the Neutron services status by running the command:
source ~/overcloudrc && openstack network agent list
5. Check the Cinder services status by running the command:
source ~/overcloudrc && openstack volume service list
6. Create a new user and login to the OpenStack Horizon Dashboard as follows:
a. From the CBIS Manager - External Tools tab, open Horizon and login to the
OpenStack Horizon Dashboard using the admin user.
b. Navigate to Horizon Dashboard > Identity > Projects > Create a Project.
c. Enter a name and click Create Project.
d. Navigate to Horizon Dashboard > Identity > Users > Create a User.
e. Enter the User Name and Password.
f. Select the Primary Project that you created in step c, above. The Role should remain
_member_.
g. Click Create User.
h. Logout from the OpenStack Horizon Dashboard.
i. Re-login with the newly created user.
7. Check the connection between the Undercloud and the Overcloud nodes by running this
command:
salt '*' cmd.run "hostname; uptime"
8. From the CBIS Manager - External Tools tab, connect to Zabbix https://[OpenStack
Horizon Dashboard IP address]/zabbix. The user and password are as configured in
CBIS Manager >LCM >Installation> Overcloud > Security Configuration.
9. If applicable, check the Ceph status on a one of the computes, controllers as follows.
From the CBIS Manager - External Tools tab, open the Ceph Storage dashboard.
o 0 (alias to connect to controller-0 via SSH)

sudo ceph health detail
10. If applicable, check the external storage status (login to NetApp or EMC controllers).
Expected 1. OpenStack Horizon Dashboard is accessible.

Result 2. The CLI session and connection to the Undercloud are established.
3. The OpenStack compute service list should return Status enabled and State UP.
4. The OpenStack network agent list should return Alive and State UP.
5. The OpenStack volume service list should return Status enabled and State UP.
6. Users can login to the OpenStack Horizon dashboard.
7. All the hosts are accessible from the Undercloud VM and the hostname and uptime
output are given back.
DN09271553 1-0 © 2021 Nokia 14

R20SP1 and R20SP2
8. The Zabbix UI is accessible. Alerts should not show. If Alerts are shown, they require a
specific examination to understand if they are expected.
9. From one of the controllers check that the Ceph health status is OK by running the
command:
sudo ceph -s
Note: If applicable, (that is when Nuage is installed with the CBIS version),
and all is working correctly with no alerts, when integrating with Nuage SDN,
the Nuage components will not be monitored by Zabbix. In addition, Nuage
SDN High Availability will require additional physical hosts (with current level
of integration between CBIS and Nuage, the VSD and VSC are installed as
VMs on the Undercloud host).
10. External storage status is OK.
Status OK | not OK
Comments
4.1.3 CRUD (Create, Update and Delete) Virtual Resources in OpenStack (Tenants,
Images, Networks, VMs, Stacks and vRouters)
Description CRUD Virtual Resources in OpenStack.
Note: Update and Delete resources will be added when available.
Test Case ID
Objective Verify the operability of CRUD operations on OpenStack virtual resources.
Estimated 2 Hours
Duration
Supported All
From Version
Prerequisite/s Retrieve ATP images and YAML files from the CloudBand team (OLCS, FTP, Support
Team).
Test Execution 1. Connect to the Undercloud VM through SSH.

2. Create new tenant using the following command:
source /home/stack/overcloudrc && openstack project create

<project-name>
DN09271553 1-0 © 2021 Nokia 15

R20SP1 and R20SP2
3. Make sure that the tenant has been created by executing the following command:
source /home/stack/overcloudrc && openstack project list
4. Create a new user for the tenant which we created in previous steps.
source /home/stack/overcloudrc && openstack user create <user

name> --project <project name or id>
5. Download the image which will be used for testing locally (to the Undercloud VM).
6. Load the image to the Overcloud (use either the command below or the Horizon).
Example:
source /home/stack/overcloudrc && openstack image create

--file /home/stack/redhat-7.4.qcow2 --disk-format qcow2 --
container-format bare --public redhat-7.4.qcow2
7. Create a few flavors on the setup (you can use the command below or use the Horizon
dashboard).
source /home/stack/overcloudrc && openstack flavor create --

ram 20480 --vcpus 2 --disk 0 --rxtx-factor 2 <flavor-name>
8. Create external network on the setup as shown in the following (can use the commands
below or use the Horizon dashboard):
a. Create the network as follows:
source /home/stack/overcloudrc && openstack network create

EXTERNAL_NETWORK_1 --external --provider-network-type vlan -
-provider-physical-network physnet0 --provider-segment 1006
--mtu 9000 --share
b. Create subnet to the network as follows:
openstack subnet create --subnet-range <CIDR of the

subnet> --ip-version 4 --dhcp --network <ID of the external
network> EXTERNAL_NETWORK_1_SUBNET
Note: Use the name "EXTERNAL_NETWORK_1" for stack compatibility.
9. Connect to the OpenStack Horizon dashboard.

10. Deploy the following stacks using the Horizon Dashboard (Navigate to Project >
Orchestration > Stacks):
a. stacks_ATP1_yaml.yml creates:
• 1 x Vxlan network (physnet0)

• 2 x VMs with 1 interface each and one network each
DN09271553 1-0 © 2021 Nokia 16

R20SP1 and R20SP2
• 2 x Volumes that each VM has one volume

b. stacks_ATP2_yaml.yml creates:
• 1 x Vxlan IPv6 network

• 1 x Vxlan unmanaged network
• 1 x Vxlan IPv4 network
• 2 x VMs with 1 interface each and one network each
Note: On the IPV6 subnet, the ipv6_address_mode: dhcpv6-

stateful, ipv6_ra_mode: dhcpv6-stateful have to define.
Note: On the 2 VMs, configure for the unmanaged network interface IP

address in the same network and subnet different then the ProvNet1
configured network (which is defined in the stack_ATP2_yaml.yml).
Note: Ensure that there is connectivity between the VMs on all the
networks.
11. Create a floating network and then deploy via Horizon (Navigate to Project >
Orchestration > Stacks) a stack that creates 1 x external network with a floating IP on
it, (use the attached file - stack_ATP3_yaml.yml):
neutron net-create --shared --router:external --

provider:network_type vlan --provider:physical_network
physnet0 --provider:segmentation_id <vlan id> EXTERNAL_NETWORK_
1
neutron subnet-create --name EXTERNAL_NETWORK_1_SUBNET --
gateway <default gateway IP> --allocation-pool start=<first
IP in pool>,end=<last IP in pool> --dns-nameserver <valid DNS
address> --disable-dhcp --ip-version 4 EXTERNAL_NETWORK_1
<network CIDR>
12. The external IP addresses should be routable IPs that can reach the tester computer.
a. You need 2 x Vxlan networks.

b. You need 1 x router (with the HA flag - use a setup with 3 controllers) that uses the
floating IP on the external network.
c. Obtain a tenant ID and perform router creation as follows:
openstack project list

neutron router-create --tenant-id
e906565ec7454f0caea84cc94752da65 test_crud_router_ha --
ha=True
d. Ensure that there is 1 VM on each network connected to the router with the following
steps:
DN09271553 1-0 © 2021 Nokia 17

R20SP1 and R20SP2
a. Edit the stack_ATP3_yaml.yml file and ensure that the “segmentation_ id:” value
is not already allocated to the resources ProviderNet1 and ProviderNet2.
Note: If the value has been allocated,change to a relevant different

segmentation_id value and save the file.
b. Launch the stack_ATP3_yaml.yml from the admin tenant.

c. Associate a floating IP to the instance my_vm1.
d. Check connectivity from the Internet to the floating IP of my_vm1 and from my_
vm1 to the internet.
e. Perform a HA connectivity test by checking on which controller the router is
active:
neutron l3-agent-list-hosting-router RouterSB
13. Check that the router name per the stack_ATP3_yaml.yml file is RouterSB as follows:
a. Run a long endless ICMP connectivity check (ping) to the floating IP address of the
VM from the Tester's computer.
b. While running the ICMP connectivity check, reboot the controller that was found
active in step (12-e).
c. The ICMP traffic is expected to break for a short time (1-2 sec) and then continue.
14. Create 1 VM, and perform the following VM lifecycle actions: start/stop/restart.
Expected 1. Tenants were created successfully.

Result 2. Users were created successfully.
3. The images loaded successfully.
4. The flavors were created successfully.
5. All stacks were deployed successfully.
6. VMs can communicate with each other.
7. The VM lifecycle operations completed successfully.
8. The Undercloud Physical Server/stack/instances information is displayed.
9. The relationship between VMs and Computes can be seen by Horizon admin user.
Status OK | not OK
Comments
4.1.4 CBIS Storage Services
4.1.4.1 CRUD Storage on HCI-Ceph (Create, Attach, Detach, Delete)
Description Create, Attach, Detach, and Delete volumes
DN09271553 1-0 © 2021 Nokia 18

R20SP1 and R20SP2
Test Case ID
Objective Verify that volumes can be created, attached, detached, and deleted.
Estimated 0.5 Hour

Duration
Supported All
From Version
Prerequisite/s The CBIS Cluster is installed successfully with HCI-Ceph.
Test Execution 1. Create 1 volume and 2 instances (volume type should be tripleo-ceph).
2. Attach the volume to one of the instances.
DN09271553 1-0 © 2021 Nokia 19

R20SP1 and R20SP2
3. Ensure that the new device was added to the instance using fdisk -l or lsblk commands,
(usually its /dev/vdb).
4. Create the file system and mount-point as follows:
mkdir <new dir>

mkfs -t ext4 <new device>
mount <new device> <new dir>
5. Create or copy files to the VM (you can use the dd utility for writing).
6. Verify the md5sum of the file which you have copied/created.
Example
sudo md5sum file-1.txt

3802de97a6f7db8fd9b7364b04b79dbf file-1.txt
7. Detach the volume from the instance and attach it to the other instance.
8. Connect to the new VM and check that the volume is attached using fdisk -l or lsblk
commands.
9. Create file system and mount-point as follows:

10. Check that all files which you created/copied from step-6 exist.
11. Detach the volume from the instance.
DN09271553 1-0 © 2021 Nokia 20

R20SP1 and R20SP2
12. Delete the volume.
Expected 1. Creation of 1 volumes and 2 instances was completed.

Result 2. The volume was attached to the instance.
3. After the volume attachment, a new device was added to the instance.
4. File system, directory and mount-point were created successfully.
5. Files were copied/created successfully.
6. md5sum was found.
7. The volume detached from the first instance and attached to the second instance.
8. The volume was attached to the instance.
9. File system and mount-point were created successfully.
10. All files which have been created/copied from step-6 exist.
11. The volume was detached from the instance.
12. The volume was deleted.
Status OK | not OK
Comments
DN09271553 1-0 © 2021 Nokia 21

R20SP1 and R20SP2
4.1.4.2 CRUD Storage on Storage Node - Ceph (Create, Attach, Detach, Delete)
Description Create, Attach, Detach, and Delete volumes on storage nodes
Test Case ID
Objective Verify that volumes can be created, attached, detached, and deleted on storage nodes
Estimated 0.5 Hour

Duration
Supported All
From Version
Prerequisite/s The latest CBIS Manager Manual is available.
The CBIS Cluster is installed successfully with storage nodes.
Test Execution 1. Create 1 volume and 2 instances (external storage type).
2. Attach the volume to one of the instances.
DN09271553 1-0 © 2021 Nokia 22

R20SP1 and R20SP2
3. Make sure that new device was added to the instance using fdisk -l or lsblk commands,
(usually its /dev/vdb).
mkdir <new dir>

5. Create or copy files to the VM (you can use dd utility for writing).
6. Verify the md5sum of the file which you copied/ created.
Example
sudo md5sum file-1.txt

3802de97a6f7db8fd9b7364b04b79dbf file-1.txt
7. Detach the volume from the instance and attach it to the other instance.
8. Connect to the new VM and check that the volume is attached using fdisk -l or lsblk
commands.

10. Check that all files which you created /copied from step-6 exist.
11. Detach the volume from the instance.
DN09271553 1-0 © 2021 Nokia 23

R20SP1 and R20SP2
12. Delete the volume.
Expected 1. Creation of 1 volume and 2 instances was completed.

Result 2. The volume was attached to the instance.
3. After the volume attachment, a new device was added to the instance.
4. File system, directory and mount-point were created successfully.
5. Files were copied/create successfully.
6. md5sum was found.
7. The volume detached from the first instance and attached to the second instance.
8. The volume was attached to the instance.
9. File system and mount-point were created successfully.
10. All files which have been created/copied from step-6 are exist.
11. The volume was detached from the instance.
12. The volume was deleted.
Status OK | not OK
Comments
DN09271553 1-0 © 2021 Nokia 24

R20SP1 and R20SP2
4.1.4.3 Ceph Fast Pools
Description Deploy CBIS with OSDs on SSD disk as new Ceph pool (volumes-fast)
Test Case ID
Objective Verify the Ceph build new pool volumes-fast, which includes ssd drives.
Estimated
Duration
Supported 18.0
From Version
Prerequisite/s The latest CBIS Manager Manual is available.
The CBIS Cluster installed successfully with storage nodes (AirFrame).
Test Execution 1. List Ceph pools.
[root@overcloud-controller-0 ~]# ceph osd lspools

0 rbd,1 metrics,2 images,3 backups,4 volumes,5 vms,6 volumes-
fast,
[root@overcloud-controller-0 ~]#
2. Login to the storage nodes.

3. Check fast pool OSDs and verify that they are the fast disks in the storage array.
[root@overcloud-storage-bvt-0 ~]# ceph osd tree -f json |

jq ".nodes[] | select(.name == \"fast-$(hostname -s)\") | .
children[]"
39
36
[root@overcloud-storage-bvt-0 ~]# for image in $(docker ps --
filter name=ceph-osd-$(hostname -s) --format '{{.Names}}'); do
docker exec ${image} mount; done | grep "ceph-36\s\|ceph-39\s"
/dev/nvme1n1p13 on /var/lib/ceph/osd/ceph-39 type xfs (rw,
noatime,seclabel,attr2,inode64,noquota)
/dev/nvme0n1p13 on /var/lib/ceph/osd/ceph-36 type xfs (rw,
noatime,seclabel,attr2,inode64,noquota)
Note:
In addition to the above, ensure that the fast disk has been partitioned to 10G
volumes for each of the local regular OSDs for storage compute by running
the following command:
For image in $(docker ps --filter name=ceph-osd-

$(hostname -s) --format '{{.Names}}'); do docker exec
DN09271553 1-0 © 2021 Nokia 25

R20SP1 and R20SP2
${image} bash -c "mount | grep -q \"ceph-36\s\|ceph-39\

s\" && lsblk"; done
Each of the 10G volumes is a journal for the regular OSD on the storage compute by
running the following command:
for image in $(docker ps --filter name=ceph-osd-$(hostname -s) -

-format '{{.Names}}'); do docker exec ${image} bash -c "mount |
grep -q \"ceph-36\s\|ceph-39\s\" && ls -l /var/lib/ceph/osd/ceph-*/
block.wal"; done
4. Reboot storage nodes. After the storage boot, check that all OSDs are up in the same
order as before the reboot.
5. Check the volume fast pool usage before creating a volume.
a. From one of the controllers execute:
ceph df
b. In the image below, we see 0 OSDs of "volumes-fast" pool.
DN09271553 1-0 © 2021 Nokia 26

R20SP1 and R20SP2
c. Create a volume from the image in the fast pool.
openstack volume create --type tripleo-ceph-fast --image

image --size 30 Vol-1
d. Ensure that only the volume fast pool usage has been changed.
a. As Overcloudrc execute:
ceph df
b. In the image below, we now see 12 OSDs of "volumes-fast" pool.
e. Create an instance from the volume.

f. List the volume from the last step as follows:
openstack volume list
a. In the image below, we can see the volume status showing, "available".
DN09271553 1-0 © 2021 Nokia 27

R20SP1 and R20SP2
g. Boot instance from the volume as follows:
nova boot --boot-volume <Vol-ID> --flavor SMALL_1 --nic net-

name=<Net-Name> <Hostname>
a. Repeat this for all volumes.

b. Before the next image, add the required command:
c. In the image below, we can now see that the volumes status has now changed
into "in-use".
h. Scale in/out storage compute with fast pools according to the CBIS Manager
manual. Fast pools should be built as needed on the relevant devices.
Expected 1. Ceph pool volume-fast has been created.

Result 2. Fast pool OSDs are mounted on the fast drives.
3. One storage node has been rebooted.
4. An Instance was created on the fastpool OSDs.
5. Scale out/in storage compute was complted sucecssfully and fast pools have been built
as needed on the relevant devices.
Status OK | not OK
Comments
4.1.4.4 Multiple Ceph Pools Support
Description Deploy CBIS with multiple Ceph pools.
Test Case ID
Objective Verify the Ceph build multiple pools according to user definitions.
DN09271553 1-0 © 2021 Nokia 28

R20SP1 and R20SP2
Estimated 1 day.
Duration
Supported 18.5
From Version
Prerequisite/s The latest CBIS Installation Manual is available.
The CBIS Node installed successfully with storage nodes (AirFrame RM/OR) and at least 2
Ceph pools.
Scale in/out will be tested in specific sections related to Scale in/out.
Test Execution 1. Check that the Ceph pools have been created according to your definitions.
2. Ensure that pools OSDs are really split into different pools.
3. Reboot one storage node.
4. Create Instance on each of the pools OSDs.
5. Attach volumes to instances.
6. Write to volume.
7. Create instances /volume snapshot.
8. Create instances from that snapshots.
Expected 1. List Ceph pools.

Result
2. The next step is as follows:
a. Login to the storage nodes.

b. Execute the ceph osd tree command to see the OSD division.
DN09271553 1-0 © 2021 Nokia 29

R20SP1 and R20SP2
c. Use the command “ceph-disk list|grep osd” to see the OSD disks mapping..
d. Compare it to the hosts_config.yaml file (from the Undercloud).

3. Ensure that after the reboot, all the OSD are up and running by using the ceph osd
tree command.
DN09271553 1-0 © 2021 Nokia 30

R20SP1 and R20SP2
4. Ensure that every instance/volume was created under the right pool by using the
following command: rbd ls <pool-name>.
5. Check that volume status is “in-use”.

6. Connect to the instance and write to the volume (you can use fallocate/FIO utility).
7. Instance snapshot should display under the images pool.
8. Instance creation from snapshot was successful.
DN09271553 1-0 © 2021 Nokia 31

R20SP1 and R20SP2
Status OK | not OK
Comments
4.1.4.5 Ceph File Storage (FS)
Description Ceph File Storage.
Test Case ID
Objective Verify that the Ceph File Storage is operational.
Estimated 20 minutes.
Duration
Supported CBIS 20.

From Version
Prerequisite/s 1. Setup is installed with a Ceph backend.

2. The VM is required to have ceph-fuse tools installed.
Test Execution 1. Logon onto Horizon and navigate to Share Types in the Share tab as an admin user.
2. Click Create Share Type and enter cephfstype in the Name box.
3. Create Share volume in your Project > Share > Shares > Create Share.
4. Go to Share Overview, view the share properties and take a capture or copy the Path
value.
5. Download a Cloud image of CentOS 7 or any other images that you can install ceph-
fuse utilities upon.
6. Create an image out of the cloud image.
7. Create a storage network with the same vlan used for storage and a range of IPs in that
subnet like this: Admin > Network > Networks > Create network.
8. Launch a VM instance based on the previously created image.
9. Add the storage interface from Horizon and initiate it.
10. Connect to the launched VMs (from one of the controllers), after identifying your network
identifier.
11. Check the storage interface status by using the ip command. In most cases it will be
down.
DN09271553 1-0 © 2021 Nokia 32

R20SP1 and R20SP2
12. Run the dhcclient \[interface name \] command to get the interface up.
13. Validate the ping, from the VM to one of the storage ip addresses (of the controllers).
14. Install ceph-common and ceph-fuse utilities on the launched VM.
15. Create a Rule for share and for cephclient.
16. Create a cephclientkeyring file for the cephclient that will be created to connect the Ceph
cluster from the VM (or from other VMs that would like to use the same share).
17. Copy the created file, ceph.client.cephclient.keyring, to the VM that was
previously created, to the directory /etc/ceph. Make sure that file mode is set to 644.
18. Validate that the file content is valid (assuming it was copied to the VM) from within the
VM.
19. Identify the virtual ip of the Ceph storage network.
20. Create a ceph.conf file in the VM file system, where you want to consume the share.
21. Mount the share by using the ceph-fuse utility inside the VM.
22. Check that the mount point is ready, using the command: df -hT.
Expected 1. Logged into Horizon as an admin user.

Result 2. The Share Type cephfstype has been created.
3. A share volume in the project has been created.
4. The Path value has been recorded.
5. A cloud image of CentOS 7 and any other images have been downloaded.
6. An image has been created out of the cloud image.
7. A storage network has been created.
8. A VM instance based on the previously created image has been launched successfully.
9. The storage interface from Horizon has been added and successfully initiated.
10. After identifying our network identifier, a successful conenction was made to the
launched VMs (from one of the controllers).
11. The storage interface status should be down.
12. The interface is now up.
13. Ping validated from the VM to one of the storage ip addresses (of the controllers).
14. ceph-common and ceph-fuse utilities have been installed on the launched VM.
15. A rule has been created for share and for cephclient.
16. A cephclient keyring file has been created.
17. The created file was copied to the VM that was previously created, to the directory /
etc/ceph and the file mode is 644.
18. The file content is valid.
19. The virtual ip of the Ceph storage network was identified.
20. The ceph.conf file was created in the VM file system.
21. The share was mounted.
22. The mount point is ready.
Status OK | not OK
DN09271553 1-0 © 2021 Nokia 33

R20SP1 and R20SP2
Comments
4.1.4.6 CRUD External Storage on NetApp (Create, Attach, Detach, Delete)
Description Create, Attach, Detach, and Delete volumes on NetApp external storage
Test Case ID
Objective Verify that volumes can be created, attached, detached, and deleted on NetApp external
storage.
Estimated 1 Hour
Duration
Supported All
From Version
Prerequisite/s Latest CBIS Installation Manual is available.
CBIS Cluster is installed successfully with NetApp external storage.
Test Execution 1. Create 3 instances on the same host.

2. Create 2 volumes for each VM. The volume type should be tripleo-netapp and the size
should be different.
3. Attach the volumes to the instances.
4. Connect to the instance and ensure that the new device was added using fdisk -l or lsblk
commands, (usually /dev/vdb).
5. Create file system, mount-point and make it into a permanent mount-point as follows:
a. Create a directory as shown here:
mkdir <new dir>1
b. Create a file system as shown here:
c. Create a mount-point:
d. Search for the device-uuid:
blkid | grep vdb
e. Configure the mount-point into a permanent mount-point:
echo "UUID=<device-uuid> <new dir> ext4 defaults 1 2" >> /

etc/fstab
DN09271553 1-0 © 2021 Nokia 34

R20SP1 and R20SP2
6. Ensure that you have 4 ISCSI paths for each volume:
[root@overcloud-sriovperformancecompute-0 ~]# multipath -ll

mpathc (3600a098000d73185000009ee5cea06d0)
dm-0 NETAPP ,INF-01-00 size=10G features='4 queue_if_no_path
pg_init_retries 50 retain_attached_hw_handle'
hwhandler='1 rdac' wp=rw |-+- policy='service-time 0' prio=14
status=active | |- 14:0:0:175 sdc 8:32
active ready running | `- 16:0:0:175 sde 8:64 active ready
running `-+- policy='service-time 0'
prio=9 status=enabled |- 15:0:0:175 sdd 8:48 active ready
running `- 17:0:0:175 sdf 8:80 active ready running
7. Create or copy files to the VM (you can use the dd utility for writing). You can use the
following example:
dd if=/dev/random of=/<newdir>/wr_file.txt bs=1073741824 dd

if=/dev/random of=/<newdir>/wr_file.txt bs=1K count=100
8. Verify the md5sum of the file which you copied/created. For example:
sudo md5sum wr_file.txt 3802de97a6f7db8fd9b7364b04b79dbf wr_

file.txt
9. Disable 2 ISCSI paths and write/copy to the volume:
echo offline > /sys/block/sdc/device/state

echo offline > /sys/block/sde/device/state
10. Write/copy to the volume while the 2 ISCSI paths are disabled.
11. Create volume with a different size from the other volumes and attach it to the third
instance.
12. Enable back the 2 ISCSI paths:
echo running > /sys/block/sdc/device/state

echo running > /sys/block/sde/device/state
13. Ensure that all ISCSI paths are active and running for all volumes.
14. Migrate the VM while copying/writing to it.
15. Detach volumes from instances.
16. Change volume-type from "tripleo_netapp" to "tripleo-ceph".
17. Delete volumes.
Expected 1. Instances were created successfully.

Result 2. Volumes were created successfully.
3. Volume attachment was successful. Volume status changed to "in-use".
4. New device was added successfully.
5. File system, directory and permanent mount-point were created successfully
DN09271553 1-0 © 2021 Nokia 35

R20SP1 and R20SP2
6. Each volume has 4 ISCSI paths.

7. Writing/Copy operation finished successfully.
8. md5sun of the file was located.
9. Disable two ISCSI paths was successful.
10. Writing/Copy operation while 2 ISCSI paths are disabled finished successfully.
11. Volume creation while 2 ISCSI paths are disabled finished successfully.
12. Enable two ISCSI paths was successful.
13. All ISCSI paths are active and running for all volumes.
14. Instance migration while writing to volume/copy to volume finished successfully.
15. Volume-type changed successfully.
16. Volumes-detach operation finished successfully.
17. Delete volumes operation finished successfully.
Status OK | not OK
Comments
4.1.5 CBIS Networking
4.1.5.1 IPv4, IPv6 and Dual Stack Virtual Networking Support
This is also covered by the CRUD (Create, Update and Delete) Virtual Resources in OpenStack
(Tenants, Images, Networks, VMs, Stacks and vRouters) on page 15 test case.
Description IPv4, IPv6 and Dual Stack Virtual Networking Support.
Test Case ID
Objective Verify IPv4, IPv6 and Dual Stack Virtual Networking Support.
Estimated 1 Hour.
Duration
Supported All.
From Version
Prerequisite/s The CBIS Cluster is installed successfully and tenant, image and flavor are already created.
Test Execution This is a manual procedure for the CRUD (Create, Update and Delete) Virtual Resources in
OpenStack (Tenants, Images, Networks, VMs, Stacks and vRouters) on page 15 stack
creation.
As Tenant:
1. Define an IPv4 network with type VxLAN:
IPv4 net
DN09271553 1-0 © 2021 Nokia 36

R20SP1 and R20SP2
2. Define an IPv6 network with type VxLAN.
IPv6 net
DN09271553 1-0 © 2021 Nokia 37

R20SP1 and R20SP2
3. Define a dual stack network with type VxLAN.

Dual-stack net
Create a Dual-stack network with one IPv4 subnet.
DN09271553 1-0 © 2021 Nokia 38

R20SP1 and R20SP2
4. Create a second subnet IPv6 for Dual-stack network.
DN09271553 1-0 © 2021 Nokia 39

R20SP1 and R20SP2
5. Create an instance for each network.
DN09271553 1-0 © 2021 Nokia 40

R20SP1 and R20SP2
Expected 1. IPv4 Network was created successfully.

Result 2. IPv6 Network was created successfully.
3. Dual stack Network was created successfully.
4. Instances were created successfully.
5. All instances have an IP address.
Status OK | not OK
Comments
4.1.5.2 Support SR-IOV on AirFrame HD Infra NIC
Description Allowing SR-IOV instances on NIC 1 (infra NIC) in AirFrame HD.
Test Case ID
Objective Creating SR-IOV instance with direct ports under NIC 1 (infra NIC).
Estimated 1 hour
Duration
Supported 20 SP2
From Version
Prerequisite/s The CBIS Cluster is installed successfully with SR-IOV mapping from physnet X to NIC 1
port 1 and/or NIC 1 port 2.
By default, the SR-IOV mapping maps physnet 1 to NIC 1 port 1 and NIC 2 port 1 and
physnet 2 is mapped to NIC 1 port 2 and NIC 2 port 2.
This is changeable via the user_config.yaml before installing the Overcloud.
Test Execution From the admin tenant, perform the following actions:
DN09271553 1-0 © 2021 Nokia 41

R20SP1 and R20SP2
1. Create VLAN network on physnet 0 (for SR-IOV).
openstack network create --provider-network-type vlan -

-provider-segment <vlan-ID> --provider-physical-network
physnet1 <network-name>
2. Create associated subnet for the above network:
openstack subnet create --network <network-ID> --subnet-range

192.168.170.1/24 --gateway 192.168.170.254 --dhcp <subnet-name>
3. Create associated direct port for the above network:
openstack port create --network <network-ID> --vnic-type

direct <port-name>
4. Create an instance using the created direct port above as follows:

openstack server create --flavor <flavor-ID> --image <image-ID> -
-nic port-id=<port-ID> <instance-name>
5. Validate that the virtual functions (VFs) from the proper interface are being used. To do
that:
a. SSH to the SR-IOV compute which the instance you just created reside on
b. Run sudo ip link show <interface name> and compare between the MAC
addresses that the instance interfaces received and the MAC addresses that are
shown under sudo ip link show <interface name>
The <interface name> is taken from what that was configured in the user_config.
yaml.
By default, the interfaces that are mapped to physnet 1 are NIC 1 port 1 and NIC 2
port 1
6. Delete the instance and re-check using sudo ip link show <interface name>
that the virtual functions (VFs) are cleared.
A used VF will show with VLAN ID
vf 82 MAC fa:16:3e:9e:d3:23, vlan 404, spoof checking on, link-state enable, trust off
An unused VF will show without VLAN ID
vf 38 MAC 86:61:cd:46:43:4a, spoof checking on, link-state enable, trust off
Note: After a VF is cleared from the system, the last MAC address is still
shown. This is a known behaviour and should not affect any functionality.
Expected 1. Network should be created.

Result 2. Subnet should be create.
3. Port should be created.
DN09271553 1-0 © 2021 Nokia 42

R20SP1 and R20SP2
4. The instance should be created and be in a running state. Inside the instance, when
running ifconfig, you should see the IP address that you configured in the Neutron
subnet or port. Run ifconfig <SR-IOV interface> within the instance to obtain the MAC
address (ether).
5. You should see the allocated VF by running from within the compute which you have
raised the instance on the command: “ip link show <MAC address>”. This is the same
MAC address taken from the SR-IOV port inside the instance.
6. The instance should be deleted and after deleting the instance the VF should be un-
allocated right away. To validate this, run: “ip link show <MAC address>”. The command
should return no results.
Status OK | not OK
Comments
4.1.5.3 Trust on VFs enabled on AirFrame HD
Description Enable trust on VF’s to receive multicast traffic
Test Case ID
Objective Send OSPF multicast traffic from a SR-IOV VM on compute X to a SR-IOV VM on compute
Y. Check (without trust enabled on the VF), that the traffic is not retrieved on the receiver
VM and then check again (with trust now enabled on the VF), that the traffic flows to the
receiver VM.
Estimated 1 hour
Duration
Supported 20 SP2
From Version
Prerequisite/s 1. The CBIS Cluster is installed successfully with at least 2 SriovPerformanceCompute

and/or TripleNicSriovPerformanceCompute computes.
2. Trust is intended for AirFrame HD (Mellanox NICs) only.
3. (Optional) – the trust state of VF’s can be changed post deployment. However, if
required, before the CBIS Cluster deployment you can configure in the user_config.
yaml file the parameter trust: true which will configure trust ON for all the VFs on a given
port(s) in the SR-IOV compute via the service enable_trust.service.
This way, the trust state (on or off) will be modified as the service enable_trust.service will
re-configure the VFs with trust set to ON.
This is optional because even without the help of the service enable_trust.service
you can change the trust state of a VF on the fly. The main issue is that the configuration
changes will be reset after reboot.
The way to configure the CBIS Cluster with trust state of the VF’s for certain port(s):
DN09271553 1-0 © 2021 Nokia 43

R20SP1 and R20SP2
From CLI:
When configuring the user_config.yaml file under the SriovPerformanceCompute and/or

TripleNicSriovPerformanceCompute host group section find the sriov mapping section and
configure trust: true under each port like in the example below:
Note: If trust: true is not set, then the default will be trust: false
sriov_mapping:
• physnet: ref:common_network_config.physnets.physnet1
port: ref:common_network_config.nic_1_port_1.name
trust: true
trust: true
trust: true
trust: true
From the CBIS Manager:
As the option to enable or disable trust on certain port(s) is not enabled in the CBIS
Manager at this time, a small workaround is needed. The workaround will require installing
the CBIS Cluster from scratch, (meaning from the Undercloud VM phase (include).
Note: The CBIS Cluster deployment can be split into several phases:
• Undercloud
• Hardware scan and generate templates
• Overcloud
1. Install the Undercloud VM from the CBIS Manager normally as you would even without
trust.
2. SSH or console to the Undercloud VM and configure trust: true in the user_config.
yaml under /home/stack as explained in the above CLI instructions.
3. After adding trust: true inside the user_config.yaml continue with the rest of the
installation from the UI.
Note: Post install sanity check is optional.
Example
DN09271553 1-0 © 2021 Nokia 44

R20SP1 and R20SP2
iperf2 (2.08) can be used to generate OSPF multicast UDP packets. The example
refers to iperf but any tool (built-in in Linux or not) that perform the same operation
is good.
Note: iperf is not usually a built-in tool.
Note: Make sure the image used for creating the VMs contains a tool
which can sniff packets such as tcpdump. For example: RedHat 7.4
image with built-in tcpdump.
Test Execution From the admin tenant:
1. Create VLAN network on SR-IOV physnet (1-n)
neutron net-create <name of network> --provider:network_

type vlan --provider:segmentation_id <available VLAN ID> --
provider:physical_network physnet#
2. Create associated subnet for the above network
neutron subnet-create --ip-version 4 --allocation-pool

start=192.168.170.1,end=192.168.170.253 --gateway 192.168.170.
254 --enable-dhcp --name <subnet name> <network UUID> 192.168.
170.0/24
3. Create 2 associated direct port for the above network
neutron port-create <network UUID> --name <port name> --vnic-

type direct
4. Create 2 instance using the above created ports. Make sure that each VM is directed to
different SR-IOV compute using the –availability-zone parameter
nova boot --flavor <flavour UUID> --image <image UUID> --

nic port-id=<port UUID> --availability-zone<zone>:overcloud-
sriovperformancecompute-# <instance name>
5. Validate that the virtual functions (VFs) from the proper interface are being used. To do
this, perform the following actions:
a. Connect via SSH or console to one of the instances that you just created and run
ifconfig to obtain the MAC address of the SR-IOV interface. Remember the MAC
address.
b. Connect via SSH to the SR-IOV compute which holds the instance from where you
have just taken the MAC address.
DN09271553 1-0 © 2021 Nokia 45

R20SP1 and R20SP2
c. Continuing from the SR-IOV compute, run:
sudo ip link show | grep -i <instance MAC address>
You should now be able to allocate your VF.

6. Enable trust ON for the instance VF.
Note: If trust is already ON for the VF, skip to step 7.
echo "ON" > /sys/class/net/<port name>/device/sriov/<VF

number>/trust
Note: The VF number is also represented when running the following:
sudo ip link show | grep -i <instance MAC address>
7. Connect via SSH or console to both of the instances that you have created. To simplify
the process, we will designate the VM where we did not change its trust state as VM A
and the VM where we recently asked to change/validate its trust state as VM B.
From VM A (which can be with trust ON or trust OFF - does not matter for this scenario)
generate OSPF multicast packet, for example iperf -c 224.0.0.5 -u
Note: iperf will translate the OSPF multicast IP address to its multicast MAC
address 01:00:5e:00:00:05
Note: Keep running the iperf OSPF multicast traffic from VM A till the end of
the test.
Now run:
tcpdump -nnepi <interface name> dst 224.0.0.5
8. SSH again to VM B and run the following command:
ip link set dev <interface name> allmulticast on
9. Continuing from VM B, now run again the tcpdump command:
tcpdump -nnepi <interface name> dst 224.0.0.5
10. Delete the instances and re-run the command:
sudo ip link show <instance MAC address>
DN09271553 1-0 © 2021 Nokia 46

R20SP1 and R20SP2
Expected 1. Network should be created.

Result 2. Subnet should be created.
3. Ports should be created.
4. The instances should be created and be in running state. Inside the instance, when
running ifconfig, you should see the IP address which you have configured in the
neutron subnet or port.
5. You should see the allocated VFs by running from within the compute which you have
raised the instance on the command: ip link show <MAC address>. This is the
same MAC address taken from the SR-IOV port inside the instance.
6. The trust should be in ON state. Run sudo ip link show <VF> and search for “trust
on”.
7. VM B should not see any OSPF multicast traffic coming from VM A with a destination IP
of 224.0.0.5
8. The SR-IOV interface on VM B should be configure with ALLMULTI. To verify run: ip
link show <interface> and validate that the output contains
“<BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP>”.
9. After VM B is configured with an allmulti flag and with trust on the OSPF multicast,
packets sent from VM A with the destination IP of 224.0.0.5 should now be seen in the
tcpdump of VM B.
10. The instances should be deleted and the VFs trust state should not change.
Status OK | not OK
Comments
4.1.5.4 Unmanaged Virtual Networks Support (no IPAM - IP Address Management)

This is covered by CRUD (Create, Update and Delete) Virtual Resources in OpenStack (Tenants,
Images, Networks, VMs, Stacks and vRouters) on page 15.
Description Unmanaged Virtual Networks Support(no IPAM – IP Address Management).
Test Case ID
Objective Verify Unmanaged Virtual Networks Support (no IPAM – IP Address Management).
Estimated 1 Hour
Duration
Supported All
From Version
Prerequisite/s 1. The CBIS Cluster is installed successfully.

2. Image, flavor and security group should be created.
Test Execution This is a manual procedure for the “CRUD Virtual Resources in OpenStack” stack creation.
DN09271553 1-0 © 2021 Nokia 47

R20SP1 and R20SP2
Port security disabled-per port
Create a VM with VXLAN as follows:
1. Use the command openstack network create <network_name>as follows:
2. Use the command openstack subnet create <subnet_name> --network

<network_name> --dhcp --subnet-range <subnet_range> as follows:
3. Use the command openstack server create --flavor <flavor_name> --

image <image_name> --security-group <sec_group_name> --network
<network_id> <server_name> as follows:
DN09271553 1-0 © 2021 Nokia 48

R20SP1 and R20SP2
4. Find instance-IP using the following command:
openstack server show <instance_id>

[stack@undercloud (overcloudrc) ~]$ openstack server show OVS_
VM
+-------------------------------------+------------------------
----------------------------------+
| Field | Value |
+-------------------------------------+------------------------
----------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | sriov_zone |
| OS-EXT-SRV-ATTR:host | overcloud-sriovperformancecompute-0.
localdomain |
| OS-EXT-SRV-ATTR:hypervisor_hostname | overcloud-
sriovperformancecompute-0.localdomain |
| OS-EXT-SRV-ATTR:instance_name | instance-0000001d |
| OS-EXT-STS:power_state | Running |
| OS-EXT-STS:task_state | None |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2019-06-04T12:35:05.000000 |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | vxlan_net1=10.10.10.4 |
| config_drive | |
| created | 2019-06-04T12:30:38Z |
DN09271553 1-0 © 2021 Nokia 49

R20SP1 and R20SP2
| flavor | SMALL_1 (3c79ebee-0d24-417d-8744-c3797a65fdf8) |

| hostId |
21fbcf8a4f2e83e2af31173b5c1c6d5b5ea37317c02ef36159001c5a |
| id | 5119cea1-c05f-4ff2-8d8d-b958bee477ad |
| image | CentOS7.qcow2 (f10cc41e-2442-4917-aab5-29dced96068f)
|
| key_name | None |
| name | OVS_VM |
| progress | 0 |
| project_id | aa251e43bedb4dfaa22e023c5b03fa3f |
| properties | |
| security_groups | name='CloudBand-Security-Group-ADMIN' |
| status | ACTIVE |
| updated | 2019-06-04T12:35:42Z |
| user_id | 48c737c5da774f8297944fc803b9c265 |
| volumes_attached | |
+-------------------------------------+------------------------
----------------------------------+
5. Find the port of the instance as follows:
openstack port list | grep <instance_ip>

[stack@undercloud (overcloudrc) ~]$ openstack port list | grep
10.10.10.4
| 99335665-0d74-47e8-b8d7-bb3de5850a84 | | fa:16:3e:92:58:c9
| ip_address='10.10.10.4', subnet_id='35e5fae5-cf63-4ada-8260-
8ce338469009' | ACTIVE |
Note: The relevant port is identified based on the subnet_id and the ip_
address above.
6. Use the command openstack port show <port_id> as follows:
openstack port show <port_id>

[stack@undercloud (overcloudrc) ~]$ openstack port show
99335665-0d74-47e8-b8d7-bb3de5850a84
+-----------------------+--------------------------------------
-------------------------------------+
| Field | Value |
+-----------------------+--------------------------------------
-------------------------------------+
| admin_state_up | UP |
| allowed_address_pairs | |
| binding_host_id | overcloud-sriovperformancecompute-0.
localdomain |
| binding_profile | |
| binding_vif_details | datapath_type='system', ovs_hybrid_
plug='True', port_filter='True' |
| binding_vif_type | ovs |
DN09271553 1-0 © 2021 Nokia 50

R20SP1 and R20SP2
| binding_vnic_type | normal |
| created_at | 2019-06-04T12:30:40Z |
| data_plane_status | None |
| description | |
| device_id | 5119cea1-c05f-4ff2-8d8d-b958bee477ad |
| device_owner | compute:sriov_zone |
| dns_assignment | None |
| dns_domain | None |
| dns_name | None |
| extra_dhcp_opts | |
| fixed_ips | ip_address='10.10.10.4', subnet_id='35e5fae5-
cf63-4ada-8260-8ce338469009' |
| id | 99335665-0d74-47e8-b8d7-bb3de5850a84 |
| mac_address | fa:16:3e:92:58:c9 |
| name | |
| network_id | 4012596a-9ead-43a7-85d1-141aad025ceb |
| port_security_enabled | False |
| qos_policy_id | None |
| revision_number | 9 |
| security_group_ids | |
| status | ACTIVE |
| tags | |
| trunk_details | None |
| updated_at | 2019-06-04T12:59:35Z |
+-----------------------+--------------------------------------
-------------------------------------+
7. Use the command openstack port show <port_id> as follows:
openstack port set <port-ID> --no-security-group --disable-

port-security
[stack@undercloud (overcloudrc) ~]$ openstack port set
99335665-0d74-47e8-b8d7-bb3de5850a84 --no-security-group --
disable-port-security
[stack@undercloud (overcloudrc) ~]$ openstack port show
99335665-0d74-47e8-b8d7-bb3de5850a84
+-----------------------+--------------------------------------
-------------------------------------+
| Field | Value |
+-----------------------+--------------------------------------
-------------------------------------+
| allowed_address_pairs | |
| binding_host_id | overcloud-sriovperformancecompute-0.
localdomain |
| binding_profile | |
| binding_vif_details | datapath_type='system', ovs_hybrid_
plug='True', port_filter='True' |
| binding_vif_type | ovs |
DN09271553 1-0 © 2021 Nokia 51

R20SP1 and R20SP2
| binding_vnic_type | normal |
| created_at | 2019-06-04T12:30:40Z |
| data_plane_status | None |
| description | |
| device_id | 5119cea1-c05f-4ff2-8d8d-b958bee477ad |
| device_owner | compute:sriov_zone |
| dns_assignment | None |
| dns_name | None |
| extra_dhcp_opts | |
| fixed_ips | ip_address='10.10.10.4', subnet_id='35e5fae5-
cf63-4ada-8260-8ce338469009' |
| id | 99335665-0d74-47e8-b8d7-bb3de5850a84 |
| mac_address | fa:16:3e:92:58:c9 |
| name | |
| network_id | 4012596a-9ead-43a7-85d1-141aad025ceb |
| security_group_ids | |
| status | ACTIVE |
| tags | |
| trunk_details | None |
| updated_at | 2019-06-04T12:59:35Z |
+-----------------------+--------------------------------------
-------------------------------------+
8. Check for the tap name which this port uses:
Note: The following command is issued on the compute node, where the
recently created VM is located.
[cbis-admin@overcloud-ovscompute-13 ~]$ sudo virsh domiflist

instance-00000019
Interface Type Source Model MAC
-------------------------------------------------------
tap2fee231f-68 bridge qbr2fee231f-68 virtio
fa:16:3e:cd:99:8d
9. Check that all iptables rules related to this port are updated with no security enforced:
[cbis-admin@overcloud-ovscompute-13 ~]$ sudo iptables-save |

grep fee231f-68
-A neutron-openvswi-FORWARD -m physdev --physdev-out
tap2fee231f-68 --physdev-is-bridged -m comment --comment
"Accept all packets when port security is disabled." -j ACCEPT
-A neutron-openvswi-FORWARD -m physdev --physdev-in
tap2fee231f-68 --physdev-is-bridged -m comment --comment
"Accept all packets when port security is disabled." -j ACCEPT
DN09271553 1-0 © 2021 Nokia 52

R20SP1 and R20SP2
-A neutron-openvswi-INPUT -m physdev --physdev-in tap2fee231f-

68 --physdev-is-bridged -m comment --comment "Accept all
packets when port security is disabled." -j ACCEPT
Port security disabled-per network
1. Create a network using the command openstack network create as follows:
[stack@undercloud (overcloudrc) ~]$ openstack network create

vxlan_net2 --disable-port-security
+---------------------------+----------------------------------
----+
| Field | Value |
+---------------------------+----------------------------------
----+
| availability_zone_hints | |
| availability_zones | |
| created_at | 2019-06-04T14:06:51Z |
| description | |
| id | a5336707-31e0-4a89-b9a1-006af0aa3ddd |
| ipv4_address_scope | None |
| ipv6_address_scope | None |
| is_default | False |
| is_vlan_transparent | None |
| mtu | 8950 |
| name | vxlan_net2 |
| provider:network_type | vxlan |
| provider:physical_network | None |
| provider:segmentation_id | 89 |
| router:external | Internal |
| segments | None |
| shared | False |
| status | ACTIVE |
| subnets | |
| tags | |
| updated_at | 2019-06-04T14:06:51Z |
+---------------------------+----------------------------------
----+
2. Create a subnet using the command openstack subnet create as follows:
openstack subnet create <subnet_name> --network <network_name>

--dhcp --subnet-range <network_CIDR>
DN09271553 1-0 © 2021 Nokia 53

R20SP1 and R20SP2
3. Create an instance using the command openstack server create as follows:
openstack server create --flavor <flavor_name> --image <image_

name> --security-group <sec_group_name> --network <network_id>
<server_name>
4. Check for the tap name that the VM uses:
[heat-admin@overcloud-compute-2 ~]$ sudo virsh domiflist

instance-00000007
Interface Type Source Model MAC
-------------------------------------------------------
tapafe466c7-a3 bridge qbrafe466c7-a3 virtio
fa:16:3e:c8:f9:4a
5. Check all iptables rules related to this port updated with no security enforced:
[cbis-admin@overcloud-compute-2 ~]$ sudo iptables-save |grep

afe466c7
a3 --physdev-is-bridged -m comment --comment "Accept all
packets
when port security is disabled." -j ACCEPT
a3 --physdev-is-bridged -m comment --comment "Accept all
packets when port security is disabled." -j ACCEPT
Expected Port security disabled-per port

Result
1. VM created successfully.
2. Expected result: port updated - security group removed.
3. Port security is disabled.
4. Found a matching Tap name.
5. No security is enforced for port.
Port security disabled-per network
1. Network created successfully.

2. Subnet created successfully.
3. VM created successfully.
4. Found a matching Tap name.
5. No security is enforced for port.
Status OK | not OK
Comments
4.1.5.5 VxLAN Overlay for East-West Traffic

This is covered by "CRUD Virtual Resources in OpenStack" test case.
DN09271553 1-0 © 2021 Nokia 54

R20SP1 and R20SP2
4.1.5.6 VGT+ (VLAN Guest Tagging)
Description VGT "VLAN Guest Tagging".
Test Case ID
Objective Check the ability to configure the VGT filter on Mellanox SR-IOV ports.
Estimated ~8 hours.
Duration
Supported CBIS 20.

From Version
Prerequisite/s Install CBIS on a platform with Mellanox NICs and vlans ranges at the flat physnets fields.
Follow these steps:
1. Go to: CBIS Manager > installation >Overcloud>VLANs > Physical Networks VLAN
Ranges.
2. You should also set VGT allowed VLAN ranges as follows. Go to: CBIS Manager >
installation > Customize host groups> SR-IOV-Performance-Compute > SR-IOV
per port configuration and VGT allowed VLAN ranges.
3. The flat network should be enabled at the switch side (vlan ranges as well):
a. At the CBIS Manager Installation tab, under the Customize host group SR-
IOV compute go to sr-iov per port and select the flat physnet at the port physnet
mapping and configure the number of VFs that you need.
b. Go to "VGT allowed VLAN ranges". For every port, select the allowed vlan range
(adding the vlan ranges will enable the VGT+ feature).
Test Execution 1. First, create 2 VMs, in different host computes (use the same flat network but different
ports for the VM).
1. Create a network using a flat physnet vlan.

2. Create a port from this network.
3. Create a VM using the ports created before. (same flat network, different port for
VM).
2. After the creation is complete and in addition to the existing port in every VM, create a
sub interface (for every machine) and attach both of the sub interfaces to the same vlan.
.
Note: Both IPs from both the interfaces should be on the same subnet.
Note: The sub interfaces vlan should not be contained in the allowed VGT+
range that was configured during the installation.
DN09271553 1-0 © 2021 Nokia 55

R20SP1 and R20SP2
3. Ping between the 2 VMs, using the sub interfaces IP address. Pinging using the
sub interface IP address will automatically result in routing of the ping through, so
communication using the blocked vlan is guaranteed. result=unsuccessful ping.
4. Negative step > Create another 2 VMs similar to the previous step but this time use a
sub interface that is in the allowed range, result=successful ping.
5. Ensure that there is connectivity between the “pure” flat to the VGT+ host.
Note: You need to have 2 zones,1 zone with VGT+ and 1 zone without VGT+.
6. Scale out a new SR-IOV compute. Test the functionality of this feature on the new
added server. The feature should work successfully. (Unallowed ranges should not be
communicating).
7. Scale in another compute from the same host group. The feature should work
successfully.
8. Check that the functionality of the feature is not affected after power off/on and reboot
actions. The feature should work successfully.
9. The traffic should blocked between a none flat VM and a flat VM as tested by the
creation of a none flat network and of a vm based on this network.
10. The ability to create a large amount of vNICS and VMs should worrk normally.
To test:
a. Create a number of VMs.

b. For each VM, create several SR-IOV ports (6 at least).
Note: All these SR-IOV ports can be configured to use VGT+ mode while
each VM consumes a big amount of vNICs [>20].
Option:
1. Create the VMs using a heat stack.

2. Verify the health of the VMs.
11. Using the heat stack:
a. Delete all the VMs above.

b. After the delete has completed, recreate the VMs.
c. Verify the health of the VMs.
d. Run these test steps for several iterations.
Note: Creation and deletion of the VMs should work fine with no problems.
12. Nova action:
a. Create a couple of VMs.
DN09271553 1-0 © 2021 Nokia 56

R20SP1 and R20SP2
b. Verify their health and their operability.

c. Start testing the different nova action options on these VMs, (pause - resume ,
suspend - resume , rebuild - soft/hard reboot , shelve instance - resume). All the
actions should work normally, and no damage should occur to the VMs.
13. The user should be able to use the max amount of VFs available.
To test:
According to the "Number of VFs" configured at the installation, create the maximum
amount of VFs per host (default ~45).
Expected 1. 2 VMs were created successfully.

Result 2. Sub interface(s) have been created and attached.
3. Successfully pinged between the 2 VMs, using the sub interfaces IP address with
result=unsuccessful ping.
4. Using a sub interface that is in the allowed range, result=successful ping.
5. Connectivity established between “pure” flat to VGT+ host.
6. Scale out of a new SR-IOV compute was successful.
7. Scale in of another compute from the same host group was successful.
8. Power on/off and reboot actions did not affect functionality.
9. Traffic was blocked between a none flat VM and a flat VM.
10. A large amount of vNICS and VMs were created and worked normally.
11. Creation and deletion of the VMs worked normally with no problems.
12. All Nova actions work normally, and no damage has occurred to the VMs.
13. The maximum amount of VFs per host (default ~45) have been created successfully.
Status OK | not OK
Comments
4.1.5.7 L3 Virtual Router with HA

This is covered by the "CRUD Virtual Resources in OpenStack" test case.
4.1.5.8 ARP Responder
Description Check that the ARP Responder is enabled.
Test Case ID
Objective Verify that the ARP Responder is enabled.
Estimated ~ 1 hour
Duration
DN09271553 1-0 © 2021 Nokia 57

R20SP1 and R20SP2
Supported CBIS 20
From Version
Prerequisite The setup is installed with at least one host with the ARP responder set to "on" and one set
to "off".
Test Execution 1. After the installation is complete, deploy a VM on each of the hosts.
2. Ping (cross-compute) different VMs from an ARP responder "on" host group.
3. Verify that the first ping to a non "ARP responder" VM takes slightly longer than the rest
of the VMS (due to the first ARP).
4. Open tcpdump on both VMs setting "ARP on & ARP off" to see the incoming ARP
requests.
5. Create a dual-stack network and test this feature in a dual-stack situation.
6. Test the functionality of this feature after performing a scale in/out procedure. Scale in a
compute and scale it out with computing in ARP set off.
7. Test the functionality of this feature after creating a new custom template.
8. Repeat this test in DPDK, OVS, and SR-IOV computes, to check that it works in all
different types of computes.
Expected 1. 2 VMs were deployed successfully.

Result 2. Different VMs from an ARP responder "on" host group were successfully pinged.
3. The first ping to a non "ARP responder" VM takes slightly longer than the rest of the
VMS (due to the first ARP).
4. tcpdump on both VMs "ARP on & ARP off" was opened and an incoming ARP request
was seen successfully.
5. A dual-stack network was created and tested in a dual-stack situation.
6. After performing a scale in/out procedure, the functionality of this feature was tested
successfully.
7. After creating a new custom template, the functionality of this feature was tested
successfully.
8. Tests in DPDK, OVS, and SR-IOV compute were repeated. It worked in all types.
Status OK | not OK
Comments
4.1.5.9 DPDK- Active/Active Bonding Test
Description Check the DPDK-Active/Active Bond Mode.
Test Case ID
Objective Verify that the DPDK - Active/Active Bond Mode works as designed.
DN09271553 1-0 © 2021 Nokia 58

R20SP1 and R20SP2
Estimated ~ 1 hour
Duration
Supported CBIS 20
From Version
Prerequisite Ensure that the CBIS Manager setup is installed with a host group and a DPDK zone where
the Bond mode is Active-Active.
Test Execution 1. After the installation is complete, check that the Tenant bond mode is set to active-
active, first ssh to the target compute and then run the command :
"ovs-appctl bond/show"
The output should be :
"bond_mode: balance-tcp"
2. To check the infra-Bond at the compute, run the following command:
"less /proc/net/bonding/infra-bond"
In the output, the infra bond mode should be "active-backup", and verify correct ports
bonding by the Identification of the ports name (for example ens5f0-ens5f1).
3. Test the functionality of the tenant Bond, and first verify correct ports bonding.
4. Check the bond operation, run a continuous ping to the target VM sitting on active-active
compute in parallel watch the state of the bond. Set the active interface down:
• One time through the switch.

• One time by physically disconnecting the cable of the interface.
When putting the interface down, you should see that the active interface has been
replaced immediately with the other interface, and the ping that was sent should not be
affected.
5. Set the interface back up and see that it recovers.
6. After the recovery of the interface, create more VMs to check the bond after the bond
test and after creating the VM test connectivity between them.
Expected 1. The installation was completed, and the Tenant bond mode was set to active-active. The
Result output was :
"bond_mode: balance-tcp"
2. The command was run, and in the output, the infra bond mode was "active-backup", and
ports bonding was correct.
3. Ports bonded correctly.
4. The bond operation was checked. A continuous ping to the target VM sitting on active-
active compute was run. The active interface was set down. The active interface
DN09271553 1-0 © 2021 Nokia 59

R20SP1 and R20SP2
was replaced immediately with the other interface, and the ping that we sent was not
affected.
5. The interface back up was set, and it recovered.
6. The interface created more VMs and checked the bond post the bond test, after creating
the VM test connectivity between them.
Status OK | not OK
Comments
4.1.6 Network Connectivity and Performance
Description Deploy VMs to use SR-IOV.
Test Case ID
Objective Verify that VMs deploy on SR-IOV computes and use SR-IOV networks.
Estimated 1 Hour
Duration
Supported All
From Version
Prerequisite Image with SR-IOV support is available (Image level optimization may be required). The
image has the iperf binary already installed.
Hardware (NICs) support SR-IOV.
An aggregate is created and added to the SR-IOV computes.
nova aggregate-create performance

nova aggregate-set-metadata 1 pinned=true
nova aggregate-add-host 1 <computes>
• 2 x Compute Nodes installed with SR-IOV support

• Switch is configured with SR-IOV VLANs
• Tenant and users are already created
• Images and flavors are already added to the setup
Example:
nova flavor-create m1.performance 6 2048 0 4

nova flavor-key 6 set hw:cpu_policy=dedicated
nova flavor-key 6 set aggregate_instance_extra_specs:pinned=true
DN09271553 1-0 © 2021 Nokia 60

R20SP1 and R20SP2
Test Execution 1. Source the rc file corresponding to your tenant:
source <rc file>
2. Create an SR-IOV network and ports
a. Create a network:
neutron net-create --provider:physical_network=physnet1 \

--provider:network_type=vlan \
--provider:segmentation_id=451 sriov_net
b. Create a subnet:
neutron subnet-create --name sriov_subnet sriov_net <CIDR>
c. Create port 1 (Network ID from ‘neutron net-list’):
neutron port-create <Network ID> --binding:vnic-type direct
d. Create port 2 (Network ID from ‘neutron net-list’):
neutron port-create <Network ID> --binding:vnic-type direct
Note: Keep track of the port IDs, they will be needed to boot the VM
3. Instantiate 2 VMs on 2 computes with SR-IOV support (VMs should have an interface on
the SR-IOV network).
nova boot --flavor m1.performance --image perf --nic port-

id=<Port 1 ID> sriov_vm1
nova boot --flavor m1.performance --image perf --nic port-
id=<Port 2 ID> sriov_vm2
4. Verify connectivity between SR-IOV VMs (ping / ssh).

5. Optional: Test the performance between VMs using the iperf tool as follows:
a. Login to the VM1 console and run the following command:
iperf -s
b. Login to the VM2 console and run the following command:
iperf -c <IP of VM1> -i 1 -l 9000
Expected 1. No output is expected.

Result 2. Network, subnet and ports are created successfully.
3. VMs are instantiated successfully and have the expected IP address.
DN09271553 1-0 © 2021 Nokia 61

R20SP1 and R20SP2
4. There is connectivity between the 2 VMs.

5. Results show a transfer rate between 8 and 9 Gbit/s. (With a Mellanox NIC, the transfer
rate is between 20 and 21 Gbit/s).
Status OK | not OK
Comments
4.1.6.1 Basic VXLAN Connectivity
Description VXLAN network connectivity testing
Test Case ID
Objective Verify that VMs can be deployed with functioning VXLAN networks.
Estimated 1 Hour
Duration
Supported All
From Version
Prerequisite The CBIS Cluster is installed successfully with OVS computes.
Test Execution 1. Source overcloudc

2. Create VXLAN network. (With neutron deprecated, use OpenStack instead).
openstack network create vxlan_net
3. Create subnet:
openstack subnet create --network vxlan_net --subnet-range 10.

21.45.0/24 ovs-sub
4. As admin (overcloudrc), create 5 VMs as follows:
openstack server create --flavor d7f1e705-b8ff-49b4-8fd9-

55fb8df3a556 --image bfab4edd-b34c-409d-be35-3da1470eadb8\
--security-group fac769ec-af17-4dc9-a991-f3d6d6095c7d --nic
net-id=80ed9510-3936-450b-80e9-ade10c71a0cf --availability-
zone \
ovs-zone:overcloud-ovscompute-delhi-0.localdomain ovs_VM
5. Make sure that all VMs in all computes received the IP.
6. Perform connectivity tests between each VM and all other VMs.
7. Perform a file transfer between each VM and all other VMs.
Expected 1. VXLAN network created.

Result 2. VMs created across computes.
3. Subnet created.
DN09271553 1-0 © 2021 Nokia 62

R20SP1 and R20SP2
4. 5 VMs created.
5. VMs get IPs from the correct namespace.
6. Ping has no packets lost between VMs.
7. Files can be transferred between VMs.
Status OK | not OK
Comments
4.1.6.2 Basic VLAN and SR-IOV Connectivity
Description Test network connectivity between OVS and SR-IOV computes.
Test Case ID
Objective Verify that there is network connectivity between OVS and SR-IOV computes.
Estimated 1 Hour
Duration
Supported All
From Version
Prerequisite The CBIS Cluster is installed successfully with OVS and SR-IOV computes.
Test Execution 1. Create VLAN Network with OVS.

2. Create VLAN Network Type and Direct Ports.
3. Create VM with the VLAN with OVS network.
4. Create VM with the VLAN network and direct port.
5. Create neutron-router and add interfaces.
6. Make sure that all VMs in all computes receive IP.
7. Perform connectivity tests between VMs that are in the same compute.
8. Perform connectivity tests between VMs that are in different computes.
9. Perform a file transfer from each VM to all other VMs.
Expected 1. VLAN with OVS networks created.

Result 2. VLAN with Direct Ports networks created.
3. VMs with the VLAN with OVS network created across computes.
4. VMs with the VLAN network and direct port created across computes.
5. Neutron-router is created and interfaces added.
6. VMs get IPs from correct namespace.
7. Ping has no packet lost between VMs (same compute).
8. Ping has no packet lost between VMs (different computes).
9. Files can be transfer between VMs
Status OK | not OK
DN09271553 1-0 © 2021 Nokia 63

R20SP1 and R20SP2
Comments
4.1.6.3 Basic VXLAN Connectivity Through Router
Description VXLAN connectivity testing via a router.
Test Case ID
Objective Verify that VM networks can communicate via a router.
Estimated 1 Hour
Duration
Supported All
From Version
Prerequisite The CBIS Cluster is installed successfully.
Test Execution 1. Create 3 VXLAN networks.

2. Create One Router with interface from each Network.
3. Create 3 VMs each with a different VXLAN Network.
4. Perform connectivity tests between each VM to all the other ones.
5. Perform file transfer between each VM to all the other ones.
Expected 1. VXLAN networks were created

Result 2. Router was created successfully
3. VMs were created across computes, VMs get IPs from the correct namespace and VMs
default gateway is the Router relevant interface
4. Ping has no packet lost to its gateway. Ping has no packet lost between VMs
5. Files can be transferred between VMs
Status OK | not OK
Comments
4.1.6.4 Basic VXLAN Connectivity and Dual Stack Through Router
Description VXLAN connectivity testing via a router over both IPv4 and IPv6 networks.
Test Case ID
Objective Verify that VM networks can communicate via a router over both IPv4 and IPv6 networks.
Estimated 1 Hour
Duration
Supported All
From Version
DN09271553 1-0 © 2021 Nokia 64

R20SP1 and R20SP2
Test Execution 1. Create 3 VXLAN networks - Each network with 2 subnets (IPv4 & IPv6)
a. IPv6 network should be created with mode SLAAC

2. Create One Router with interface from each Network (IPv4 & IPv6)
3. Create 3 VMs each with a different VXLAN Network
4. Perform connectivity test between each VM to all others.
• The default route on each instance can be checked with the following commands:
ip -6 route/sbin/route -A inet6 add default gw <IPv6 gateway

address>
5. Perform file transfer between each VM to all other
Expected 1. VXLAN network was created.

Result 2. Router was created successfully.
3. VMs created across computes. VMs get IPs from the correct namespace (IPv4 & IPv6)
and VMs default gateway is the Router relevant interface (IPv4 & IPv6).
4. Ping has no packet lost to its gateway. Ping has no packet lost between VMs.
5. Files can be transferred between VMs.
Status OK | not OK
Comments
4.1.6.5 Basic Floating IPs Connectivity
Description Test Floating IP connectivity.
Test Case ID
Objective Verify that Floating IPs can be assigned and test network connectivity using these IPs.
Estimated 0.5 Hour

Duration
Supported All
From Version

2. Create 3 VMs each with a different VXLAN Network.
3. Create an External Network.
5. Set External Network as gateway for the router.
6. Associate IPs from external network to VMs.
DN09271553 1-0 © 2021 Nokia 65

R20SP1 and R20SP2
7. Perform connectivity test between each VM to all the others.

8. ssh/ping to a VM from an external network or from a network routed to external.
Expected 1. VXLAN network was created.

Result 2. VMs created across computes, VMs get IPs from the correct namespace.
3. External Network is created successfully.
4. VMs default gateway is the Router relevant interface.
5. Gateway for the router created successfully to an External Network.
6. IPs from the external network are successfully assigned to the VMs.
7. Ping has no packet lost to its gateway. Ping has no packet lost between VMs.
8. ssh/ping to VMs external IP from external to VMs in successful.
Status OK | not OK
Comments
4.1.6.6 Live Migrate VMs with VXLAN Networks
Description Live Migrate VMs with VXLAN networks.
Test Case ID
Objective Verify that live migration is successful for VMs with VXLAN networks and there is no packet
loss during the migration.
Estimated 0.5 Hour

Duration
Supported All
From Version

2. Create 2 VMs, each with the 3 VXLAN networks..
3. Make sure that the VM has received IPs from each VXLAN Network.
4. Perform connectivity tests between VMs.
5. Perform Live Migrate to the VMs.
nova live-migration <Instance ID>
Expected 1. VXLAN networks were created.

Result 2. VMs were created across computes.
3. VMs obtained IPs from the correct namespace.
4. Ping has no packet lost between VMs.
5. Live Migrate VMs was a success.
DN09271553 1-0 © 2021 Nokia 66

R20SP1 and R20SP2
a. Each VMs changes Undercloud Physical Server.

b. VM still has 3 VXLAN networks each.
c. There might be packet losses due to Community Bug: https://bugs.launchpad.net/
neutron/+bug/ 1815989*
Status OK | not OK
Comments
4.1.6.7 Live Migrate VMs with VXLAN Networks and Floating IP
Description Live Migrate VMs with VXLAN networks and Floating IPs.
Test Case ID
Objective Verify that live migration is successful for VMs with VXLAN networks and Floating IPs and
there is no packet loss during the migration.
Estimated 0.5 Hour

Duration
Supported All
From Version
Test Execution 1. Create 1 VXLAN network.

2. Create VM with VXLAN network.
3. Make sure VM received IPs from each VXLAN Network.
4. Create External Network.
6. Set External Network as gateway for the router.
7. Associate IPs from external network to the VM.
8. Perform connectivity test between Namespace to VM and from external to VM.
9. Perform Live Migrate to the VM.
Expected 1. VXLAN networks created.

Result 2. VMs are created.
3. VMs get IP addresses from the correct namespace.
4. External Network is created.
5. Router is created with the External Network as a gateway.
6. External network is set as gateway for the router.
7. External IP association to VM succeeded.
DN09271553 1-0 © 2021 Nokia 67

R20SP1 and R20SP2
8. Ping/ssh from External to VM succeeded.
Note:
There might be packet losses due to Community Bug: https://bugs.launchpad.

net/neutron/+bug/ 1815989*
9. Live Migrate VMs success.
a. VM changed Undercloud Physical Server.

b. VM still has an associated IP.
c. Ping/ssh succeeded through external Network to VM.
Note:
There might be packet losses due to Community Bug: https://bugs.

launchpad.net/neutron/+bug/ 1815989*
Status OK | not OK
Comments
4.1.6.8 Live Migrate VMs with OVS-DPDK Networking
Description Live Migrate VMs with networks based on OVS-DPDK.
Test Case ID
Objective Verify that live migration is successful for VMs using OVS-DPDK networks and there is no
packet loss during the migration.
Estimated 0.5 Hour

Duration
Supported All
From Version
Prerequisites • The CBIS node is installed successfully.

• The CBIS node is installed with OVS-DPDK host-groups.
• The OVS-DPDK group includes at least 2 different compute nodes.
Test Execution 1. Create 3 VXLAN network.

2. Create 2 VMs on the OVS-DPDK host group.
3. Each VM is using the 3 VXLAN networks.
4. Ensure that the VM received IPs from each VXLAN Network.
5. Perform connectivity tests between VMs .
DN09271553 1-0 © 2021 Nokia 68

R20SP1 and R20SP2
6. Perform Live Migrate to the VMs as follows:
Expected 1. VXLAN networks were created.

Result 2. VMs were created across computes on the OVS-DPDK host group.
3. VMs get IPs from correct namespaces.
4. Ping has no packet loss between VMs.
5. Live Migrate of VMs was a success.
a. Each VMs changes an Undercloud Physical Server to another compute on the same
host group.
b. VM still has 3 VXLAN networks each.
c. There might be packet losses due to Community Bug: https://bugs.launchpad.net/
neutron/+bug/ 1815989*
Status OK | not OK
Comments
4.1.6.9 Basic Connectivity Test - Infra NIC Bonding
Description Infrastructure networks connectivity testing with interface failover over the 1st NIC.
Test Case ID
Objective Verify that infrastructure networks continue operating when a bonded interface link is
disconnected.
Estimated 1 Hour
Duration
Supported All
From Version
Prerequisite 1. The CBIS Cluster is installed successfully.

2. For hardware types: AirFrame Dell or HP, at least 1 OVS compute exists in the setup.
Test Execution 1. Set the source to the overcloudrc admin user as follows:
source ~/overcloudrc
2. From the Undercloud VM, check OVS active port of infra-bond in each compute.
salt '*ompute*' cmd.run 'sudo cat /proc/net/bonding/infra-bond

|grep "Currently Active Slave"' '
DN09271553 1-0 © 2021 Nokia 69

R20SP1 and R20SP2
3. Run a constant connectivity check (ping for example) from one of the controllers on
one of the infra networks (for example internal network usually 172.31.0.x) to one of the
computes.
4. Disconnect the active bond port of the target compute either by disconnecting the cable
directly from the server or by logging into the connected switch and shutting down the
appropriate port.
5. Ensure that the Active Interface in the bond has been changed and now the active port
is the one that was inactive before the disconnection.
6. Validate that the connectivity check is still running and there are no dropped packets.
7. Login to the Zabbix portal and validate that the alarms interface <downed active
interface name> appears on the <server name>.
8. Reconnect the compute port that was disconnected.
9. Login again to Zabbix portal and validate that the alarms interface <downed active
interface name> on the <server name> has disappeared. Note that it may take few
minutes for the alarms to disappear and reset.
10. Repeat steps 3-7 for the new active port.
11. From the Undercloud VM, check OVS active port of infra-bond in each controller.
salt 'controller*' cmd.run 'sudo cat /proc/net/bonding/infra-

bond |grep "Currently Active Slave"' '
12. Run a constant connectivity check (ping for example) from one of the computes on one
of the infra networks (for example internal network usually 172.31.0.x) to one of the
controllers.
13. Disconnect the active bond port of the target controller either by disconnecting the cable
appropriate port.
17. Reconnect the controller port that was disconnected.
Expected 1. After sourcing to overcloudrc, you should see the prompt change to [stack@undercloud
Result (overcloudrc) ~]$
2. The active port is listed. Example: active slave mac: 24:8a:07:38:1a:f1
(ens6f1)
3. The connectivity check runs constantly and successfully.
DN09271553 1-0 © 2021 Nokia 70

R20SP1 and R20SP2
4. The interface should change to state DOWN as shown here:
Expected Output:
ens6f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq

master ovs- system state DOWN qlen 1000>
5. The standby interface has become Active.

6. The connectivity check should still run constantly and successfully.
7. Check in Zabbix that you have one alarm for each compute that is <active
interface> was downed. (For example: interface ens6f0 down on
overcloudovscompute-2)
8. The interface should now have changed to state UP when you check the following:
salt '*ompute*' cmd.run 'ip address show <downed interface>'
Expected Output:

master ovs- system state UP qlen 1000>
9. Check in Zabbix that the “interface down” alarms have been cleared and are not
showing. Note that it may take few minutes for the alarms to disappear and reset.
10. The results are the same as results 3-7 for the new active port.
(ens6f1)
12. The connectivity check runs constantly and successfully.
Expected Output:


16. Check in Zabbix that you have one alarm for each controller that is <active
overcloudovscompute-2).
Expected Output:

DN09271553 1-0 © 2021 Nokia 71

R20SP1 and R20SP2
19. The results are the same as results 10-14 for the new active port.
Status OK | not OK
Comments
4.1.6.10 Basic Connectivity Test – tenant NIC Bonding
Description Tenant network connectivity testing with tenant NIC failover.
Test Case ID
Objective Verify that tenant networks continue operating when a bonded interface link is disconnected.
Estimated 1 Hour
Duration
Supported All
From Version

2. For hardware types: AirFrame, Dell or HP, at least 1 OVS compute exists in the setup.
Test Execution 1. Set the source to the overcloudrc admin user as follows:
2. Create a VXLAN network.

3. Create 5 VMs with the network created in step 1.
Note: If you use hardware type AirFrame, Dell or HP, you have to direct the
VMs to OVS compute.
4. From the Undercloud VM, check OVS active port of tenant-bond in each compute.
salt '*ompute*' cmd.run 'ovs-appctl bond/show tenant-bond |

grep "active slave mac"'
5. Run a constant connectivity check (ping for example) from the namespace of the VXLAN
network to the VMs IP address. This connectivity check needs to run constantly from this
point until the end of the test.
6. Disconnect the active bond port of the target controller either by disconnecting the cable
appropriate port.
DN09271553 1-0 © 2021 Nokia 72

R20SP1 and R20SP2
10. Reconnect the compute port that was disconnected.
Expected 1. After sourcing to overcloudrc, you should see the prompt change to [stack@undercloud
Result (overcloudrc) ~]$
2. VXLAN networks have been created successfully.
3. VMs have been created successfully.
(ens6f1)
5. The connectivity check should run constantly and successfully.
Expected Output:


9. Check in Zabbix that you have one alarm for each compute that is <active
overcloudovscompute-2)
Expected Output:

Status OK | not OK
Comments
DN09271553 1-0 © 2021 Nokia 73

R20SP1 and R20SP2
4.1.6.11 Basic Connectivity Test - OVS L2 Population
Description VXLAN network connectivity is isolated
Test Case ID
Objective Verify that the VXLAN networks are isolated and that broadcast traffic from one network is
not present on any other network.
Estimated 1 Hour
Duration
Supported All
From Version

2. Three (3) available computes.
Test Execution 1. Create 2 internal VXLAN networks (for example vxlan-net-1 and vxlan-net-2).
2. Create 1 VM (VM-1) on compute 1 using vxlan-net-1, 1 VM (VM-2) on compute 2 using
vxlan-net-1, 1 VM (VM-3) on compute 3 using vxlan-net-2.
3. Login to VM-1 via SSH or console and send a constant ARP broadcast.
Note: One way to achieve this is to send a ping to an unallocated IP address

from the network vxlan-net-1.
ping <unallocated IP address from vxlan-net-1>
4. Let the broadcast ARP traffic run until the end of the test.
5. Login to all the 3 used computes and run the command tcpdump:
sudo tcpdump -nnepi any arp
Expected 1. The networks are created.

Result 2. VMs are successfully created.
3. VMs are allocated with an IP address from the given network.
4. The broadcast ARP traffic should run constantly until the end of the test.
5. Broadcast traffic is received and shown in tcpdump only for the computes that have a
VM from network vxlan-net-1 which are compute 1 and compute 2. Compute 3, which
has a VM from network vxlan-net-2, should not receive the ARP requests.
Status OK | not OK
Comments
DN09271553 1-0 © 2021 Nokia 74

R20SP1 and R20SP2
4.1.6.12 HugePages
Description HugePages Configuration Validation.
Test Case ID
Objective Validate that HugePages configuration and availability.
Estimated 1 Hour
Duration
Supported All
From Version
Prerequisite/s The CBIS cluster is installed with at least 1 compute node with 1G HugePages enabled.
Validate the compute configured with huge pages:
cbis-admin@overcloud-compute-0 ~]$ less /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-3.10.0-327.28.3.el7.x86_64
root=UUID=b977c2c1-dc3c-4853-a353-0598c429fce6 ro console=tty0
console=ttyS0,115200n8 crashkernel=auto rhgb quiet intel_iommu=on
default_hugepagesz=1G hugepagesz=1G hugepages=87 isolcpus=2-11,13-
23,26-35,37-47
Test Execution 1. Show that HugePages is configured on both the compute and OpenStack (in which the
host is part of the relevant huge pages host aggregate).
salt 'Compute*' cmd.run "grep -i Huge /proc/meminfo | grep -Ev

'Anon|Rsvd|Surp'"
nova aggregate-list | grep HugePages1G| 1 | HugePages1G | -
nova aggregate-details HugePages1G
+----+-------------+-------------------+-------------------------
---------------------------------------------+-------------------
---------+
| Id | Name | Availability Zone | Hosts | Metadata |
+----+-------------+-------------------+-------------------------
---------------------------------------------+-------------------
---------+
| 1 | HugePages1G | - | <the compute on which it was enabled
should appear in this list> | 'hw:mem_page_size=1048576' |
+----+-------------+-------------------+-------------------------
---------------------------------------------+-------------------
---------+
2. Create a matching flavor.
nova flavor-create ATPtest auto 10240 0 2
DN09271553 1-0 © 2021 Nokia 75

R20SP1 and R20SP2
nova flavor-key ATPtest set hw:mem_page_size=1048576
3. Instantiate a VM on the compute node with hugepages.
nova boot --flavor d7f1e705-b8ff-49b4-8fd9-55fb8df3a556

--image bfab4edd-b34c-409d-be35-3da1470eadb8 --nic net-
id=cbf8f8f0-a840-489d-a47d-b925f8a478f9 --security-groups
f2dc6eba-b117-411b-9fe8-ddbaf88e8653 ATP_HugePages_VM
4. Check the HugePages used amount again.
salt 'compute*' cmd.run "grep -i Huge /proc/meminfo | grep -Ev

'Anon|Rsvd|Surp'"
Expected 1. HugePages configuration is listed on both the compute and OpenStack.

Result 2. Flavor is created with HugePages metadata.
3. The instantiated VM uses the configured HugePages.
4. The HugePages values change.
Status OK | not OK
Comments
4.1.6.13 NUMA Awareness
Description Test the NUMA awareness behavior.
Test Case ID
Objective Verify the proper behavior of Nova NUMA awareness.
Estimated 2 hours.
Duration
Supported All
From Version
Prerequisite/s 1. SR-IOV compute free from instances.

2. CPU allocation ratio should be 1.0.
3. Flavors should be set with cpu_policy=dedicated.
4. The /etc/nova/nova.conf file in the controllers has the following default filters:
• AggregateInstanceExtraSpecsFilter
• NUMATopologyFilter
salt 'controller*' cmd.run "grep scheduler_default_filters= /etc/

nova/nova.conf | grep -v '^[ ]*#' | sed 's/[=,]/\n/g' | grep -e
AggregateInstanceExtraSpecsFilter -e NUMATopologyFilter"
DN09271553 1-0 © 2021 Nokia 76

R20SP1 and R20SP2
Test Execution 1. Check what computes are SR-IOV computes.

From the Undercloud VM, run:
source /home/stack overcloudrc
OpenStack aggregate shows sriov_performance_compute_nodes.
After checking which are the SR-IOV computes, select spontaneously 1 SR-IOV
compute that you will use to create VMs upon.
Make sure that this compute is empty (no existing instances).

2. Login to the SR-IOV compute upon which you will create VMs and run the following
command to understand the mapping of each physical interface to logic physnets:
sudo grep 'pci_passthrough_whitelist =' /etc/nova/nova.conf |

grep -v '#'
3. Login to the SR-IOV compute upon which you will create VMs and run the following
command to understand what physical interface goes to what NUMA (0 or 1):
cat /sys/class/net/ens1f0/device/numa_node
On AirFrame combo it should be allocated as such:
• ens1f0 and ens1f1 (physnet 1 and 2) goes to NUMA 0

• ens12f0 and ens12f1 (physnet 3 and 4) goes to NUMA 1
4. Check the PCPUs allocation on each NUMA on the SR-IOV compute.
From the SR-IOV compute, run the following command:
numactl --hardware | grep cpus
NOTICE: The PCPUs presented in the numactl command do not take

into account that some PCPUs are isolated from the operating system. To
understand which PCPUs are isolated for the SR-IOV compute operating
system, run the following command from the compute:
sudo -S grep CPUAffinity= /etc/systemd/system.conf| sed -e 's/\

<CPUAffinity\>=//g'
Remember: When PCPUs are isolated, some PCPUs are taken from NUMA
0 and others from NUMA 1. The system always takes an even number of
PCPUs from each NUMA according to the pCPU siblings.
Example
DN09271553 1-0 © 2021 Nokia 77

R20SP1 and R20SP2
• In the case of 6 isolated PCPUs (default), the system will take 4 PCPUs
from 1 NUMA and 2 more PCPUs from the second NUMA.
• In detail, the first 1 siblings pair from NUMA 0 (2 pCPU), the second 1
siblings pair from NUMA 1 (2 pCPU), and finally the 1 siblings pair from
NUMA 0 (2 pCPU).
• This is important to understand in case you receive the following error,
"not enough resources available" error when creating VM".
5. Ensure that the SR-IOV compute that you are using is cleared from VMs. If not, remove
all VMs from that compute.
6. Create via CLI or via Horizon, 2 provider vlan networks, deploying one for each NUMA,
depending on the corresponding physical interface (from physnet 1 to physnet 4).
For each network create also subnet and a port.
CLI Example for physnet 1:
neutron net-create SR-IOV_NETWORK_PHYSNET1 --provider:network_

type=vlan --admin-state-up --provider:segmentation_id=400 --
provider:physical_network=physnet1 --shared
neutron subnet-create SR-IOV_NETWORK_PHYSNET1 110.110.110.0/
24 --gateway 110.110.110.1 --name SR-IOV_SUBNET_PHYSNET1 --
allocation-pool start=110.110.110.10,end=110.110.110.100
neutron port-create SR-IOV_NETWORK_PHYSNET1 --binding:vnic-
type direct --name SR-IOV_PORT_PHYSNET1
7. Create a flavor with the number of available CPUs in NUMA-0 with SR-IOV and CPU
pinning flags.
a. To calculate how many available CPUs there are in NUMA-0, see the following
examples.
b. If the compute has 48 CPUs, and 6 CPUs are isolated, then 42 CPUS are left.
c. Knowing that 4 of the isolated CPUs are from NUMA-0, and 2 CPUs from NUMA-1,
we can see that NUMA-0 have 20 CPUs, and NUMA-1 have 22 CPUs.
Examples
nova flavor-create sriov_pinning_4vcpu_flavor auto 0 0 4

nova flavor-key sriov_pinning_4vcpu_flavor set SR-IOV=True
nova flavor-key sriov_pinning_4vcpu_flavor set hw:cpu_
policy=dedicated
8. Create VM with the number of available CPUs in NUMA-0, using the matching network
for the NUMA.
To validate what PCPUs VM uses:
1. Login to the compute and run:
sudo -i
DN09271553 1-0 © 2021 Nokia 78

R20SP1 and R20SP2
2. Get the instance ID
virsh list
3. Obtain the used PCPUs by the VM
virsh vcpupin <instance ID>
4. Check if all the used PCPUs by the VM are taken from NUMA 0
run numactl --hardware | grep cpus
Example
nova boot --flavor 7ae9aa4a-d814-4d85-bcb8-4afcd3b61f5d

--image 03872cad-4ede-4436-8ff0-95818e3920f4 --nic port-
id=d83f6c0a-8198-4adc-a6ab-acb1db4c64e5 --availability-
zone=sriov_zone:overcloud-sriovcompute-1.localdomain <VM name>
9. Create a flavor with the number of available CPUs in NUMA-1 minus one, with SR-IOV
and CPU pinning flags.
Examples

policy=dedicated
10. Create VM with the number of available CPUs in NUMA-1 minus one, using the
matching network for the NUMA.
Example

11. Create a flavor with 2 CPUs, with SR-IOV and CPU pinning flags.
Example

policy=dedicated
12. Create VM with 2 CPUs for NUMA-1, using the matching network for the NUMA.
DN09271553 1-0 © 2021 Nokia 79

R20SP1 and R20SP2
Example

13. Create a flavor with 1 CPU, with SR-IOV and CPU pinning flags.
Example

policy=dedicated
14. Create VM with 1 CPU for NUMA-1, using the matching network for the NUMA.
Example

Expected 1. Make sure this compute is empty (no existing instances).

Result 2. Example of the command output on AirFrame combo:
pci_passthrough_whitelist = {"devname":"ens1f0","physical_
network":"physnet1"}
3. Network devices NUMA allocation is displayed.
4. PCPUs allocation is displayed.
5. SR-IOV compute is cleared from VMs.
6. 2 networks are created. Each network uses different physnet (physnets 1 to 4).
7. Flavor created.
8. The VM was created successfully. Validate that the VM CPUs are taken from NUMA 0,
and that NUMA 0 is in full capacity.
9. Flavor created.
and that NUMA 1 have only one free vCPU
11. Flavor created.
DN09271553 1-0 © 2021 Nokia 80

R20SP1 and R20SP2
12. VM creation/boot failed.

13. Flavor created.
and that NUMA 1 is in full capacity.
Status OK | not OK
Comments
4.1.6.14 OVS-DPDK
Description Consuming ovs-dpdk VMs.
Test Case ID
Objective Verify that the system deploys the VM with ovs-dpdk networking.
Estimated 5 hours
Duration
Supported CBIS 18.5

From Version
Prerequisite/s CBIS node is deployed successfully with at least one DPDK dedicated compute.
Test Execution 1. Make sure you have an availability-zone with DPDK computes only (at least one DPDK
compute). If not, create an availability-zone DPDK_zone and add at least one DPDK
compute into that zone.
2. Make sure you have a flavor with HugePages extra spec. If the DPDK host group is
configured with 1G HugePages (default) the flavor needs to be configured accordingly
with 1G HugePages extra spec.
Configure HugePages on flavor example (CLI):
source ~/overcloudrc && openstack flavor set <flavor name> --

property hw:mem_page_size=large
3. Make sure you have a Security Group which allows ingress and egress ICMP packets.
4. Create a VM on the DPDK_zone availability-zone using the permissive security group
and the flavor with the HugePages metadata.
5. Wait until the VM:
• is ACTIVE
• is running
• its operating system is fully loaded
6. Send ICMP (ping) packets to the VM.
DN09271553 1-0 © 2021 Nokia 81

R20SP1 and R20SP2
Expected 1. There is an availability zone with DPDK only computes.

Result 2. There is a flavor that is configured with the HugePages metadata.
3. There is a permissive security group for connectivity check to the VM.
4. The VM is successfully created on a DPDK compute.
5. The VM is active and running.
6. The VM is accessible.
Status OK | not OK
Comments
4.1.6.15 CPU Pinning
Description CPU Pinning testing.
Test Case ID
Objective Verify that NOVA CPU pinning operates as expected. Items checked:
• Simple CPU pinning with different number of CPUs (up to the number in NUMA).
• CPU pinning with NUMA awareness (positive and negative).
Estimated 2 Hours
Duration
Supported All
From Version
Prerequisite/s The latest CBIS Manager Manual is available and the CBIS Cluster is installed successfully.
Note: You need to account for how many CPUs were allocated to the OS.
If the compute has total of 48 CPUs and you have allocated 6 isolated CPUs for the OS, the
VMs will be able to use only 42 CPUs.
If you wish to verify how many isolated CPUs were allocated for the OS, check in /home/
stack/user_config.yaml.
Test Execution 1. Create a flavor with dedicated CPU pinning as follows:
nova flavor-create m1.pinning auto 4096 0 6

openstack flavor set m1.pinning --property hw:cpu_
policy=dedicated
nova flavor-show m1.pinning
Create several VMs using this flavor to use all CPUs on the
compute.
DN09271553 1-0 © 2021 Nokia 82

R20SP1 and R20SP2
2. Check that each VM VCPU uses different CPUs and the CPUs are not overlapping.
3. To connect to the compute use the command below (depending on the type of the
compute):
c_ovs<compute number> or
c_sriov<compute number>
for i in `sudo virsh list | grep -vE '"--"|Name' |awk '{print
$2}'`; do echo; echo $i:; sudo virsh dumpxml $i | grep cpu;
done
4. Try to exceed the maximum number of CPUs allowed by the NUMAs. Create an extra
VM after all the CPUs are already used by existing VMs.
5. After exceeding the NUMAs CPU capacity, delete several VMs to free CPUs. Re-create
more VMs and verify that the CPUs are reused.
6. Leave 2 CPUs free on each NUMA and try to create a VM with 3 or more VCPUs.
7. Open one of the dedicated CPU pinning VM consoles and run the command "yes > /
dev/null &".
8. Connect using SSH to the compute which holds the VM and run "top".
Expected 1. A flavor with dedicated CPU pinning was created.

Result 2. If the compute has 24 CPUs available for the VMs, 6 VMs with 4 VCP each can be
created.
3. The dedicated CPU pinning VMs each use different CPUs and do not overlap.
4. A failure with the expected resources error occurs.
5. The CPUs became available and were reused.
6. A failure with the expected resources error occurs.
7. The VMs CPU usage is maximized.
8. The expected CPUs are shown in use inside the compute’s environment.
Status OK | not OK
Comments
4.1.6.16 CPU Allocation Ratio per Host Group
Description Create enough VMs to verify CPU allocation ratio limits are not exceeded
Test Case ID
Objective Verify that the CPU allocation ratio limits are not exceeded
Estimated 1 Hour
Duration
Supported All
From Version
DN09271553 1-0 © 2021 Nokia 83

R20SP1 and R20SP2
Choose a compute you want to validate CPU allocation ratio on and make sure this
compute is empty from VMs.
It is important to understand that the CPU allocation ratio is meaningless when using CPU
pinning. Therefore, you should use a flavor without CPU pinning metadata.
Test Execution 1. Deploy CBIS with default values cpu_allocation_ratio: 1.0.
If CBIS is already deployed check the current cpu allocation ratio in the computes.
c <compute number>
sudo grep 'cpu_allocation_ratio' /var/lib/config-data/puppet-
generated/nova_libvirt/etc/nova/nova.conf
If the cpu_allocation_ratio is not = 1.0 change it to 1.0 in /var/lib/config-data/

puppet-generated/nova_libvirt/etc/nova/nova.conf and restart nova_
compute and nova_libvird services
sudo docker restart nova_compute
sudo docker restart nova_libvirt
Wait 2 minutes for the services to restart and check if they are running.
sudo sudo docker ps

2. Calculate the number of CPU's in the compute available for the VMs.
Total available PCPUs minus the number of PCPUs isolated for the compute OS.
To check the number of isolated PCPUs on specific compute
c <compute number>
sudo -S grep CPUAffinity= /etc/systemd/system.conf| sed -e 's/\
<CPUAffinity\>=//g'
3. Create 1 or more VMs that will allocate all the available CPUs for the VMs.
Note: It does not matter how it is done. It can be with many VMs that have 1
VCPU each or even 1 VM that uses a flavor with all the VCPUs.
4. With cpu_allocation_ratio=1.0 configured in nova you can create 1:1 ratio of VCPUs.
Meaning, if you have 42 available PCPUs for the VMs you will be able to create
maximum of 42 VCPUs. After that if you try to create more VMs it should fail.
Check that you cannot create more VMs after reaching the full allocation of the PCPUs.
This is how to check the total CPUs against the used CPUs in a compute:
source /home/cbis-admin/overcloudrc
openstack host show overcloud-(ovs|sriovperformancecompute
)-(desired computer number).localdomain
DN09271553 1-0 © 2021 Nokia 84

R20SP1 and R20SP2
5. From within the compute you are working on, change the CPU allocation ratio to 2.0.
To connect to the compute, use the command below (depending on the type of the
compute):
c_ovs <compute number> or c_sriov <compute number>
change in /var/lib/config-data/puppet-generated/nova_libvirt/
etc/nova/nova.conf the cpu_allocation_ratio=1.0 parameter to cpu_
allocation_ratio=2.0 and restart nova_compute and nova_libvird services

Wait 1 minute for the services to restart and check if they are running.
sudo sudo docker ps

6. In step 4, additional VMs could not be created since maximum VCPUs ratio was
reached.
After changing the CPU ratio to 2:1, use of x2 VCPUs is enabled.
Check that additional VMs can be created according to the new ratio.
7. Check that you cannot create more VMs after reaching the full allocation of the PCPUs.
This is how to check the total CPUs against the used CPUs in a compute:
c_# <compute number>

source /home/cbis-admin/overcloudrc
openstack host show overcloud-#compute-#.localdomain
8. From the compute, change the CPU allocation ratio back to original value 1.0
To connect to the compute, use the command below (depending on the type of the
compute):
c_ovs <compute number> or c_sriov <compute number>
change in /var/lib/config-data/puppet-generated/nova_libvirt/
etc/nova/nova.conf the cpu_allocation_ratio=1.0 parameter to cpu_
allocation_ratio=2.0 and restart nova_compute and nova_libvirt services
Wait 2 minites for the services to restart and check if they are running.
sudo sudo docker ps
9. Delete the VMs
Expected 1. Deployment was successful, and cpu_allocation_ratio have value of 1.0 and services
Result are active/running.
2. Example case HPc7k -> 48 cores - 6 dedicated = 42 ; -> 42 vCPU available for VM
instances (ratio: 1.0).
3. Create success max number of VMs (1*vCPU flavor).
4. VM creation fail.
DN09271553 1-0 © 2021 Nokia 85

R20SP1 and R20SP2
5. CPU allocation ration value changed and services are active/running.

6. Additional VMs creation was successful.
7. VM creation fail
8. CPU allocation ratio changed back to original value 1.0 successfully.
9. Delete success.
Status OK |
Comments
4.1.7 Software RAID 1
Description Software RAID 1
Test Case ID https://jiradc2.ext.net.nokia.com/browse/CBIS-13529
Objective The software RAID 1 feature enables the configuration of /dev/sda and /dev/sdb
physical disks in RAID 1 allowing redundancy and helping with avoiding the operating
system from failing upon disk failure.
Estimated 60 minutes ~ (not including the CBIS deployment and preparartions to obtain another disk).
Duration
Supported CBIS 19A

From Version
Prerequisite/s Software RAID can only be configured on controllers or computes and cannot be configured
on storage nodes or monitoring servers.
Software RAID 1 is only available for the Airframe RM/OR family from RM/OR 17 and
above.
Software RAID can only be enabled on a fresh CBIS deployment and cannot be deployed
post-deployment. However, using the custom template feature, you can scale out a new
compute host group with software RAID 1 enabled. (only relevant for computes).
Software RAID 1 requires the /dev/sdb disk to be unused and unpartitioned. This means,
that software RAID 1 can't work along-side other features that may allocate the /dev/sdb
disk to their own benefit. Features such as ELK DB disk and local storage. Local storage (if
used – default off) can be used on another disk other then /dev/sdb or /dev/sda and
the ELK DB can be configured on different disk as well other then /dev/sdb or /dev/
sda (assuming there is a third or more disks within the servers).
Test Execution 1. Login to the CBIS Manager.

2. Navigate to LIFE CYCLE MANAGEMENT > CBIS Installation.
3. Under Customize host groups enable the software Raid 1 (/dev/sda, /dev/sdb)
slider for both the controller host group and the compute host groups.
DN09271553 1-0 © 2021 Nokia 86

R20SP1 and R20SP2
4. Continue with the rest of the CBIS deployment configuration and deploy CBIS.
5. Once CBIS is successfully deployed, select the under-test servers that have software
RAID 1 configured (controller or compute).
6. Physically unplug the sda disk, (to simulate a failed disk).
7. Follow the CBIS Operation Manual - Replace a failed disk in Software RAID 1
procedure to properly remove the failed disk from the operating system.
8. Follow the CBIS Operation Manual - Adding a new disk procedure to add a new disk
to replace the failed disk. (If the disk is not new, it has to be unpartitioned and formatted
like any new out of the box disk).
9. Check the RAID 1 status using cat /proc/mdstat.
Expected 1. Successfully logged-in to CBIS Manager.

Result 2. The CBIS Installation page is successfully opened.
3. Software RAID 1 is enabled on all the relevant host groups.
4. CBIS is deployed successfully.
5. Under-test server is selected.
6. Even after unplugging, the operating system still works seamlessly without any issues/
errors expected.
7. The operating system still works seamlessly without any issues/errors expected and the
removal of the disk successfully finished.
8. Adding a new disk operation should be successful and the operating system should
operate seamlessly without any issues/errors expected.
9. The RAID 1 status should be "OK" for both the disks. The command should show (2/2)
as seen in the example below:
[root@overcloud-controller-rome-0 ~]# cat /proc/mdstat

Personalities : [raid1]
active raid1 sda2[2] sdb2[1]
390574144 blocks super 1.2 [2/2] [UU]
bitmap: 3/3 pages [12KB], 65536KB chunk
active raid1 sda1[2] sdb1[1]
3072 blocks super 1.2 [2/2] [UU]
unused devices: <none>
Status OK | not OK
DN09271553 1-0 © 2021 Nokia 87

R20SP1 and R20SP2
Comments
4.2 CBIS Non-Functional Test Cases
4.2.1 CBIS High Availability
4.2.1.1 Neutron Router HA Test (3 Controllers Setup)
Description Check network connectivity via a router while rebooting the active controller.
Test Case ID
Objective Verify that network connectivity via a router is operational after rebooting the active
controller.
Estimated 0.5 Hour

Duration
Supported All
From Version
Prerequisite/s The CBIS Cluster is installed successfully with 3 controllers.
Test Execution
Note: Unless requested differently, all the following test commands should
be executed from the Undercloud VM while sourced to the admin tenant
(overcloudrc).
1. Login to the admin tenant (overcloudrc).
2. Create 2 VXLAN networks.
a. Create the networks:
openstack network create --provider-network-type vxlan --

external neutron-router-ha-network-1
openstack network create --provider-network-type vxlan --
external neutron-router-ha-network-2
b. Obtain the network ID:
openstack network list | grep neutron-router-ha-network
DN09271553 1-0 © 2021 Nokia 88

R20SP1 and R20SP2
c. Create subnet for each network:
openstack subnet create --network <network-1 ID> --gateway

192.168.1.1 --subnet-range 192.168.1.0/24 neutron-router-ha-
subnet-1
openstack subnet create --network <network-2 ID> --gateway
192.168.2.1 --subnet-range 192.168.2.0/24 neutron-router-ha-
subnet-2
d. Obtain the subnets ID:
openstack subnet list | grep neutron-router-ha-subnet
3. Create a permissive security group:
a. Create security group:
openstack security group create neutron-router-ha-security-

group
b. Create permissive security group rules and attach them to the created security
group:
openstack security group rule create --ingress --protocol

ICMP neutron-router-ha-security-group
openstack security group rule create --egress --protocol
ICMP neutron-router-ha-security-group
c. Obtain the security group ID as follows:
openstack security group show neutron-router-ha-security-

group
4. Create 2 VMs each with different VXLAN network:
openstack server create --image <image ID> --flavor <flavor ID>

--security-group <neutron-router-ha-security-group ID> --network
<neutron-router-ha-network-1> neutron-router-ha-vm-1
openstack server create --image <image ID> --flavor <flavor ID>
--security-group <neutron-router-ha-security-group ID> --network
<neutron-router-ha-network-2> neutron-router-ha-vm-2
5. Create a neutron router with an interface from each VXLAN network:
a. Create neutron router:
openstack router create --ha neutron-router-ha-router
b. Obtain the router ID:
openstack router show neutron-router-ha-router
DN09271553 1-0 © 2021 Nokia 89

R20SP1 and R20SP2
c. Attach the router with the created network interfaces:
openstack router add subnet <router ID> <neutron-router-ha-

subnet-1 ID>
openstack router add subnet <router ID> <neutron-router-ha-
subnet-2 ID>
6. Ensure that router L3 agents are alive in each controller:
openstack network agent list --router neutron-router-ha-router

--long
7. Perform ICMP (ping) continual connectivity check between the 2 created VMs.
8. Reboot the controller where the L3 agent is active.
Note: The following steps will request the that the active controller is
rebooted. Before, during and post reboot, it is important to constantly monitor
the VMs ICMP connectivity to check that no unexpected packet drops or
connectivity termination occurs.
a. Locate the active HA state controller host:
openstack network agent list --router neutron-router-ha-

router --long | grep active
b. Login to the active controller host.

c. Reboot the controller:
sudo reboot
d. Check that another l3-agent controller became active:
openstack network agent list --router neutron-router-ha-

router --long
Note: It may take several minutes for the other l3-agent controller to show
active.
e. Check that the ICMP (ping) continual connectivity check between the VMs is still
running.
9. Wait for the rebooted controller to finish its reboot process and become fully active and
operational (normally, it takes between 10 to 15 minutes) and validate that its L3 agent is
Alive and UP.
Note: Even after the rebooted controller is accessible via SSH it takes several
minutes for the L3 agent of that controller to become Alive and UP.
DN09271553 1-0 © 2021 Nokia 90

R20SP1 and R20SP2
Expected 1. Successfully sourced to the overcloudrc file.

Result 2. VXLAN networks are successfully created.
3. Security group is successfully created.
4. VMs are successfully created.
5. Router is successfully created, and the networks successfully attached to the router.
6. All controllers are ‘alive’ and one controller ha_state is ‘active’.
7. The ICMP (ping) check between the VMs should work constantly and fluently. Expecting
no out of the order packet drops.
Note: If there are unordinary packet drops, check the network quality before
continuing to the next steps.
8. The controller successfully reboots and during the controller reboot the ICMP (ping)
connectivity show continue running without packet drops. In the meanwhile, a new
controller should take the lead for the l3-agent active controller.
9. The controller should be back up and its L3 agent should be Alive and UP.
Status OK | not OK
Comments
4.2.1.2 Controller Nodes HA (Server Failovers and CBIS Monitoring)
Description Power-off active controller and verify CBIS functionality during and after the failure.
Test Case ID
Objective Verify that CBIS continues to function after a controller failure.
Estimated 1 Hour.
Duration
Supported All.
From Version
Prerequisite/s The CBIS Cluster is installed successfully with 3 controllers.
Do not start this test case if:
• Ceph is in error state.

• In pacemaker Galera is not primary in 3 controllers.
• In pacemaker Redis is not 2 Secondary and 1 Primary.
• In pacemaker except Galera and Redis all other services status is Started.
Test Execution 1. Create 2 IPv4 VXLAN networks and 2 IPv6 VXLAN networks. Each network on different
subnet.
2. Create a router and add the networks interfaces (the default gateway of the networks).
DN09271553 1-0 © 2021 Nokia 91

R20SP1 and R20SP2
3. Create a security group that allow ingress and egress ICMP traffic for both IPv4 and
IPv6 to all addresses.
4. Create VM for each network with the created security group.
5. Create 4 volumes.
Note: The volume size is set according to the user requirements and if there
are no specific requirements, use the default values.
6. Attach and mount each volume to each VM and check the network connectivity and
volume accessibility of the VMs.
7. Run continuously ping connectivity check between the IPv4 VMs and between the IPv6
VMs. The connectivity check needs to run though all the test period.
8. Check which controller is running the internal API VIP as follows:
a. From the Undercloud VM run the following:
source ~/overcloudrc && openstack catalog list | grep

internal | grep -v swift
b. The IP address that will be presented is the internal API VIP. To find which controller
holds the internal API VIP, from one of the controllers run the command:
sudo pcs status | grep <internal API VIP>
Note: In this test, we will be calling the controller which holds the internal API
VIP, the PRIMARY controller, and the rest of the controllers will be referred to
as SECONDARY controllers.
9. Login to the PRIMARY controller.

10. Capture and save the "sudo docker ps" command output from the PRIMARY
controller.
11. Capture and save the “sudo pcs status” command output from the PRIMARY
controller.
12. Capture and save the “sudo ceph -s” command output from the PRIMARY controller.
13. Check that Zabbix, Horizon, Ceph and Kibana are working.
14. Power off the PRIMARY controller. Allow 5 minutes of grace period. Run:
$ sudo shutdown +5
15. Disable networking during the 5-minute grace:
$ sudo systemctl stop network
16. Verify that there is still connectivity to the previously deployed VMs.
DN09271553 1-0 © 2021 Nokia 92

R20SP1 and R20SP2
17. At this point, the internal API VIP should have failed-over to a new controller. Perform
the same steps as previously mentioned in this test to obtain the new PRIMARY
controller.
18. Login to the new PRIMARY controller.
19. Capture and save the “sudo pcs status” command output.
20. Check that Zabbix, Kibana and Horizon are working.
21. Create VMs (SR-IOV and OVS) and check the connectivity to the VMs.
22. Start the powered off controller using the same method that was used to power off the
controller. This can now be used to power on the controller.
Note: It takes around 10-12 minutes ~ (depends on the hardware type and
shape) for the controller to become fully functional again. Note that it might
take few more minutes for the controllers to sync. This depends on how much
time the controller was down and how much data needs to be synchronized to
the started controller.
23. Login to previously powered off controller.

24. Capture and save the "sudo docker ps" command output.
25. Capture and save the “sudo pcs status” command output.
26. Capture and save the “sudo ceph -s” command output.
27. Check that Kibana, Zabbix, Ceph and Horizon are working.
28. Verify that the connectivity is still working as expected between the previously deployed
VMs.
Expected 1. Networks are created.

Result 2. The router is created and the adding of the network's interfaces succeeded.
3. The security group is created.
4. VMs are created and the network connectivity validation has succeeded.
5. The volumes are created.
6. The volumes are attached and mounted. Use df -h inside the VMs to view and verify
the mount volumes.
7. Ping is running in the back-group.
8. The PRIMARY controller is obtained.
9. Successfully logged in to the PRIMARY controller.
10. The “sudo docker ps” command output should show all the running containers. No
container should show an unhealthy status (some containers might do not show the
healthy/unhealthy status at all. This is not an issue).
11. The “sudo pcs status” command output should show all the containers as Started
except for galera and redis. All 3 galera containers should show Primary and from the 3
redis containers 2 should act as Secondary and 1 as Primary
DN09271553 1-0 © 2021 Nokia 93

R20SP1 and R20SP2
Note: You might see at the bottom of the sudo pcs status output, one or
more failed actions. Note that these contain a history of previous pacemaker-
controlled services that failed. These do not reflect the status of the services
and should be ignored in this specific test.
12. The “sudo ceph -s” command output should show “HEALTH_OK”
13. Zabbix, Horizon, Ceph and Kibana portals are all reachable and are operating. Zabbix
shows alarms (in case the trigger requirement is met). Kibana receives logs.
14. The compute is powered off.
15. The connectivity check to the VMs should succeed. No packets should be lost during
the powering off the controller. However, if there is a minimal packet loss during the
controller power off, check to see if it is within the acceptable packet loss threshold of
the VNF.
16. You should see that the internal API now resides on a new controller.
17. Successfully logged in to the new PRIMARY controller.
18. The “sudo pcs status” command should show all the services that were residing on
the powered off controller as Stopped except for galera and redis.
a. For galera, 2 containers should show Primary and the container that is on the
powered off controller should display as Stopped.
b. For the redis containers 1 should act as Secondary, 1 should act as Primary and the
one that was on the powered off controller should display as Stopped.
c. For the other containers that are not redis or galera, the containers should show 2
out of 3 Started and 1 out of 3 should display as Stopped.
Note: For some specific containers, that are working as active-standby

such as zabbix-server, the status should show as Started.
Note: You might notice at the bottom of the sudo pcs status output, one
or more failed actions. These contain a history of previous pacemaker-
controlled services that failed. They do not reflect the status of the services
Note: The “sudo ceph -s” command should show “WARNING”. This
is because one Ceph Monitoring is now down and Ceph reports this. If
this is the only warning, then this is expected and Ceph should still work
normally.
19. Kibana and Zabbix portals are reachable and are operating. Zabbix shows alarms (in
case the trigger requirement meets). Kibana receives logs.
20. The VMs are created and reply to the connectivity check successfully.
21. Start the powered off controller.
22. You should be able to login to the previously powered off controller.
DN09271553 1-0 © 2021 Nokia 94

R20SP1 and R20SP2
23. The “sudo docker ps” command output should show identical information as before.
24. The “sudo pcs status” command should show all the containers as Started except
for galera and redis.
Note: All 3 galera containers should show Primary and from the 3 redis
containers 2 should act as Secondary and 1 as Primary.
Note: You might notice at the bottom of the sudo pcs status output one or
more failed actions. Note that these contain a history of previous pacemaker-
controlled services that failed. They do not reflect the status of the services
25. The “sudo ceph -s” command should show “HEALTH_OK”.

26. Kibana, Zabbix, Ceph and Horizon portals are reachable and are operating. Zabbix
shows alarms (in case the trigger requirement meets). Kibana receives logs.
27. The connectivity check to the VMs should succeed. There should be no expected packet
loss during the powering on of the controller. However, if there is a minimal packet loss
during the controller power off, check to see if it is within the acceptable packet loss
threshold of the VNF.
Status OK | not OK
Comments
4.2.1.3 Storage Volume Recovery (NetApp, EMC, HCI-Ceph, SN-Ceph failovers and CBIS monitoring)
Description Shutdown compute with instances having attached volumes.
Test Case ID
Objective Verify that instances with attached volumes auto-evacuate after a compute shutdown.
Estimated 0.5 Hour

Duration
Supported All versions (except Vitrage) supported from CBIS R20.

From Version
Prerequisite/s • The CBIS Cluster is installed successfully.

• There are at least 2 computes which can host OVS VXLAN VMs.
• The 2 computes should be empty, to allow evacuation without unexpected resources
limit.
• Auto-Evacuate is enabled – follow the Auto-Evacuate Procedure in the CBIS Operation
Manual.
• If initiating the VM on a specific availability-zone, verify that the zone has more than 1
empty compute.
DN09271553 1-0 © 2021 Nokia 95

R20SP1 and R20SP2
Test Execution 1. Create an OVS VXLAN VM and understand on which compute the VM was created on.
From the Undercloud VM, execute:
source ~/overcloudrc && openstack server show <created VM ID>
2. Create a volume (use the default values).

3. Attach the volume to the VM.
4. Login to the created VM to mount the newly created volume.
a. Check for the new volume index – the last volume in list:
fdisk -l (normally it will be /dev/vdb)
b. Create new directory to mount the volume to:
mkdir <new directory>
c. Format the volume (ext4):
mkfs -t ext4 </dev/device-name>
d. Mount the volume:
mount </dev/device-name> <new directory>
e. Configure the mount point to survive the reboot:
echo '</dev/device-name> <new directory> ext4 defaults 1 2'

>> /etc/fstab
5. Write data to the new mounted volume directory.
Note: It is not mandatory to fill all the volume capacity, although this is
possible if required. Several Megabytes will suffice.
6. Shut down the compute on which the created VM is running.
Note: Both graceful (poweroff command) and ungraceful (plug off the
electricity of the compute) are valid options for shutting down the compute.
7. Verify the VM was evacuated to another compute.
Note: It may take several minutes for the VM to fully rebuild on the new
compute.
From inside the VM, execute:
source ~/overcloudrc && openstack server show <created VM ID>
DN09271553 1-0 © 2021 Nokia 96

R20SP1 and R20SP2
Note: The estimated time for auto evacuate to trigger is 10 minutes.
8. Verify the VM is active, running and accessible.

9. Verify the volume is still attached to the evacuated VM.
10. Verify the created directory is still mounted.
From inside the VM, execute:
sudo df -h
Note: If the mounted directory does not appear in the output of df -h, mount
the directory again.
11. Verify that the created data still resides inside the mounted directory.
Expected 1. The VM was successfully created and is accessible.

Result 2. The volume is successfully created.
3. The volume is successfully attached to the VM.
4. The volume is successfully mounted.
5. Some data (at least several Megabytes) is written to the mounted directory.
6. The compute is successfully shutdown.
7. The VM successfully evacuated and now reside on another compute.
8. The VM is accessible and working as expected.
9. The volumes should still be attached to the VM.
10. The mount directory should still be presented.
11. The data should still exist as it was before the evacuation of the VM.
Status OK | not OK
Comments
4.2.2 CBIS Monitoring (Alarms and Logging)
4.2.2.1 Zabbix Monitoring (System Health)
Description Zabbix system health check.
Test Case ID
Objective Verify that Zabbix is operational.
Estimated 1 Hour.
Duration
DN09271553 1-0 © 2021 Nokia 97

R20SP1 and R20SP2
Supported All
From Version
Test Execution 1. Check that the Zabbix process is up and running.

2. Check that the Zabbix process does not use high CPU usage when idle.
3. Open the Zabbix GUI portal and login.
Log in to zabbix using one of the following URLs:
https://<horizon _VIP>/zabbix or https://<horizon _VIP>/zabbix
default credentials:
user: cbis-admin
password: zabbix
4. Check again that the CPU usage of the Zabbix process is stable and not high.
5. 5 minutes of GUI sanity to verify that all the tabs are accessible.
6. Check that Zabbix is monitoring all installed computes/controllers/ceph-
storage.
7. Check that Zabbix contains templates/tests/triggers/items related to CBIS (and not just
the Zabbix default templates/tests/triggers/items).
8. Inside Zabbix portal navigate to Monitoring > Dashboard and check that Zabbix show
alarms in the “Last 20 issues” field. If it does show an alarm, check in your system that
the alarm in Zabbix reflects a condition and that the alarm is not a false negative.
If the alarms are real, fix them and see that the alarms are cleared from Monitoring >
Dashboard
If Zabbix show alarms that are a false negative, there are 2 options:
a. There is an issue within Zabbix and need to investigate with support.

b. For some alarms it takes up to 1 hour for Zabbix to re-diagnose and show the real
state of the system. This timeout configuration may be different for each alarm.
Otherwise, if there are no alarms in Monitoring > Dashboard we assume that the
system is healthy and you may move to the next step.
9. Configure Zabbix to send all its triggered alarms SNMP traps to an SNMP server.
To configure Zabbix, see Registering for SNMP Traps in the latest CBIS Operation
Manual.
10. Intentionally create a fault in the system that will raise a corresponding alarm in Zabbix.
Example
Shutdown a compute host from the BMC (IPMI)
Example of Expected Output
ICMP, SSH and Zabbix agent alarms should raise in Zabbix. CEPH alarms may raise as
well in such scenario.
DN09271553 1-0 © 2021 Nokia 98

R20SP1 and R20SP2
Note: You should see under Zabbix > Configuration > Hosts > Open a
host such as overcloud-ovscompute-1.localdomain and then access the
‘Triggers’ tab. All the triggers that may raise an alarm if a condition is met for
that compute are shown.
11. Fix the issues that you created in the previous step such as powering up the compute
and see that after approximately 10 minutes, all the alarms that went up after shutting
down the compute, now disappear from Zabbix > Monitoring > Dashboard
12. Check the past event under Zabbix > Monitoring > Problems and validate that you see
all the alarms that were up from the previous steps.
13. Open the Horizon GUI portal and log in (there was a bug that Zabbix/Vitrage might kill
httpd processes).
Expected 1. Zabbix is installed.

Result 2. The Zabbix process is up and running.
3. The Zabbix process does not use high CPU usage when idle.
4. Successful login to the Zabbix GUI portal.
5. The Zabbix process CPU usage is stable and not high.
6. All the GUI tabs can be accessed and are working as expected.
7. Zabbix is monitoring all installed Computes/Controllers/ceph-storage nodes.
8. Zabbix templates/tests/triggers/items that related to CBIS exist.
9. Alarms in “Last 20 issues” section are shown and validated.
10. When an alarm is triggered in Zabbix, it runs an action script that sends a SNMP v2 trap
to your SNMP server.
11. An Alarm is raised which corresponds to the created fault.
12. Alarms were cleared.
13. Past events are displayed correctly.
14. Successful login to Horizon GUI.
Status OK | not OK
Comments
4.2.2.2 Zabbix Metrics Exporter
Description The Zabbix Metrics Exporter is a feature for exporting metrics from a Zabbix server
into a CSV file periodically.
Test Case ID
Objective On each of the controllers verify that the Zabbix metrics are exported and contains
the expected information.
The following columns are expected to present within the csv files:
DN09271553 1-0 © 2021 Nokia 99

R20SP1 and R20SP2
• timestamp
• hostID
• hostname
• templateItemID
• itemID
• name
• key
• units
• value
• error
Duration
Supported 18.0
From Version
Prerequisite/s It is advised to wait at least 3 hours from CBIS deployment until the execution of
this test, to allow the controllers to become filled with several metrics in the csv
files.
Test Execution 1. Login to one of the controllers. Check the existence of the directory /var/
log/zabbix/metrics.
Note: To allocate the controller which holds the OpenStack external

VIP execute the following command from the Undercloud VM:
salt "[cC]ontroller*" cmd.run "sudo pcs status

| grep ip-"
2. In that folder, see that the 10 last files, ordered by date, were created every 15
minutes, until current operating system time.
3. Check that the last 10 files end with “.csv” and not with “inprogress” (except
for the last one). If the last one is “in progress”, wait for it to be “.csv” (up to 15
minutes wait).
4. “less” the last csv file (must end with “.csv”). Example:
less zbx_metrics_11012018_0930.csv.
5. Make sure that this job is scheduled every 15 minutes:
sudo crontab -l | grep metrics
6. Repeat 1 – 5 on the rest of the controllers.
Expected 1. The folder exists.

Result
DN09271553 1-0 © 2021 Nokia 100

R20SP1 and R20SP2
2. The last 10 files were created every 15 minutes, until current time.
3. The last 10 files end with “.csv”.
4. The last line isn’t “cut” and contains these expected columns:
• timestamp
• hostID
• hostname
• templateItemID
• itemID
• name
• key
• units
• value
• error
Note: Except "units" and "error" columns, all the other columns
are expected to contain data while the "units" and "error"
columns will not present data if there is no relevant data to
display.
5. There is a cron job scheduled.
Example:
*/15 * * * * python /usr/lib/zabbix/alertscripts/zbx_

metrics_exporter.py > /var/log/zabbix/metrics/last_run.
status
6. The same results are found on the remaining controllers.
Status OK | not OK
Comments
4.2.2.3 HW Alarms Northbound Forwarding
Description Zabbix is pre-defined with the "General HW Templates" template which monitors SNMP
traps coming from the hardware switches.
By default, the SNMP traps, coming from the switch that will cause Zabbix to raise the
following alarms are:
• Contact Details Changes

• Device Name Change
• Hard boot occurred
• Ping check
• SNMP Authentication Failure
DN09271553 1-0 © 2021 Nokia 101

R20SP1 and R20SP2
• Soft boot occurred

• System Description Changed
• Network interfaces
Test Case ID
Objective Verify that Zabbix monitors the SNMP traps that were pushed from the switches.
Estimated 1 Hour.
Duration
Supported R19.0 MP2.

From Version
Test Execution 1. Configure the switch to send SNMP traps to the public VIP of the controllers. The public
VIP is the same IP address as the OpenStack Horizon web portal.
Common SNMP settings to configure in the switch:
• Community
• Access
• Trap destination
• SNMP version
Note: This test focuses on SNMPv2. The switch needs to be configured to

send the traps using SNMPv2.
NOTICE: Retrieving SNMP traps from the switch can be configured either
from PRE CBIS deployment from the CBIS Manager (recommended) or
POST CBIS deployment.
2. For PRE CBIS deployment from the CBIS Manager, perform the following steps and
then skip to step 6 (Create Zabbix alarm...).
a. From CBIS Manager navigate to CBIS Installation > Overcloud > Zabbix
Optional Parameters and check these 3 parameters:
• Add IPMI connections to hosts

• Add SNMP connections to hosts
• Add SNMP addresses of switches
DN09271553 1-0 © 2021 Nokia 102

R20SP1 and R20SP2
b. Click on the + icon and add your switch name, switch IP address, community string
and switch SNMP port.
c. After all the CBIS Manager parameters are configured, click on Deploy to start the
CBIS deployment.
d. To continue, skip to step 6 (Create Zabbix alarm...)
3. For POST CBIS deployment, continue with Step 4 (Login to the Zabbix portal...)
Note: The following two steps; #4 Login to the Zabbix portal and #5 Create
and configure “Host” for SNMP alarms, are only to be performed for post
CBIS deployment.
4. Login to the Zabbix portal via https://<OpenStack_External_VIP>/zabbix
DN09271553 1-0 © 2021 Nokia 103

R20SP1 and R20SP2
5. Create and configure Host for SNMP alarms.
a. Configuration > Hosts and click Create host.
b. Enter the Host name.

c. In the Groups field, click Select and select the group General HW Templates.
d. Remove the default Agent interfaces.
e. Add SNMP interfaces and the port (default 161) and type the IP address of the
switch that sends the SNMP traps.
f. Navigate to the Templates tab:
• Click Select. From the new window, open the Group drop-down list.
DN09271553 1-0 © 2021 Nokia 104

R20SP1 and R20SP2
• Click General HW Templates.

• Check the options for Template HW Switch General and Template HW
Switch Networking.
• Click Select.
• After the 2 templates are selected, click on the upper Add to apply the linked
templates.
g. Click Macros and within the Macro field enter {$SNMP_COMMUNITY}. In the
Value field, write the community string as configured in the switch (the value
"public" is only an example).
DN09271553 1-0 © 2021 Nokia 105

R20SP1 and R20SP2
h. Click Add to complete the configuration.

i. Open the Configuration > Hosts page and wait till SNMP under Availability is
green.
Note: After SNMP Availability becomes active (green), Zabbix should send
alarms about problematic interfaces in the switch (link down or administratively
down). Initially it takes up to an hour for Zabbix to collect all the information
(items and triggers).
6. Create Zabbix alarm by shutting down administratively an active interface in the switch.
7. Cancel an alarm by turning on the switch interface.
Expected 1. Switch is configured correctly.

Result 2. Logged in successfully to the Zabbix portal.
3. SNMP under Availability is green, configuration passed successfully.
4. Simulate the interface shutdown alarm:
DN09271553 1-0 © 2021 Nokia 106

R20SP1 and R20SP2
Note: There should be a received alarm at “system status”, “host status”, “last
20 issues” sections.
5. The Alarm will be cleared.
Status OK | not OK
Comments
4.2.2.4 Ceph Monitoring
Description Ceph monitoring dashboard check.
Test Case ID
Objective Verify that Ceph monitoring is operational.
Estimated 15 minutes ~
Duration
Supported CBIS 20
From Version
Prerequisite/s Setup is installed with the Ceph backend.
Test Execution 1. Open CBIS Manager.
DN09271553 1-0 © 2021 Nokia 107

R20SP1 and R20SP2
2. Select "EXTERNAL TOOLS" at main page.

3. Check that the Ceph monitor dashboard is enabled.
4. Click the Ceph monitor dashboard, and verify that it has opened.
5. Check that the display of the data are correct, such as - Ceph health, number of
monitors, pools, and logs.
6. Check the monitor main pages display – Cluster, Block and Filesystems.
Expected 1. CBIS manager is opened and running.

Result 2. The "EXTERNAL TOOLS" page has been reached.
3. The Ceph monitor dashboard is enabled.
4. The Ceph monitor dashboard has been opened.
5. The Ceph display data is correct, this includes Ceph health, number of monitors, pools,
and logs.
6. The monitor main pages display is oeprating normally and displaying clusters, blocks
and filesystems.
Status OK | not OK
Comments
4.2.2.5 Zabbix to monitor the Undercloud VM
Description Undercloud VM alarm visibility is implemented in Zabbix.
Test Case ID
Objective Verify that the Undercloud VM alarms are visible in Zabbix.
Estimated 0.5 hour.

Duration
Supported 20 SP2
From Version
Test Execution 1. Login to Zabbix portal https://<public_VIP>.
2. Trigger a Zabbix alarm on the Undercloud VM by performing the following actions:
a. Verify that the Glance API service is up and running:
[stack@undercloud (stackrc) ~]$ sudo docker ps | grep

glance_api
DN09271553 1-0 © 2021 Nokia 108

R20SP1 and R20SP2
b. Stop the Glance API service:
[root@undercloud (stackrc) ~]# sudo docker stop glance_api
c. Verify that the Glance API service is down:
[stack@undercloud (stackrc) ~]$ sudo docker ps -a | grep

glance_api
3. Cancel the Zabbix alarm by performing the following:
a. Start the Glance API service:
[root@undercloud (stackrc) ~]# sudo docker start glance_api
b. Verify that the Glance API service is up and running:
[stack@undercloud (stackrc) ~]$ sudo docker ps | grep

glance_api
Expected 1. User has been successfully entered and logged in.

Result 2. The Zabbix alarm [Glance API is down on undercloud] is triggered.
Note: It should take up to 30 seconds for the alarm to appear or disappear in

Zabbix.
3. The Zabbix alarm [Glance API is down on undercloud] is canceled.
Status OK | not OK
Comments
4.2.2.6 Alarm Manager (SNMP v2c, V3)
Description Send SNMP traps after creating an alarm.
Test Case ID
DN09271553 1-0 © 2021 Nokia 109

R20SP1 and R20SP2
Objective Verify that SNMP traps are sent for each alarm activity.
Estimated 1 Hour
Duration
Supported All
From Version
Prerequisite The latest CBIS Manager Manual is available and the CBIS Cluster is installed successfully.
Test Execution 1. On each controller, modify /etc/cbis-snmp/snmp_trap_dests.yaml and add the

destination and other parameters for where the SNMP V2 traps will be sent.
Follow CBIS Operation Manual Procedure – “Registering for SNMP Traps”, and
configure the IP address for which you want to send the SNMP traps.
2. Raise an alarm in Zabbix (set Zabbix trigger on).
3. Use the MIB browser program or Wireshark to examine the SNMP traps.
Expected 1. SNMP V2 trap destination set in all controllers.

Result 2. Alarm is raised after Zabbix was triggered.
3. SNMP trap is received on the destination.
Status OK | not OK
Comments
4.2.2.7 ELK
Description Kibana logging system check.
Test Case ID
Objective Verify that Kibana is operational and log collection is active.
Estimated 0.5 Hour.

Duration
Supported All
From Version
Prerequisite/s ELK is enabled in the CBIS Manager before the Overcloud installation as follows:
CBIS Manager > CBIS Installation > Overcloud > General Optional Parameters >
Deploy ELK
DN09271553 1-0 © 2021 Nokia 110

R20SP1 and R20SP2
Test Execution 1. Log into Kibana at the URL https://<$OpenStack_External_VIP>/kibana

2. Verify that the hosts charts are visible and contain data:
a. Navigate to the Dashboard page.

b. In the Search box type "host".
c. Open the item [Metricbeat System] Host overview.
d. From the Selection-metric drop-down boxes select the Cloud and Host you wish to
monitor.
e. Check that there are values presented in the [Metricbeat System] Host overview
page and that the values are accurate as expected.
Note: If the values are not as expected, refresh the web-page and check
again.
3. Create a new project (tenant).
Note: If there is an already existing non-admin project, use it instead of

creating a new one.
4. Create a VM using the created (or the existing) project.

5. Verify that the VMs charts are visible and contain data:
a. Navigate to the Dashboard page.

b. In the Search box type "vtop".
c. Open the item Instance-Vtop.
d. From the Vtop-Controls drop-down boxes select the Cloud, Project, Host and the
Instance you wish to monitor.
e. Check that there are values presented in the Instance-Vtop page and that the values
are accurate as expected.
Note: If the values are not as expected, refresh the web-page and check
again.
6. Check that logs are reaching Kibana as follows:
DN09271553 1-0 © 2021 Nokia 111

R20SP1 and R20SP2
a. Navigate to the Discover page.

b. From the filters drop-down list check that the default filters exist:
• cloud-*
• ipmitool-*
• ceph-*
• metricbeat-*
• openstack-*
• vtop*
c. Make sure the cloud-* filter is selected and logs are displayed.
Note: If the pre-defined filters do not return logs, it may be due to the filters not
having any logs to show. You can also create a wider custom filter. Navigate to
"Management page > Index Patterns > Create Index Pattern"For example, To
filter in all the logs, create the filter '*'.
Note: If logs are still not showing, this could be due to a small time range.
Increase the time range to potentially show more logs. For example, if the Kibana
time range is configured to monitor only the last 5 minutes, increase the time
range to obtain potentially more results.
DN09271553 1-0 © 2021 Nokia 112

R20SP1 and R20SP2
Expected 1. Successfully logged into the Kibana GUI portal.

Result 2. The hosts are shown as expected, the charts and graphs are visible, and the values are
as expected.
3. Project is created successfully.

4. VM is created successfully and is fully functional.
5. The VM is shown as expected, the charts and graphs are visible, and the values are as
expected.
DN09271553 1-0 © 2021 Nokia 113

R20SP1 and R20SP2
6. The logs are shown as expected.
Status OK | not OK
Comments
4.2.2.8 Log Auditing (Rsyslog)
Description When deploying CBIS, the user can configure target rsyslog servers. For post-CBIS
deployment, selected logs are sent via UDP port 514 from the CBIS Overcloud hosts to the
configured rsyslog servers.
DN09271553 1-0 © 2021 Nokia 114

R20SP1 and R20SP2
Test Case ID
Objective Verify that the rsyslog server receives log files from the CBIS Overcloud.
Estimated 0.5 Hour.

Duration
Supported All
From Version
Prerequisite/s 1. Rsyslog is enabled in the CBIS Manager before the CBIS Overcloud deployment.
Navigate as follows:
CBIS Manager > CBIS Installation > Overcloud > ELK Optional Parameters >
Rsyslog Servers > Type the Rsyslog server IP address > Click ADD
2. The UDP port 514 (syslog) is opened in the firewall, and the rsyslog server receives
traffic on port 514.
Test Execution Connect to the rsyslog server and check the logs sent from CBIS.
Example of some log files should be sent:
• /var/log/containers/neutron/server.log
• /var/log/container/nova/nova-compute.log
• /var/log/messages
Note: Within the folder /etc/filebeat/conf.d/, inside the Overcloud controllers,

there is a list of yml files.
Each of these files represent a different service.
Each file contains the paths of the log files which are sent to the rsyslog
server.
DN09271553 1-0 © 2021 Nokia 115

R20SP1 and R20SP2
Expected All the relevant logs from all the CBIS hosts are sent through UDP port 514 (default) to the
Result rsyslog server.
Even if the server is not configured with rsyslog, the UDP port 514 packets should still arrive
at their destination.
To check if the syslog packets reached the rsyslog server run from the rsyslog server the
command:
sudo tcpdump -nnepi <interface> port 514 -vvv
Note: The way that the logs are presented in the rsyslog server is determined by
the end user who initiated and configured the rsyslog server.
Status OK | not OK
Comments
4.2.2.9 User Action Auditing (Horizon)
Description By default, user actions made inside the OpenStack Horizon web portal are not audited.
When deploying CBIS the user can now enable the Horizon Audit Logging feature which
logs all of them user's actions.
Test Case ID
Objective To validate proper auditing of actions done in openstack horizon.
Estimated 0.5 hour.

Duration
Supported 18.0
From Version
Prerequisite/s • The setup is installed with the parameter Enable Horizon Audit Logging enabled.
DN09271553 1-0 © 2021 Nokia 116

R20SP1 and R20SP2
• In this test, the user will perform an action in the UI and will check the horizon log on the
controllers (only the controller that holds the internal API VIP is the one that will get the
logs). The Horizon log location is /var/log/containers/horizon/horizon.log
Note: This feature can be used along with rsyslog and ELK. But in the ATP,
this will be tested separately to focus on it. This is not a prerequisite in itself.
Test Execution 1. Login to the openstack dashboard.

2. Create a new project test_audit.
3. Create a new user named user_audit inside the project test_audit.
4. As user admin, create a new flavor.
5. As user admin, create a new network named audit-net.
6. As user admin, create a new subnet named audit-subnet inside audit-net network.
7. As user admin, create a new image.
8. As user admin, create a new VM.
9. Repeat steps 5-8 as user.
Expected 1. User logged in successfully.

Result 2. Project created successfully and the horizon log includes the action audit.
3. User is created successfully and the horizon log includes the action audit.
4. Flavor is created successfully and the horizon log includes the action audit.
5. Network is created successfully and the horizon log includes the action audit.
6. Subnet is created successfully and the horizon log includes the action audit.
7. Image is created successfully and the horizon log includes the action audit.
8. VM is created successfully and the horizon log includes the action audit.
9. Same results as steps 5-8.
Status OK | not OK
Comments
4.2.2.10 Instance Monitoring
Description Zabbix is now used for monitoring VMs instead of Ceilometer.
VM Level Monitoring is implemented in Zabbix as a template called Template VM Metrics

which is linked to compute hosts.
Zabbix constantly monitors the following statistics for all the VMs:
VM System Statistics
• Total memory
• Used memory
DN09271553 1-0 © 2021 Nokia 117

R20SP1 and R20SP2
• Free memory
• CPU time
• CPU user time
• CPU system time
• Number of CPUs
• Number of used CPUs
• CPU utilization
• Total disk space
• Number of adapters
• Uptime
• State
VM Disk Statistics
• Read bytes on disk

• Read bytes rate on disk
• Read requests rate on disk
• Write bytes on disk
• Write bytes rate on disk
• Write requests rate on disk
VM Network Statistics
• Incoming network bytes rate on interface

• Incoming network drops on interface
• Incoming network drops rate on interface
• Incoming network errors on interface
• Incoming network packets on interface
• Incoming network packets rate on interface
• Outgoing network bytes rate on interface
• Outgoing network drops on interface
• Outgoing network drops rate on interface
• Outgoing network errors on interface
• Outgoing network packets on interface
• Outgoing network packets rate on interface
Alarms
Apart from the constant VMs statistics monitoring, Zabbix is set to raise an alarm for the
following triggers:
• Low Available memory [10% - 30%] (Warning)

• Low Available memory [<= 10%] (High)
• Raised when CPU usage [70% - 90%] (Warning)
• Raised when CPU usage [>= 90%] (High)
DN09271553 1-0 © 2021 Nokia 118

R20SP1 and R20SP2
• Continuous incoming network drops on Interface (warning)

• Continuous outgoing network drops on Interface (warning)
Note: More information can be obtained from the CBIS Operation Manual
document under Monitoring > Zabbix Monitoring in CBIS > VM
Level Monitoring.
Test Case ID
Objective Verify that the Zabbix VM level monitoring is functioning.
Estimated 0.5 Hour.

Duration
Supported All
From Version
Test Execution 1. Create a flavor with 1 VCPU, 1024 MB RAM and 100G disk space.
2. Initiate a VM using the created flavor.
3. Login to Zabbix from CBIS Manager > EXTERNAL TOOLS > Zabbix or alternatively
navigate to https://<OpenStack_External_VIP>/zabbix
4. Inside Zabbix navigate to Monitoring > Latest data.
a. Type the created VM ID into the Name field.

b. Click Apply.
5. Login to the VM and consume 90%+ CPU usage. This can be done by any means
necessary.
Note: This can be achieved synthetically by running the following command

inside the VM:
yes > /dev/null &
6. After 60 seconds Zabbix is expected to trigger the alarm: High usage of CPU on VM:
<VM ID>
7. Stop the CPU consumption on the VM.
Note: If you used the command yes > /dev/null & remember to kill it!
( Using killall might also work).
8. After 60 seconds from the moment, the VM VCPU is below 70% the Zabbix alarm High
usage of CPU on VM:<VM ID> is expected to disappear.
Expected 1. Flavor is created successfully.

Result 2. VM is created successfully.
DN09271553 1-0 © 2021 Nokia 119

R20SP1 and R20SP2
3. Logged-in successfully to Zabbix portal.

4. All the monitoring items as mentioned in the test description should be visible, with their
values. Keep in mind that all the dynamic items (items with values that are subjected to
change at any given moment) are updated every 60 seconds.
5. The VM should reach 90%+ CPU utilization. It should stay like this till the Zabbix alarm
High usage of CPU on VM: <VM ID> is shown (estimated time for the alarm to trigger is
60 seconds).
6. The alarm High usage of CPU on VM: <VM ID> should show in Zabbix dashboard.
7. The VM CPU utilization is reduced to less than 70%.
DN09271553 1-0 © 2021 Nokia 120

R20SP1 and R20SP2
8. The alarm High usage of CPU on VM: <VM ID> disappeared from Zabbix dashboard
Status OK | not OK
Comments
4.2.3 CBIS Security
4.2.3.1 Customer Certificate
Description Installation with Customer certificate - via the CBIS Manager.
Test Case ID
Objective Verify that customer can provide the CA and Server certificates; specifically, to support the
TLS service.
Estimated 4 Hours.
Duration
Supported 20 SP2
From Version
Prerequisite/s
Test Execution 1. Follow the CBIS Manager manual.

2. In the overcloud tab enable “user provided TLS certificate and key” button
3. Place the certificates in a directory on the Undercloud Physical Server.
4. Complete the paths of the certificates:
DN09271553 1-0 © 2021 Nokia 121

R20SP1 and R20SP2
5. Deploy the setup
• Verify: All statuses are OK.

• Verify: VMs created OK
• Verify: Kibana/Novl/Zabbix/Horizon/Ceph dashboard are all responsive
• Verify: Kibana/Novl/Zabbix/Horizon/Ceph dashboards all have valid certificates
through the browser
Expected 1. All fields are configured and validation is green.

Result 2. “user provided TLS certificate and key” button enabled.
3. Certificates are placed in a directory on the Undercloud Physical Server.
4. Fields are configured and validation is green.
5. Deployment ends successfully.
• All statuses are OK

• VMs created OK
• Kibana/Novl/Zabbix/Horizon/Ceph dashboards are all responsive
• Kibana/Novl/Zabbix/Horizon/Ceph dashboards all have valid certificates through the
browser
Status OK | not OK
DN09271553 1-0 © 2021 Nokia 122

R20SP1 and R20SP2
Comments
4.2.3.2 CBIS TLS for External Services (Horizon/Keystone Zabbix, Kibana)
Description CBIS is automatically installed with TLS enabled. Connections between the client and server
are encrypted using TLS. Accessing any of the external web-interfaces such as Horizon,
Zabbix and Kibana is done only via HTTPS. Trying to access one of the web-interfaces via
HTTP will automatically redirect you to HTTPS.
Test Case ID
Objective Verify that CBIS TLS installed and that only HTTPS access is available to CBIS GUIs.
Estimated 0.5 Hour.

Duration
Supported 18.0
From Version
Test Execution 1. Access Horizon using HTTPS URL https://<$OpenStack_External_VIP>

2. Access Zabbix using HTTPS URL https://<$OpenStack_External_VIP>/
zabbix
3. Access Kibana using HTTPS URL https://<$OpenStack_External_VIP>/
kibana
4. Access Horizon using HTTP URL http://<$OpenStack_External_VIP>
5. Access Zabbix using HTTP URL http://<$OpenStack_External_VIP>/zabbix
6. Access Kibana using HTTP URL http://<$OpenStack_External_VIP>/kibana
Expected 1. Horizon can be accessed on https://<$OpenStack_External_VIP>/

Result 2. Zabbix can be accessed on https://<$OpenStack_External_VIP>/zabbix
3. Kibana can be accessed on https://<$OpenStack_External_VIP>/kibana
4. Horizon can be accessed after the web-page is redirected from HTTP to HTTPS
5. Zabbix can be accessed after the web-page is redirected from HTTP to HTTPS
6. Kibana can be accessed after the web-page is redirected from HTTP to HTTPS
Status OK | not OK
Comments
DN09271553 1-0 © 2021 Nokia 123

R20SP1 and R20SP2
4.2.3.3 Host and Operating System Hardening Check Mode
Description The CBIS platform offers the possibility of performing security hardening of the system as
a post-installation automated procedure. The post-installation hardening is done using the
Ansible framework.
The hardening includes multiple configurable parameters and defines the set of supported
security specifications (STIG, CIS, ANSSI and others), and their default value settings.
The hardening includes multiple domains and detailed below:
• RPM Hardening
• Password Hardening
• Boot Hardening
• Advanced Intrusion Detection Environment (AIDE) Configuration
• Kernel Hardening
• File/Directory Permissions
• Audit Rules
• SSH Hardening
• TLS Hardening
• Web Hardening
• NTP Configuration
• User Management
• OpenStack Hardening Configuration
• Miscellaneous
Note: In check mode, you can check every supported hardening task without
applying the changes on the system.
Navigate to the CBIS Manager > SECURITY > Security Hardening Check
Mode as shown here:
DN09271553 1-0 © 2021 Nokia 124

R20SP1 and R20SP2
Test Case ID
Objective Deploy the security hardening check mode and verify that the ssh ClientAliveInterval
can be changed on all hosts, Undercloud and the Undercloud Physical Server.
Estimated 20 minutes – 2 hours.

Duration
Supported CBIS 20
From Version
Prerequisite/s Successful Overcloud deployment.
Test Execution 1. Check the RHEL-07-040190 / CIS-5.2.13 - Terminate SSHsession after a period
of inactivity - Idle Timer security task, that it is not configured on the system yet, by
executing the following:
• For the Undercloud Physical Server, run:
cat /etc/ssh/sshd_config |grep ClientAliveInterval
• For the Undercloud, run:
• For all hosts, run the following from the Undercloud VM:
salt "*" cmd.run "cat /etc/ssh/sshd_config |grep

ClientAliveInterval"
2. Run the Security Hardening check-mode for RHEL-07-040190 / CIS-5.2.13 - Terminate

SSH session after a period of inactivity - Idle Timer security task, and navigate as
follows:
CBIS Manager > SECURITY > Security Hardening > In the Task selection –check
mode Security Tasks select Specific TAG(s) and write the tag as shown here:
Navigate to CBIS Manager > Security > Security Hardening Check Mode >
SSH hardening Check Mode and ensure that the RHEL-07-040190 / CIS-5.2.13 -
Terminate SSHsession after a period of inactivity - Idle Timer security task button is
enabled and set the interval to 60 (sec) as shown here:
DN09271553 1-0 © 2021 Nokia 125

R20SP1 and R20SP2
Click DEPLOY.
3. Check the RHEL-07-040190 / CIS-5.2.13 - Terminate SSHsession after a period of
inactivity - Idle Timer security task, that it is not configured on the system after running
the check-mode, by executing the following:

Expected 1. The result returned from the salt command and from the Undercloud and the Undercloud
Result Physical Server should be empty. (On condition that the security hardening was not run
before).
2. The Log window is displayed as follows:
DN09271553 1-0 © 2021 Nokia 126

R20SP1 and R20SP2
Note: See that the changed column contains progressive numbering as this
means that the value can be implemented when hardening will take place.
3. The result returned from the salt command and from the Undercloud and the
Undercloud Physical Server should be empty, (on condition that the security hardening
was not run before).
Status OK | not OK
Comments
4.2.3.4 Host and Operating System Hardening
Ansible framework.
The hardening includes multiple configurable parameters and defines the set of supported
security specifications (STIG, CIS, ANSSI and others), and their default value settings.
The hardening includes multiple domains and detailed below:
• RPM Hardening
• Boot Hardening
• Audit Rules
• SSH Hardening
• TLS Hardening
DN09271553 1-0 © 2021 Nokia 127

R20SP1 and R20SP2
• Web Hardening
• User Management
• Miscellaneous
Note: In check mode, you can check every supported hardening task without
applying the changes on the system.
Navigate to the CBIS Manager > SECURITY > Security Hardening as

shown here:
Test Case ID
Objective Deploy the security hardening and verify that the ssh ClientAliveInterval changed on
all hosts, Undercloud and the Undercloud Physical Server.

Duration
Supported CBIS 20
From Version
Test Execution 1. Check the RHEL-07-040190 / CIS-5.2.13 - Terminate SSHsession after a period
of inactivity - Idle Timer security task, that it is not configured on the system yet, by
executing the following:
DN09271553 1-0 © 2021 Nokia 128

R20SP1 and R20SP2

2. Run the Security Hardening check-mode for Disable the idle timer - keep the idle SSH
session active security task, and navigate as follows:
CBIS Manager > SECURITY > Security Hardening > In the Task selection –Security
Tasks select Specific TAG(s) and write the tag as shown here:
Navigate to CBIS Manager > Security > Security Hardening Check Mode >
SSH hardening Check Mode and ensure that the RHEL-07-040190 / CIS-5.2.13 -
Terminate SSHsession after a period of inactivity - Idle Timer security task button is
enabled and set the interval to 60 (sec) as shown here:
Click DEPLOY.
3. Check the Terminate SSHsession after a period of inactivity - Idle Timer security
task, that it is not configured on the system after running the check-mode, by executing
the following:
DN09271553 1-0 © 2021 Nokia 129

R20SP1 and R20SP2

4. ssh to the Undercloud Physical Server and from there ssh to the Undercloud VM. Wait
60 seconds and see that the session was terminated.
Expected 1. The result returned from the salt command and from the Undercloud and the Undercloud
Result Physical Server should be empty. (On condition that the security hardening was not run
before).
Undercloud Physical Server should be empty, (on condition that the security hardening
was not run before).
4. The session terminated towards the Undercloud after 60 seconds.
Status OK | not OK
DN09271553 1-0 © 2021 Nokia 130

R20SP1 and R20SP2
Comments
4.2.3.5 Host and Operating System Hardening Rollback
Ansible framework.
The Hardening Rollback supports for most of the hardened tasks in the system.(see security
hardening doc.)
Hardening Rollback includes multiple domains and detailed below:
• RPM Hardening
• Boot Hardening
• Audit Rules
• SSH Hardening
• TLS Hardening
• Web Hardening
• User Management
• Miscellaneous
Navigate to the CBIS Manager > SECURITY > Security Hardening as shown here:
Test Case ID
DN09271553 1-0 © 2021 Nokia 131

R20SP1 and R20SP2
Objective Deploy the security hardening rollback and verify that the ssh ClientAliveInterval was
disabled on all hosts, Undercloud and the Undercloud Physical Server.

Duration
Supported CBIS 20
From Version
Test Execution 1. Check the RHEL-07-040190 / CIS-5.2.13 - Disable the idle timer - keep the idle
SSH session active security task, that it is configured on the system, by executing the
following:

2. Run the Security Hardening check-mode for RHEL-07-040190 / CIS-5.2.13 - Disable

the idle timer - keep the idle SSH session active security task, and navigate as
follows:
CBIS Manager > Security > Security Hardening Rollback > In the Task selection –
Security Tasks select Specific TAG(s) and write the tag as shown here:
DN09271553 1-0 © 2021 Nokia 132

R20SP1 and R20SP2
Navigate to CBIS Manager > Security > Security Hardening Rollback > SSH
hardeningand ensure that the RHEL-07-040190 / CIS-5.2.13 – Disable the idle timer -
keep the idle SSH session active security task button is enabled as shown here:
Click DEPLOY.
3. Check the RHEL-07-040190 / CIS-5.2.13 – Disable the idle timer - keep the idle
SSH session active security task, that it is removed from the system after running the
security hardening Rollback, by executing the following:

4. ssh to the Undercloud Physical Server and from there ssh to the Undercloud VM. Wait
60 seconds and see that the session was terminated.
Expected 1. The result returned from the salt command and from the Undercloud and the
Result Undercloud Physical Server should be ClientAliveInterval 60.
DN09271553 1-0 © 2021 Nokia 133

R20SP1 and R20SP2
Undercloud Physical Server should be empty.
4. The session terminated towards the Undercloud after 60 seconds.
Status OK | not OK
Comments
4.2.3.6 Multi-Tenancy (OpenStack Tenants)
Description Networks which are associated with different Tenants have no connectivity between them.
Test Case ID
Objective Verify that different tenants can use the same CIDR values but these tenants have no
network connectivity between their VMs.
Estimated 1 Hour.
Duration
Supported 19A MP3.

From Version
Test Execution 1. Create two networks where each network is associated with a different tenant but
includes the same subnet (same CIDR).
2. Create a security group which allows incoming/outgoing ICMP packets.
DN09271553 1-0 © 2021 Nokia 134

R20SP1 and R20SP2
3. Create a VM for each tenant using its designated network and the previously created
"Allow ICMP security group".
4. Ping from one VM to the other’s network address.
Expected 1. The networks are created.

Result 2. The security group is created.
3. The VMs are created with network addresses in the same subnet.
4. There is no ping response as expected.
Status OK | not OK
Comments
4.2.3.7 External LDAPS server
Description Supports authentication through the external LDAPS server.
Test Case ID
Objective Verify that different users in different roles can be authenticated through an external LDAPS
server.
Estimated 2 Hours.
Duration
Supported 20 SP2
From Version
Prerequisite 1. An External LDAPS server is installed and configured.

2. LDAP Configuration and management Client, for example: LDAPAdmin.
Test Execution 1. On the LDAPS server:
a. Configure users under a group, specific for the OpenStack users. (use LDAPAdmin
for that).
DN09271553 1-0 © 2021 Nokia 135

R20SP1 and R20SP2
b. Also assign a password for those users.

c. Using the CBIS Manager, deploy the LDAP service, by selecting the "External
LDAP Deployment" option.
d. Users are created and should be visible in the LDAP management interface.
2. Inspect the /var/log/cbis/ansible/ansible.log
The Ansible playbook completed successfully and LDAP was set correctly..
check on all controllers that configuration was done successfully:
sudo -i
cd /etc/keystone/domains/
cat keystone.LDAP.conf
3. Ensure that the default domain and LDAP domain are present, as follows:
. overcloudrc
openstack domain list
4. Ensure that users under the default domain are displayed, as follows:
openstack user list
DN09271553 1-0 © 2021 Nokia 136

R20SP1 and R20SP2
5. Ensure that users that have been configured on the LDAP Server are displayed, as
follows:
. overcloudrc
openstack user list --domain ldap
6. On the Hypervisor CLI as overcloudrc, issue these commands:
a. To verify the domain list, run:
openstack domain list
b. Check the OpenStack user list using this CLI command:
openstack user list
c. Check the LDAP user list using this CLI command:
openstack user list --domain ldap
d. Create a project using the following CLI command:
openstack project create --domain LDAP --enable tenant1
7. Ensure that the project is presented as part of the LDAP domain as follows:
openstack project list --domain ldap
DN09271553 1-0 © 2021 Nokia 137

R20SP1 and R20SP2
8. Using the UUID obtained earlier, (see steps 4 and 5), assign one of the LDAP users as
an admin role for the project just created:
openstack role add --project tenant1 --user

2bf5685c1a844c68bd07a876f8070268 admin
9. Login to the Horizon interface with the admin LDAP User/specifying the LDAP domain
and password configured under LDAP.
10. Create with the user resources such as network, subnet, volume and create VMs.
11. On the OpenStack CLI assign one of the LDAP users as a member role for the project
just created:
openstack role add --project tenant1 --user de345bc3cb76cfa3b9e769238b6a70d _

member_
12. Login to the Horizon interface with a member LDAP User/specifying the LDAP domain
and password configured under LDAP.
DN09271553 1-0 © 2021 Nokia 138

R20SP1 and R20SP2
13. Create with the user, resources such as network, subnet, volume and create VMs.
14. Login to Horizon with the admin user defined under the default domain.
15. Create VMs with user resources such as network, subnet and volume.
Expected 1. The project is displayed as part of the LDAP domain.

Result 2. Ensure that the operation was configured successfully.
3. Ensure that the default domain and LDAP domain are present.
4. Ensure that users under the default domain are displayed.
5. Ensure that users that have been configured on the LDAP Server are displayed.
6. Ensure that the project is presented as part of the LDAP domain.
7. Ensure that the admin user can login with appropriate authorization.
8. Resources and VMs created successfully.
9. Logon to the Horizon interface with the admin LDAP User/specifying the LDAP domain
and password configured under LDAP was successful.
10. VMs created with user resources such as network, subnet, volume.
11. Ensure that the member user can login with appropriate authorization.
12. Logon to the Horizon interface was successful.
14. Ensure that the admin user on default domain can login with appropriate authorization.
Status OK | not OK
Comments
DN09271553 1-0 © 2021 Nokia 139

R20SP1 and R20SP2
4.2.4 CBIS Operations
4.2.4.1 Undercloud VM Backup
For a detailed description of this procedure, see the CBIS Operations Manual, DN09259995.
Description Perform Undercloud backup using the CBIS Manager.
Test Case ID
Objective Verify that a complete snapshot of the Undercloud VM is successfully backed up.
Estimated 10 min.
Duration
Supported CBIS 19.0MP2

From Version
Prerequisite/s 1. The CBIS Cluster has been installed successfully.

2. The backup file is saved in the Undercloud Physical Server. The user needs to ensure
that the Undercloud Physical Server has enough disk space available for the backup
file. The backup file weights around 55GB when the database is considerably small.
Therefore, the suggestion is to have at least 200-300GB available disk space.
Test Execution 1. Follow the "Undercloud Backup" procedure in the CBIS Manager manual.
2. After a successful Undercloud VM backup procedure, check that the backup file exists in
the backup directory. The default backup directory (if it has not been manually changed)
should be:/root/cbis-backup in the Undercloud Physical Server.
Note:
• The operator should store the backup files in an external repository to avoid a
situation where the Undercloud Physical Server is damaged and the backup
files which reside in the Undercloud Physical Server are lost.
• The Undercloud should be backed up manually each time a new node is
added or removed (compute, storage), or if a controller node is replaced.
Expected 1. The procedure was completed successfully.

Result
DN09271553 1-0 © 2021 Nokia 140

R20SP1 and R20SP2
2. The Undercloud VM backup file is saved under the backup directory.
Status OK | not OK
Comments
4.2.4.2 Undercloud VM Restore
For a detailed description of this procedure, see the CBIS Operations Manual, DN09259995.
Description If the Undercloud hardware fails, new hardware of the same type will replace it, and the
Undercloud VM will be restored from the backup snapshot. Perform an Undercloud VM
Restore using the CBIS Manager.
Test Case ID
Objective Verify that a complete snapshot of the Undercloud VM is successfully restored.
Estimated 30 min.
Duration
Supported CBIS 19.0

From Version
Prerequisite/s An appropriate Undercloud VM backup file is available and resides in the Undercloud
Physical Server.
Test Execution 1. Select one compute from the CBIS Cluster computes. This compute will be erased
purposely and then restored as part of the Undercloud VM restore procedure.
source ~/stackrc && openstack server list
2. SSH to that compute.
DN09271553 1-0 © 2021 Nokia 141

R20SP1 and R20SP2
Note: If, for any reason you are unable to SSH to the selected compute,
select a different compute to which you can SSH.
3. Before starting the Undercloud VM restore, the Undercloud VM database needs to

be changed. After the Undercloud VM restore, a check is required to see if the same
database that was changed is now reverted to its original state.
From the Undercloud VM, as the stackrc user, delete the nova compute that was
selected in the previous step as follows:
source ~/stackrc && openstack server delete <selected compute

ID or hostname>
Example
source ~/stackrc && openstack server delete overcloud-

dpdkperformancecompute-0
4. Validate that the nova compute server is deleted and does not exist in the nova servers
list:
5. Follow the "Undercloud Restore" procedure as shown in the CBIS Manager manual.
Note: You need to enter the Backup directory where the backup file resides
for the backup(s) to show in the "Backup file to restore the "drop-down list".
6. Connect with SSH to the Undercloud VM.

7. Source to the internal admin project (stackrc):
source ~/stackrc
8. Display all baremetal(ironic) nodes:
openstack baremetal node list
9. Display all nova instances:
openstack server list
10. Display the Overcloud heat stack:
openstack stack list
11. Connect with SSH to the compute that was eradicated before the Undercloud VM
restore.
12. Create a VM on the restored compute.
DN09271553 1-0 © 2021 Nokia 142

R20SP1 and R20SP2
Expected 1. Selected compute for deletion.

Result 2. Connect with SSH to the selected compute.
3. The nova server deletion is successful.
4. The deleted nova server is gone from the nova servers list.
5. The Undercloud VM is restored successfully, as shown here:
6. The SSH login was successful.

7. Successfully sourced the stackrc admin project.
8. All physical servers are displayed with their proper status.
9. All servers are displayed including the server that was deleted before the restore.
10. Overcloud stack is displayed as either CREATE_COMPLETE or UPDATE_COMPLETE.
11. Connection with SSH was performed to the selected compute.
12. The VM is successfully created and is accessible.
Status OK | not OK
Comments
4.2.4.3 Overcloud Database Backup
Description The Overcloud is backed up, once a day at 2:00 AM, (by default or as user defined in
installation) using the CBIS backup solution. The Overcloud backup is a complete backup of
the database taken from all the controllers.
Backups are stored on the Undercloud VM at /mnt/backup which is mapped to the NFS
mountpoint. This is configured in the CBIS Manager under CBIS Installation > Overcloud
> Backup Optional Configuration.
Note: The default NFS mountpoint is the setup Undercloud Physical Server /
root/backup directory.
DN09271553 1-0 © 2021 Nokia 143

R20SP1 and R20SP2
The backup files are encrypted for security purposes.
Test Case ID
Objective Validate that the Overcloud database was backed up.
Duration
Supported CBIS R17.0

From Version
Prerequisite/s As the Overcloud database backup is performed automatically, once a day at 2:00 AM, it is
required to wait at least 1 backup iteration before you are able to have backup files.
Test Execution 1. Wait until you have at least 1 backup file under /mnt/backup in the Undercloud VM.
This requires waiting 1 backup iteration which takes place at 2:00 AM each day.
2. Check that the backup files exist in the Undercloud VM /mnt/backup directory.
a. The backup file is taken simultaneously from each controller, separately for HA
purposes. Therefore, it is expected that a directory for each Overcloud controller will
be seen:
[stack@undercloud (stackrc) backup]$ ls -l

drwxr-xr-x. 17 stack stack 4096 Jul 18 02:00 overcloud-
controller-0
controller-1
controller-2
b. Inside each Overcloud controller directory there are more directories named by the
date and time that the backup was taken:
[stack@undercloud (stackrc) overcloud-controller-0]$ ls -l

drwxr-xr-x. 2 stack stack 27 Jul 1 02:00 2019.07.01.02.00.
01
01
02
c. In addition, inside each of the directories represented by date and time the encrypted
db_backup.enc backup file should appear:
[stack@undercloud (stackrc) 2019.07.01.02.00.01]$ ls -l

-rw-r--r--. 1 stack stack 4047584 Jul 1 02:00 db_backup.enc
3. Validate /mnt/backup on the Undercloud VM is mapped to the NFS mountpoint.
DN09271553 1-0 © 2021 Nokia 144

R20SP1 and R20SP2
df | grep /mnt/backup
4. Check that the backup files reside in the NFS mountpoint.
Note: If using an external NFS server (not the default Undercloud Physical
Server) make sure to allow access from the Undercloud VM to the NFS server
(ports, firewall and etc..).
Expected 1. The setup was up and running from 2:00 AM to at least 3:00 AM, (According to the
Result default backup time, or 1 hour after the manually defined backup time).
2. Assuming the setup was working correctly at 2:00 AM, it is expected that there will be
backup files under /mnt/backup in the Undercloud VM.
3. The /mnt/backup directory is mapped to the NFS mountpoint as configured in the
CBIS Manager.
4. The backup files should reside in the NFS mountpoint.
Status OK | not OK
Comments
4.2.4.4 Overcloud Database Restore
Description Check the Overcloud Database Restore
Test Case ID
Objective Verify that the the Overcloud database restore is performed correctly.
Estimated ~ 2 hours. The procedure duration depends on the size of setup and the amount of data.
Duration
Supported CBIS 20
From Version
Prerequisite/s A valid backup: it was taken on the same CBIS cluster. A backup file cannot be used on
new CBIS deployments.
Test Execution 1. Open CBIS Manager > CBIS OPERATIONS and verify that the Overcloud database
restore option is displayed.
2. Enter the Backup Folder Location. The location of the folder containing the backup file
(db_backup.enc).
Configured in user_config.yaml under backup_nfs_mountpoint (default is /root/

backup/).
DN09271553 1-0 © 2021 Nokia 145

R20SP1 and R20SP2
Backup folder example:
/root/backup/overcloud-controller-0/2018.12.05.04.00.02
3. On the setup, create a number of VMs (the user can choose from 5 to 10). Create at
least 1 volume and attach it to 1 VM. Mount the volume and create a large file on it.
Now, get the md5sum from the file. Check that all VMs are up and running.
4. Make sure you have the relevant backup file in the Hypervisor under /root/backup
and in the Undercloud VM under /mnt/backup.
Note: The files are created each night automatically in a cronjob.
Note: It should have a working backup with a working database.
5. Stop the DB.
One of the examples of stopping the DB is:
1. Log in to the controller-0.

2. Run the following commands:
sudo -i
mysql
DROP database nova
6. Perform the Overcloud database restore procedure:
• At the CBIS Manager, enter the operation section, click on the Overcloud
database restore tab, fill the Backup Folder Location field, enter the Backup
Password as set in the user_config.yaml.
The Overcloud DB should be restored to the previous working version without the
issue.
7. Verify all created VMs on step 1 are also restored and are up and running.
8. Verify that the large volume is attached and undamaged, comparing the md5sum with
the md5sum from step 1.
All VMs created on step 1 are restored and up and running.

9. Run sanity and several tests, such as create VMs etc., to validate that everything is
working.
10. Verify Login to Zabbix. Check that Zabbix shows NO unexpected alarms.
Expected 1. CBIS Manager/Operation was opened. The Overcloud database restore block was
Result presented. The CBIS Manager > CBIS OPERATIONS was reached and the Overcloud
database restore option is displayed.
DN09271553 1-0 © 2021 Nokia 146

R20SP1 and R20SP2
2. The Backup Folder Location was entered. The location of the folder containing the
backup file was (db_backup.enc).
3. Running VMs on the setup were created. The volume was attached to one of the VMs. A
large file was created successfully. All VMs were up and running.
4. The relevant backup file was in the Hypervisor under /root/backup and in the
Undercloud VM under /mnt/backup.
5. The DB was ended.
6. The Overcloud database restore procedure was performed.
7. All created VMs on step 1 were also restored and were up and running.
8. The large volume was attached and unhurt. All VM created on step 1 were restore and
up and running.
9. Sanity and several tests, such as create VMs and so on, were performed. Everything
was working.
10. Zabbix had NO unexpected alarms.
Status OK | not OK
Comments
4.2.4.5 Scale In Compute (HCI or none-HCI compute)
Description Remove a compute host from the CBIS Cluster.
Test Case ID
Objective Verify that the compute has been successfully removed from the Overcloud.
Estimated 0.5 Hours for a non-HCI compute.

Duration
Note: It is expected that the estimated duration of an HCI compute scale in will
take longer than a none-HCI compute due to Ceph rebalancing kicking-in when
scaling-in an HCI compute. The duration depends on the number of objects and
the amount of data within the Ceph cluster which corresponds to the time it takes
for Ceph to finish rebalancing. These variables make it harder to determine an
accurate duration for the scale in process. As an example, it may take few more
hours if the Ceph cluster disk space is 50% allocated.
Supported All
From Version
Prerequisite/s 1. CBIS Cluster has been installed successfully.

2. There is at least 1 more HCI computes than the configured Ceph replication (only
relevant for HCI system). This is to avoid the corruption of Ceph during compute scale-
in.
DN09271553 1-0 © 2021 Nokia 147

R20SP1 and R20SP2
Test Execution 1. If the CBIS Cluster is an HCI (Hyper-converged infrastructure) system (meaning
the Ceph OSDs reside on the computes) check the Ceph status before the scale in
operation. From one of the controllers, execute:
sudo ceph -s
Note: It is not advisable to start the compute scale-in operation if Ceph status
isn't HEALTH_OK. If the Ceph status returns HEALTH WARNING make sure
to acknowledge the warning. If the warning is acknowledged and accepted
continue to the scale-in operation. Otherwise, fix the Ceph issues and then
continue with the scale-in operation. If Ceph status returns HEALTH ERROR,
it is required to fix the Ceph error and only then continue with the scale-in
operation.
2. Follow the "Scale in Compute/Storage" procedure in the CBIS Manager manual.

3. Check that the scaled-in compute does not exist in the baremetal (ironic) database.
source ~/stackrc && openstack baremetal node list
4. Check that the scaled in compute does not exist in nova servers list.
5. If the scaled in host is an HCI compute, check that all the OSDs are removed as
expected and that the Ceph status is HEALTH_OK or, Ceph status is as it was before
the scale-in operation. From one of the controllers, execute:
sudo ceph -s
sudo ceph osd tree
6. Check that the compute was removed from Zabbix.
a. Login to Zabbix web-interface via https://[OpenStack_External_VIP]/zabbix

b. Within Zabbix, navigate to Configuration > Hosts and check that the removed
compute has been removed from the list of hosts.
Expected 1. The Ceph status is acknowledged and accepted before starting the scale in procedure.
Result 2. The scale in operation finished successfully.
3. The compute has been removed from baremetal (ironic).
4. The compute has been removed from nova servers list.
5. Ceph health is either HEALTH_OK or as it was before the scale out operation. The
compute and its OSDs are removed from the sudo ceph osd tree table.
DN09271553 1-0 © 2021 Nokia 148

R20SP1 and R20SP2
Note: Step 5 above is only relevant if the CBIS Cluster is an HCI (Hyper-
converged infrastructure) system ( that is the Ceph OSDs reside on the
computes).
6. The compute is no longer in Zabbix.
Status OK | not OK
Comments
4.2.4.6 Scale Out Compute (HCI or none-HCI compute)
Description Add a compute host from the CBIS Cluster.
Test Case ID
Objective Verify that a compute has been successfully added to the Overcloud.
Estimated 3-4 hours.

Duration
Supported All
From Version
Prerequisite/s • An available unused compute for the scale out operation, (you will need its IPMI
address). If there is none, you need to scale in a compute before the scale out
operation.
• CBIS allows the user to automatically execute the security hardening on the scaled-
out compute by enabling the "Run Security Hardening Post Scale out" radio button
in the scale-out operation page in CBIS Manager. This is only relevant if the security
hardening is applied to the CBIS Cluster before the scale-out operation.
DN09271553 1-0 © 2021 Nokia 149

R20SP1 and R20SP2
Test Execution 1. If the CBIS Cluster is an HCI (Hyper-converged infrastructure) system (meaning the
Ceph OSDs reside on the computes) check the Ceph status before the scale-out
sudo ceph -s
Note: It is not advisable to start the compute scale-out operation if the Ceph
status is not displaying HEALTH_OK. If the Ceph status returns HEALTH
WARNING make sure to acknowledge the warning. If the warning is
acknowledged and accepted, continue to the scale-out operation. Otherwise,
fix the Ceph issues and then continue with the scale-out operation. If the Ceph
status returns HEALTH ERROR, it is required to fix the Ceph error and only
then continue with the scale-out operation.
2. Follow the "Scale-Out Compute/Storage/Monitoring" procedure in the CBIS Manager

manual.
3. Check that the scaled-out compute exists in the baremetal(ironic) database. From the
Undercloud VM, execute:
4. Check that the scaled-out compute exists in the nova servers list.
5. Connect with SSH to the scaled out compute.

6. If the scaled out host is an HCI compute, check that all the OSDs are added as expected
and that the Ceph status is HEALTH_OK or, Ceph status is as it was before the scale-
out operation. Otherwise skip to the next step. From one of the controllers, execute:
sudo ceph -s
sudo ceph osd tree
7. Initiate a VM on the scaled-out compute using a permissive security group (only relevant
to OVS/DPDK VMs.
Expected 1. The Ceph status is acknowledged and accepted before starting the scale-out procedure.
Result 2. The scale out operation finished successfully.
3. The compute is shown in baremetal(ironic).
4. The compute is shown in nova servers list.
5. Connection made with SSH to the scaled out compute.
6. Ceph health is either HEALTH_OK or as it was before the scale-out operation. The
number of presented OSDs as shown in sudo ceph osd tree correspond to the
number of physical disks of the added compute.
DN09271553 1-0 © 2021 Nokia 150

R20SP1 and R20SP2
Note: Step 6 above is only relevant if the CBIS Cluster is an HCI (Hyper-
converged infrastructure) system (meaning the Ceph OSDs reside on the
computes).
7. A VM was successfully initiated on the scaled-out compute and is accessible.
Status OK | not OK
Comments
4.2.4.7 Scale In Storage Node
Description Perform Scale In of a storage node through the CBIS Manager.
Test Case ID
Objective Verify that a storage node is successfully removed from the Overcloud.
Estimated ~1 hour
Duration
Note: The duration of scaling in storage node depends on the number of objects
and the amount of data within the Ceph cluster which corresponds with the time it
takes for Ceph to finish rebalancing. These variables make it harder to determine
an accurate estimated duration for the scale in process. As an example, it may
take several hours long if the Ceph cluster is 50% allocated.
Supported 18.0
From Version
Prerequisite/s 1. The CBIS Cluster is installed successfully.

2. The number of storage nodes must be greater than the configured Ceph replication
number. If not, and a storage node is scaled in, Ceph will enter ERROR state.
Test Execution 1. Check the Ceph status before the scale-in operation. From one of the controllers
execute:
sudo ceph -s
Note: It is not advisable to start the storage node scale-in operation if Ceph
status is not HEALTH_OK. If the Ceph status returns HEALTH_WARNING
make sure to acknowledge the warning. If the warning is acknowledged and
accepted as a valid warning, continue with the scale-in operation. Otherwise,
fix the warning root cause and then, continue with the scale-in operation. If
Ceph status returns HEALTH_ERROR, it is mandatory to fix the Ceph error and
only then continue with the scale-in operation.
2. Follow the "Scale in Compute/Storage" procedure in the CBIS Manager manual.
DN09271553 1-0 © 2021 Nokia 151

R20SP1 and R20SP2
3. Check that the scaled-in storage node does not exist in the baremertal (ironic) database.
4. Check that the scaled-in storage node does not exist in nova servers list. From the
Undercloud VM, execute:
5. Check that all the OSDs are removed as expected and that the Ceph status returns
HEALTH_OK, or that the Ceph status is as it was before the scale-in operation. From one
of the controllers, execute:
sudo ceph -s
sudo ceph osd tree
6. Check that the storage node was removed from Zabbix.
a. Login to Zabbix web interface via https://<OpenStack _External_VIP>/

zabbix
b. Within Zabbix navigate to Configuration > Hosts and check the removed storage
node is gone from the list of hosts.
Expected 1. The Ceph status is acknowledged and accepted before starting the scale-in procedure.
Result 2. The scale in operation finished successfully.
3. The storage node has been removed from the baremetal (ironic).
4. The storage node has been removed from nova servers list.
5. The Ceph health is either HEALTH_OK or as it was before the scale-out operation. The
storage node and its OSDs are removed from the sudo ceph osd tree table.
6. The storage node has been removed from Zabbix.
Status OK | not OK
Comments
4.2.4.8 Scale Out Storage Node
Description With CBIS, you can add a storage node to an existing working CBIS system via the CBIS
Manager. By adding more storage nodes, the capacity of the Ceph cluster is increased as
well as the maximum IOPS.
Test Case ID
Objective Verify that a storage node has been successfully added to the Overcloud.
Estimated 3-4 hours.

Duration
DN09271553 1-0 © 2021 Nokia 152

R20SP1 and R20SP2
Supported CBIS 19.0MP2

From Version
Prerequisite/s 1. The CBIS Cluster has been installed successfully.

2. There is an available unused storage node for the scale out operation. If there is none, it
is required to scale-in a storage before the scale out operation.
3. CBIS allows the user to automatically execute the security hardening on the scaled out
storage node by enabling the "Run Security Hardening Post Scale out" radio button in
the scale-out operation page in CBIS Manager.
Note: Enabling "Run Security Hardening Post Scale out" is relevant if the
security hardening was applied on the CBIS Cluster before the scale-out
operation.
Test Execution 1. Check the Ceph status before the scale out operation.
• From one of the controllers, execute:
sudo ceph -s
Note: It is not advisable to start the storage node scale-out operation if

the Ceph status is not showing HEALTH_OK. If the Ceph status returns
a HEALTH WARNING, ensure that the warning is acknowledged. If
the warning is acknowledged and accepted, continue to the scale-out
operation. Otherwise, fix the issue and then continue with the scale-out
operation. If the Ceph status returns a HEALTH ERROR, it is mandatory
to fix the Ceph error before starting the scale-out operation.
2. Follow the "Scale out Compute/Storage/Monitoring" procedure in the CBIS Manager

manual.
DN09271553 1-0 © 2021 Nokia 153

R20SP1 and R20SP2
3. Check that the scaled-out storage node exists in the baremetal (ironic) database. From
the Undercloud VM, execute:
4. Check that the scaled-out storage node exists in nova servers list. From the Undercloud
VM, execute:
5. Connect with SSH to the scaled out storage node.

6. Check that all the OSDs are added as expected and that the Ceph status reports
HEALTH_OK or that the Ceph status is as it was before the scale-out operation. From
one of the controllers, execute:
sudo ceph -s
sudo ceph osd tree
Expected 1. The Ceph status is acknowledged and accepted before starting the scale-out procedure.
Result 2. The scale-out operation has finished successfully.
3. The storage node is shown in baremetal (ironic).
4. The storage node is shown in nova servers list.
5. Connected with SSH to the scaled out storage node.
6. The Ceph health is either HEALTH_OK or as it was before the scale out operation. The
number of presented OSDs is as shown in sudo ceph osd tree corresponds to the
number of physical disks of the added storage node.
Status OK | not OK
Comments
4.2.4.9 Set to Maintenance Compute
Description A compute node will need to be set to maintenance (gracefully shutdown) to allow
the replacing of faulty HW components such as disks, NICs, memory or any kind of
maintenance required to be performed on a single node at a time to allow the system High
Availability features to retain normal operation.
The CBIS Manager Maintenance Mode Compute, Storage and Monitoring feature enables
you to Set/unset compute, storage and monitoring nodes to/from maintenance mode.
Note: As of CBIS R20, this feature enables you to set to maintenance several
non-HCI computes at once (as this does not affect storage redundancy), and/or a
single storage / HCI compute at a time.
DN09271553 1-0 © 2021 Nokia 154

R20SP1 and R20SP2
Available options, as well as suggested default parameters settings (such as Replicate

Ceph data and Ignore Ceph errors) are automatically set, but can be manually overridden.
As Replicate Ceph data may take a long time with large storage, when you have planned
larger activities involving several storage nodes one after the other with enough storage
redundancy, it is advisable to execute storage nodes set/unset to maintenance without
Replicate Ceph data option, and allow this option only during unset from maintenance of last
storage node. This will enable replication by the end of the procedure and the return to a
normal cluster storage state.
Remember: To run Storage/HCI - Last unset from maintenance with Replicate

Ceph data option selected.
Test Case ID
Objective Verify that a compute node can be set to maintenance (gracefully shutdown) and can be
powered up, easily and with minimal effect on the rest of the CBIS cluster.
Estimated 2 hours ~
Duration
Supported V19A (With some sub features available as of V20)

From Version
Prerequisite/s 1. The CBIS cluster is installed successfully with 3 controllers.

2. The CBIS Manager manual is available.
3. Any existing VM on the intended compute should be migrated to other computes
according to related VNF rules (and if needed returned at the end of the maintenance
procedure).
Test Execution 1. If the required node is an HCI (Hyper-converged infrastructure) system (meaning
sudo ceph -s
Note: It is not advisable to start the set to maintenance compute operation

if Ceph status is not HEALTH_OK. If the Ceph status returns HEALTH
WARNING make sure to acknowledge the warning. If the warning is
acknowledged and accepted, then continue to the set to maintenance
compute operation. Otherwise, fix the Ceph issues and then continue with the
set to maintenance compute operation. If the Ceph status returns HEALTH
ERROR, you need to fix the Ceph error and only then continue with the set to
maintenance compute operation.
DN09271553 1-0 © 2021 Nokia 155

R20SP1 and R20SP2
2. Check that all the CBIS web portals such as Kibana, Zabbix, Horizon, Vitrage (inside
Horizon up to V19A) and Ceph manager dashboard are accessible and viable. Access
the web portals via CBIS Manager > External Tools.
3. Check the Zabbix Dashboard for any existing failures which should better be fixed
before starting or record the status in order to be able to verify status of the cluster
returned to same condition after unset from maintenance. You can also track the events
reported during the procedure using Zabbix – Monitoring – Problems – History – Last 2
days.
4. Invoke Maintenance Mode Compute – Set to maintenance, select the required
compute node, and if needed, change the default selected parameters and click
DEPLOY. Track the execution log and verify that it ends successfully.
5. Verify that the compute was set to maintenance and shutdown using an openstack
baremetal node list and check the Zabbix and Ceph Dashboards for any existing
failures. Now you can perform any required HW maintenance.
6. Run the Maintenance Mode Compute – Unset from maintenance, select the required
compute node, and if needed, change the default selected parameters and click
DEPLOY. Track execution log, verify it ends successfully.
7. Verify that the compute was unset from maintenance and shutdown using openstack
baremetal node list and check the Zabbix and Ceph Dashboards for any existing
failures compared to the previous state before the start of this test.
8. You may now create new VM and/or migrate back any previously removed VMs.
9. Check that the VMs are up and running and reply to the ICMP packets (ping).
10. Once again, access the CBIS web portals such as Kibana, Zabbix, Horizon, Vitrage
(inside Horizon up to V19A) and the Ceph manager dashboard and check that they are
accessible and viable as was before the Set to maintenance procedure.
Expected 1. Ceph status is OK or acknowledged.

Result 2. All the web portals should be accessible and functional.
3. No unexpected Zabbix alarms.
4. Set to maintenance activity completed successfully.
5. The compute node is powered OFF and marked in maintenance.
6. Unset from maintenance activity completed successfully.
7. The compute node is powered ON and not marked in maintenance. No unexpected
alarms in Zabbix, Ceph and other Dashboards.
8. VMs have been created or migrated back.
9. The VMs are successfully initiated and reply to the ICMP requests.
10. All the web portals should be accessible and functional as before the procedure.
Status OK | not OK
Comments
DN09271553 1-0 © 2021 Nokia 156

R20SP1 and R20SP2
4.2.4.10 Reboot Server, Host Group
Description The Reboot Servers operation in CBIS Manager enables you to gracefully reboot specific
compute, storage, monitoring and controller node/s, as well as reboot all nodes belonging
to an aggregate, node type or entire cluster, or any user selected combination out of the
above.
Reboot activity is performed one node at a time out of selected list according to a suggested
order to reduce traffic and storage failure starting from controllers, storage, HCI and then all
non-HCI computes together.
Reboot is used as part of the upgrade procedure and in any other case where rebooting
nodes is needed.
Test Case ID
Objective Verify that the cluster nodes can be gracefully rebooted, easily and with minimal effect on
the rest of the CBIS cluster.
Estimated 2 hours ~.
Duration
Supported CBIS 19A

From Version
Prerequisite/s 1. The CBIS cluster is installed successfully with 3 controllers.

2. The CBIS Manager manual is available.
3. Any existing VM on the intended compute/s should be shut off or migrated to other
computes according to related VNF rules (and if needed returned at the end of the
reboot procedure).
Test Execution 1. If the desired node is an HCI (Hyper-converged infrastructure) system (meaning
sudo ceph -s
Note: It is not advisable to start the reboot server operation if the Ceph
status is not HEALTH_OK. If the Ceph status returns HEALTH WARNING,
make sure to acknowledge the warning. If the warning is acknowledged
and accepted, continue to the reboot server operation. Otherwise, fix the
Ceph issues and then continue with the reboot server operation. If the Ceph
status returns HEALTH ERROR, you need to fix the Ceph error and only then
continue with the reboot server operation.
DN09271553 1-0 © 2021 Nokia 157

R20SP1 and R20SP2
2. Check that all the CBIS web portals such as Kibana, Zabbix, Horizon, Vitrage (inside
Horizon up to V19A) and Ceph manager dashboard are accessible and viable. Access
the web portals via CBIS Manager > External Tools.
3. Check Zabbix Dashboard for any existing failures which should better be fixed before
starting or record the status in order to be able to verify status of the cluster returned to
same condition after reboot server. You can also track the events reported during the
procedure using Zabbix – Monitoring – Problems – History – Last 2 days.
4. Invoke Reboot Servers, select desired server node/s, aggregates etc. combination, if
needed change default selected parameters and click DEPLOY. Track execution log,
verify it ends successfully.
5. Verify all selected computes were rebooted Check Zabbix and Ceph Dashboards for any
existing failures – compare to previous state before start of this test.
6. You may now create new VM and/or migrate back any previously removed VMs.
7. Check that the VMs are up and running and reply to the ICMP packets (ping).
(inside Horizon up to V19A) and the Ceph manager dashboard and check that they are
accessible and viable as was before the Set to maintenance procedure.
9. Optionally repeat for other types of servers or groups.
Expected 1. Ceph status is OK or acknowledged.

Result 2. All the web portals should be accessible and functional.
3. No unexpected Zabbix alarms.
4. Check that the Reboot Servers operation completed successfully.
5. The servers and computes were rebooted and returned to a normal condition.
6. New VMs were created or migrated back from any previously removed VMs.
8. No unexpected alarms in Zabbix, Ceph and other Dashboards. All web portals are
accessible and viable.
9. Results were repated successfully for other types of servers or groups.
Status OK | not OK
Comments
4.2.4.11 Replace Controller
Description Check that the Replace Controller is enabled in CBIS Manager.
Test Case ID
Objective Verify that the Replace Controller action is performed via the CBIS Manager.
Estimated ~ 4 hours.
Duration
DN09271553 1-0 © 2021 Nokia 158

R20SP1 and R20SP2
Supported CBIS 20.

From Version
Prerequisites The setup should be installed and one server should be left out to be wiped as a candidate
for the new controller.
Test Execution 1. After the installation is complete, deploy a VM on each of the hosts.
2. Verify ping (cross compute) different VMs.
3. Verify on one of the controllers that the Ceph health is OK and that the PCS status is
OK.
4. Perform Replace Controller using CBIS Manager under the CBIS OPERATIONS menu
using the disk wiped server.
5. Verify ping (cross compute) that different VMs are not lost during the Replace Controller
operation.
6. Verify that the Replace Controller operation has passed OK and check the logs.
7. Deploy a VM and verify ping (cross compute) that different VMs are not lost after the
Replace Controller operation has been performed.
8. Connect the new Controller from the Undercloud and verify that the Ceph health is OK
and that the PCS status is OK.
9. Verify that the old controller has been removed from the baremetal node list in the
Undercloud and from the Nova service-list in the Overcloud.
Expected 1. A VM was deployed successfully on each of the hosts.

Result 2. Different VMs were pinged successfully.
3. The Ceph health is OK and the PCS status is OK.
4. The Replace Controller operation was performed successfully.
5. No VMs were lost during the Replace Controller operation.
6. The Replace Controller operation has passed OK and logs look OK.
7. After deploying a VM and verifying with a ping (cross compute) no VMs were lost after
the Replace Controller operation had completed.
8. After connecting the new Controller from the Undercloud, the Ceph health was OK and
the PCS status was OK.
9. The old controller has been removed from the baremetal node list in the Undercloud and
also from the Nova service-list in the Overcloud.
Status OK | not OK
Comments
DN09271553 1-0 © 2021 Nokia 159

R20SP1 and R20SP2
4.2.4.12 CBIS Cluster Shutdown/Power Up
Description In some cases, a CBIS Cluster will need to be shipped between locations. The CBIS Cluster
Shutdown/Power Up procedure allows the user to gracefully shutdown the CBIS Cluster to
re-locate it with minimum disruption.
In addition, the CBIS Cluster Shutdown/Power Up procedure should also be used for any
kind of maintenance that is required on the the entire CBIS Cluster.
(For gracefully shutting down/up specific compute/monitoring/storage nodes as of CBIS

R19, you can now use the Set to Maintenance operation in CBIS Manager).
Test Case ID
Objective Verify that a CBIS cluster can be put into a maintenance mode and can be powered up after
a graceful shutdown procedure.
Estimated 5 hours.
Duration
Supported All
From Version
Prerequisite/s 1. The CBIS Cluster has been installed successfully with 3 controllers.
2. The CBIS Manager Manual is available.
3. The Leaf/Spine switch backups have been completed and saved outside the CBIS
cluster. This is relevant when the CBIS Cluster is relocated and when the CBIS Cluster
will be connected to other switches and reconfiguring the switches is required.
Test Execution 1. Check that all the CBIS web portals inside Horizon such as Kibana, Zabbix, Horizon,
(Vitrage -only up to CBIS R19A-not available in CBIS R20) and the Ceph Manager
dashboard are accessible and viable. Access the web portals via CBIS Manager >
External Tools as shown here:
2. Create a security group that will allow ingress/egress ICMP packets (ping).
3. Create a VM on each unique CBIS host group. For example, if the existing host groups
are SriovPerformanceCompute, DpdkPerformanceCompute and OvsCompute, create
at least 1 x OVS VXLAN VM on the OVS compute, 1 x SR-IOV VM on the SR-IOV
compute and 1 x OVS VLAN VM on the DPDK compute using a flavor with HugePages.
Use the created permissive security group.
4. Validate that the VMs are up and running and reply to the ICMP request (ping).
DN09271553 1-0 © 2021 Nokia 160

R20SP1 and R20SP2
5. Follow the Manual Graceful Shutdown and Startup procedure (only in versions up to
CBIS 19) in the CBIS Operation Manual document.
Note: from CBIS 19A up, use the CBIS Manager manual featuring the Cluster
Shut Down and Powerup Plugin.
6. Verify that all cluster nodes are powered off (using ping to their IPMI addresses), then
continue to the startup phase.
7. Check that the VMs are still up and running and reply to the ICMP packets (ping).
(inside Horizon) and the Ceph manager dashboard and check that they are accessible
and viable as was before the CBIS Cluster Shutdown/Power Up procedure.
Expected 1. All the web portals should be accessible and functional.

Result 2. The security group is successfully created and modified to allow egress/ingress ICMP
packets.
3. All the hosts groups are allocated with at least 1 x VM.
5. The Manual Graceful Shutdown and Startup procedure is successfully executed.
6. All cluster nodes are powered off (using ping to their IPMI addresses).
7. The VMs are up and running and still answer to the ICMP packets (ping).
8. All the web portals should be accessible and functional.
Status OK | not OK
Comments
4.2.4.13 Patch Management
Description Using the CBIS Manager patch management feature.
Note: The given patches are dummy patches, meaning they will not perform
any changes in the system but will be written in the list of applied patches after
loading.
Test Case ID
Objective Verify that the CBIS Manager installs patches.
Estimated 1 hour
Duration
Supported CBIS 18.5

From Version
Prerequisites • The CBIS node is installed successfully.
DN09271553 1-0 © 2021 Nokia 161

R20SP1 and R20SP2
• Dummy patch files (can be downloaded separately from NOLS, see Appendix 2: Artifact
Files for ATP Tests on page 174 in the sub-section titled Patch Files on page 174.
Test Execution
Note: Follow the Patch Management procedure in the CBIS Manager document
to install/rollback the given dummy patches.
1. Install CBIS-20.100.1-DUMMY-PP1.tar.gz
2. Install CBIS-20.100.1-DUMMY-PP1.tar.gz.sha1
3. Rollback CBIS-20.100.1-DUMMY-PP2.tar.gz
4. Rollback CBIS-20.100.1-DUMMY-PP2.tar.gz.sha1
Expected 1. CBIS-20.100.1-DUMMY-PP1.tar.gz is successfully installed and added to the list of

Result installed patches.
2. CBIS-20.100.1-DUMMY-PP1.tar.gz.sha1 is successfully installed and added to the list of
installed patches.
3. CBIS-20.100.1-DUMMY-PP2.tar.gz is successfully rolled-back and removed from the list
of installed patches.
4. CBIS-20.100.1-DUMMY-PP2.tar.gz.sha1 is successfully rolled-back and removed from
the list of installed patches.
Status OK | not OK
Comments
4.2.4.14 Patch Distribution
Description With Multi CBIS clusters, the user can manage other CBIS Managers with the ability to
add, remove, edit and manage patches on each of the CBIS clusters using the Multi CBIS
Operations feature.
Test Case ID
Objective Install patches on remote clusters via the CBIS Manager.
Estimated 1 hour.
Duration
Supported CBIS 18.5

From Version
Prerequisites • The CBIS Cluster is installed successfully.

• An accessible and installed remote CBIS cluster. The remote CBIS cluster needs to be
accessible from the Undercloud Physical Server via the CBIS Manager as it is used for
distributing the patches. Make sure to execute the Patch Management on page 161
ATP test, before this test.
DN09271553 1-0 © 2021 Nokia 162

R20SP1 and R20SP2
• Dummy patch files and their corresponding sha1 checksum files can be downloaded
separately from NOLS. See Appendix 2: Artifact Files for ATP Tests on page 174 in
the sub-chapter titled Patch Files on page 174.
Test Execution
Note: Follow the Multi CBIS Operations procedure from the CBIS Manager
manual. The Multi CBIS Operations assumes that the procedure Manage Multi
CBIS was already executed and that there is at least one remote CBIS cluster in
the Multi CBIS management clusters list.
1. Install CBIS-20.100.1-DUMMY-PP1.tar.gz on the remote cluster(s).

2. Install CBIS-20.100.1-DUMMY-PP1.tar.gz.sha1 on the remote cluster(s).
3. Rollback CBIS-20.100.1-DUMMY-PP2.tar.gz from the remote cluster(s).
4. Rollback CBIS-20.100.1-DUMMY-PP2.tar.gz.sha1 from the remote cluster(s).
Expected 1. CBIS-20.100.1-DUMMY-PP1.tar.gz is successfully installed and added to the list of

Result installed patches and it can be viewed from both the distributer CBIS cluster under CBIS
Manager > Multi CBIS Operations > Multi CBIS Operations and from the remote
CBIS cluster under CBIS Manager > Operations > Patch Management.
2. CBIS-20.100.1-DUMMY-PP1.tar.gz.sha1 is successfully installed and added to the list of
installed patches and it can be viewed from both the distributer CBIS cluster under CBIS
Manager > Multi CBIS Operations > Multi CBIS Operations and from the remote
CBIS cluster under CBIS Manager > Operations > Patch Management.
3. CBIS-20.100.1-DUMMY-PP2.tar.gz is successfully rolled-back and removed from the
list of installed patches. The patches are gone from both the distributer CBIS cluster
under CBIS Manager > Multi CBIS Operations > Multi CBIS Operations and from the
remote CBIS cluster under CBIS Manager > Operations > Patch Management.
4. CBIS-20.100.1-DUMMY-PP2.tar.gz.sha1 is successfully rolled-back and removed from
the list of installed patches. The patches are gone from both the distributer CBIS cluster
under CBIS Manager > Multi CBIS Operations > Multi CBIS Operations and from the
remote CBIS cluster under CBIS Manager > Operations > Patch Management.
Status OK | not OK
Comments
DN09271553 1-0 © 2021 Nokia 163

CBIS Acceptance Test Procedures (ATP) - R20GA, Appendix 1: Basic CLI
R20SP1 and R20SP2
5 Appendix 1: Basic CLI
5.1 Connecting with CLI

Connect using SSH to a Hypervisor Physical Undercloud server IP using any tool you prefer.
Open a tmux session to keep the session persistent and allow sharing in case of failure:
tmux new -s <name>
or join an existing one using:
tmux ls
tmux a -t <name/number of session (left part of ls result)>
Now SSH to the Undercloud using:
ssh stack@uc
5.2 Source Undercloud/Overcloud Credentials

• Source the relevant credential file for your relevant project in the Undercloud:
[stack@undercloud (overcloudrc) ~]$. stackrc

[stack@undercloud (stackrc) ~]$
• Source the relevant credential file for your relevant project in the Undercloud:
[stack@undercloud (stack) ~]$. overcloudrc

[stack@undercloud (overcloudrc) ~]$
5.3 VM/Instances and Hosts Listing

• List hosts:
Source the relevant credential file for your relevant project in the Undercloud
[stack@undercloud (stackrc) ~]$. overcloudrc

[stack@undercloud (overcloudrc) ~]$
DN09271553 1-0 © 2021 Nokia 164

R20SP1 and R20SP2
Then execute:
[stack@undercloud (overcloudrc) ~]$ nova host-list
• List of VMs:
Execute:
[stack@undercloud (overcloudrc) ~]$ nova list
5.4 Availability Zone and Host Aggregate

• List of availability zones:
Execute:
[stack@undercloud (overcloudrc) ~]$ nova availability-zone-list
• Add availability zones:
Execute:
[stack@undercloud (overcloudrc) ~]$ nova aggregate-create <host-

aggregate-name> <availability-zone-name>
• Add host to availability zones:
Execute:
[stack@undercloud (overcloudrc) ~]$ nova aggregate-add-host <host-

aggregate-name> <host-name>
• Add metadata to availability zones:
Execute:
[stack@undercloud (overcloudrc) ~]$ nova aggregate-set-metadata <host-

aggregate-name> <key=value> [<key=value> ...]
5.5 Flavors
• List of Flavors:
Execute:
[stack@undercloud (overcloudrc) ~]$ nova flavor-list
DN09271553 1-0 © 2021 Nokia 165

R20SP1 and R20SP2
• Create Flavor:
Execute:
[stack@undercloud (overcloudrc) ~]$ nova flavor-create <name> <id>

<ram in MB > <disk in GB> <vcpus>
Example:
nova flavor-create SR-IOV-flavor auto 4096 0 4
5.6 Images
• List of Images:
Execute:
[stack@undercloud (overcloudrc) ~]$ nova image-list
• Create Image:
Execute:
[stack@undercloud (overcloudrc) ~]$ openstack image create <image-

name> --location <URL of the image>
[stack@undercloud (overcloudrc) ~]$ openstack image create <image-
name> --file <local file path of image>
glance image-create --name "Rhel7Generic" --is-public true \
--disk-format qcow2 --container-format bare \
--copy-from http://<http server ip>/image/redhat-7.4-ATP.qcow2
5.7 Volumes
• List of volumes:
Execute:
[stack@undercloud (overcloudrc) ~]$ nova volume-list
• Create a volume:
Execute:
[stack@undercloud (overcloudrc) ~]$ nova volume-create <size in GB>
• Attach a volume:
DN09271553 1-0 © 2021 Nokia 166

R20SP1 and R20SP2
Execute:
[stack@undercloud (overcloudrc) ~]$ nova volume-attach <instance-id>

<volume-id>
• Detach a volume:
Execute:
[stack@undercloud (overcloudrc) ~]$ nova volume-detach <instance-id>

<volume-id>
5.8 Security Group

• List a security group:
Execute:
[stack@undercloud (overcloudrc) ~]$ nova secgroup-list
• Create a security group:
Execute:
[stack@undercloud (overcloudrc) ~]$ nova secgroup-create <name>
• Create security group rules:
Execute:
[stack@undercloud (overcloudrc) ~]$ nova secgroup-add-rule <secgroup-

id> <ip-proto> <from-port> <to-port> <cidr>
5.9 VM/Instance Creation

• Create VM with VXLAN network and subnet ( physnet0 )
For creating VM with network and subnet execute:
nova boot --flavor <Flavor-id> --image <Image-id> --security-group

<security group-id> --nic net-id=<Network-id> <VM name>
• Create VM with VLAN network & subnet ( physnet1 for example )
DN09271553 1-0 © 2021 Nokia 167

R20SP1 and R20SP2
For creating VM with network and subnet execute:
nova boot --flavor <Flavor-id> --image <Image-id> --nic port-

id=<Network-id> <VM name>
• Create multiple VMs with VXLAN network and subnet ( physnet0 )
For creating multiple VMs with network and subnet, execute the following, (VMs are spread
between computes in a round-robin):
nova boot --flavor <Flavor-id> --image <Image-id> --security-group

<security group-id> --nic net-id=<Network-id> --max-count <number of
created VMs> <VM name>
• Create multiple VMs with VLAN network and subnet ( physnet1 for example)
For creating multiple VMs with network and subnet, execute the following, (VMs are spread
between computes in round-robin):
nova boot --flavor <Flavor-id> --image <Image-id> --nic port-

id=<Network-id> --max-count <number of created VMs> <VM name>
• Create VM with VXLAN network and subnet on a specific compute (physnet0 )

For creating VM with VXLAN network and subnet on specific compute, execute:
Note: This command can be executed only as admin – ( . overcloudrc).
nova boot --flavor <Flavor-id> --image <Image-id> --security-

group <security group-id> --nic net-id=<Network-id> --
availability_zone zone1:<host-name> <VM name>
5.10 Stack Management

Templates enable creation of most OpenStack resource types, such as instances, floating IP
addresses, volumes, security groups, and users. The resources, once created, are referred
to as stacks. For more information, see https://docs.openstack.org/mitaka/user-guide/
dashboard_stacks.html.
• Stack List:
Execute:
openstack stack list
• Create tenant:
DN09271553 1-0 © 2021 Nokia 168

R20SP1 and R20SP2
Execute:
openstack stack create --parameter InstanceType=m1.medium --parameter

AvailabilityZone=ovs-zone --parameter VolumeType=tripleo-ceph --parameter Image_
Name=redhat-7.4-255NIC --template stack_ATP3_yaml.yml ATP3
5.11 Tenants Management

• List tenants:
Execute:
[stack@undercloud ~]$ keystone tenant-list
• Create tenant:
Execute:
[stack@undercloud ~]$ keystone tenant-create --name <tenant-name>
5.12 Networks
• List Networks:
Execute:
[stack@undercloud ~]$ neutron net-list
• List Subnets:
Execute:
[stack@undercloud ~]$ neutron subnet-list
• List Neutron ports:
Execute:
[stack@undercloud ~]$ neutron port-list
• Create new VXLAN network and subnet ( physnet0 > 1st nic):
For VXLAN network and subnet, execute:
neutron net-create <NetName>

neutron subnet-create --name <subName> <NetName> <CIDR>
DN09271553 1-0 © 2021 Nokia 169

R20SP1 and R20SP2
The above will stdout the new networks parameters. You can always use network net-show
<net id>to list all details.
[stack@undercloud ~]$ neutron subnet-create --name SubX VXLAN 192.168.

40.0/24
The above will stdout the new Subnet parameters. You can always use network subnet-show
<sub id> to list all details
• Unmanaged Network:
If you want to create a neutron network but disable DHCP for the instances, just use:
neutron net-create vxlan_net1 --port_security_enabled=False --enable-

dhcp-false
• Create New VLAN Network and subnet (physnet0 > 1st nic):
For a VLAN network and subnet, execute:
neutron net-create --provider:network_type vlan --provider:segmentation_id

<vlanID> --provider:physical_network physnet0 <NAME>
neutron subnet-create --name <subName> <NetName>
and for subnets with <CIDR> as described below:
An example of VLAN network creation actual details is shown here:
[stack@undercloud ~]$ neutron net-create --provider:network_type vlan --

provider:segmentation_id 333 --provider:physical_network physnet0 VLAN-Net
The above will stdout the new networks parameters. You can always use network net-show
<net id> to list all details.
An eample of subnet creation actual details is shown here:
[stack@undercloud ~]$ neutron subnet-create --name SubY VLAN-Net 192.

168.40.0/24
The above will stdout the new Subnet parameters. You can always use network subnet-show
<sub id> to list all details.
The commands below can be performed with admin credentials:
DN09271553 1-0 © 2021 Nokia 170

R20SP1 and R20SP2
• Create a New VLAN Network with Direct ports and subnet:
neutron net-create --provider:network_type vlan --

provider:segmentation_id <vlanID> --provider:physical_network
physnet1 <NAME>
For direct ports (SR-IOV), we use computes that have been deployed as SR-IOV. There, all NICs
other than the first NIC are dedicated to serving instances on direct ports.
Example:
[stack@undercloud ~]$ neutron net-create --provider:network_type vlan

--provider:segmentation_id 444 --provider:physical_network physnet1
VLAN-Phys1
And for subnet
[stack@undercloud ~]$ neutron subnet-create --name SubXXX VLAN-Phys1

192.168.100.0/24
To create ports in the above network, use:
neutron port-create --vnic-type <direct | macvtap> <network>
[stack@undercloud ~]$ neutron port-create –name portA --vnic-type

direct VLAN-Phys1
The above will stdout relevant parameters for the created port. You can use neutron port-
show <id> also for getting the detailed parameters list.
• Create Neutron Router (HA true default):
neutron router-create <RouterName>

[stack@undercloud~(keystone_cb1)]$ neutron router-create RouterSB
To see an Active/Standby L3 neutron agent relevant for the router, use:
[stack@undercloud~(keystone_cb1)]$ neutron l3-agent-list-hosting-

router RouterSB
To add a router interface for the network created, run:
neutron router-interface-add <RouterName> <SubNet>
[stack@undercloud~(keystone_cb1)]$ neutron router-interface-add

RouterSB SubXXX
DN09271553 1-0 © 2021 Nokia 171

R20SP1 and R20SP2
Added interface 6c6a3478-1cd5-4ca9-b0a4-0f92b06fb970 to router RouterSB.
To list all Router Interfaces, enter:
[stack@undercloud~(keystone_cb1)]$ neutron router-port-list RouterSB
To set the Router with an external network as a gateway, run:
neutron router-gateway-set <RouterName> <Ext Net>
[stack@undercloud ~]$ neutron router-gateway-set RouterSB EXTERNAL_

NETWORK_1
An external network should be setup with the correct subnet/vlan/gateway of the environment, to
the defined external network
Just add the argument “--router:external True” to the net-create.
• Define Floating IP on Instance:

Ensure that:
– The Instance to which you would like to associate the Floating IP has a private neutron
network in the router.
– You have external network defined and the router is set as a gateway with this network.
Create Floating ip from the external
[stack@undercloud ~]$ neutron floatingip-create EXTERNAL_NETWORK_1
Created a new floatingip:

+---------------------+--------------------------------------+
| Field | Value |
+---------------------+--------------------------------------+
| fixed_ip_address | |
| floating_ip_address | 172.16.19.155 |
| floating_network_id | 326d1d60-5edf-45ca-ab98-32f35c7b0ccb |
| id | 184a5054-8bab-4eb5-b12e-bdae8f13941b |
| port_id | |
| router_id | |
| status | DOWN |
| tenant_id | 9b21312f3fc4491586a9f0316954714f |
+---------------------+--------------------------------------+
With a Neutron floatingip-list, you will see the list of floating IPs and their status.
[stack@undercloud ~]$ neutron floatingip-list
DN09271553 1-0 © 2021 Nokia 172

R20SP1 and R20SP2
+--------------------------------------+------------------+-------------
--------+---------+
| id | fixed_ip_address | floating_ip_
address | port_id |
+--------------------------------------+------------------+-------------
--------+---------+
| 184a5054-8bab-4eb5-b12e-bdae8f13941b | | 172.16.19.
155 | |
+--------------------------------------+------------------+-------------
--------+---------+
The next step is to find the ID of the instance port that you would like to associate the floating IP:
neutron port-list | grep <Internal IP>

neutron floatingip-associate <floating id> <neutron port id>
Then:
[stack@undercloud ~]$ neutron floatingip-associate 184a5054-8bab-4eb5-

b12e-bdae8f13941b 0ed071f3-4b98-444e-bf37-dd585683cb34
Associated floating IP 184a5054-8bab-4eb5-b12e-bdae8f13941b

[stack@undercloud ~]$
[stack@undercloud ~]$ neutron floatingip-list
+--------------------------------------+------------------+-------------
--------+--------------------------------------+
| id | fixed_ip_address | floating_ip_
address | port_id |
+--------------------------------------+------------------+-------------
--------+--------------------------------------+
| 184a5054-8bab-4eb5-b12e-bdae8f13941b | 70.0.0.4 | 172.16.19.
155 | 0ed071f3-4b98-444e-bf37-dd585683cb34 |
+--------------------------------------+------------------+-------------
--------+--------------------------------------+
[stack@undercloud ~]$
DN09271553 1-0 © 2021 Nokia 173

CBIS Acceptance Test Procedures (ATP) - R20GA, Appendix 2: Artifact Files for ATP Tests
R20SP1 and R20SP2
6 Appendix 2: Artifact Files for ATP Tests
2 key prerequisites are:
• VM_Connectivity_validation_extend.py script.
• stack_ATP3_yaml.yml
6.1 Images
In the ATP, there are 2 images that can be used (note that the ATP usually refers to only one). They
are:
• redhat-7.4-ATP.qcow2
Credentials for both images are as follows:
user: root
password: password
6.2 Stack Files

In the ATP there are 3 stack files used for several tests, they are as follows:
6.3 Patch Files

There are two patch files (and their corresponding sha1 certificates files) in the ATP. These patch files
are dummies, meaning they will not perform any changes in the system but will be written in the list of
applied patches after loading. The files names are listed below:
• CBISHF_0.5_FRAMEWORK_DUMMY_19A_1.tar.gz
• CBISHF_0.5_FRAMEWORK_DUMMY_19A_1.tar.gz.sha1
• CBISHF_0.5_FRAMEWORK_DUMMY_19A_2.tar.gz
• CBISHF_0.5_FRAMEWORK_DUMMY_19A_2.tar.gz.sha1
6.4 Scripts
Using the following scripts may help simplify some of the tests:
DN09271553 1-0 © 2021 Nokia 174

CBIS Acceptance Test Procedures (ATP) - R20GA, Appendix 2: Artifact Files for ATP Tests
R20SP1 and R20SP2
• Automatically check connectivity between ALL VMs or just VMs with a specific name string using
the VM_Connectivity_validation_extend.py script.
• Automatically check System Health properties - for CBIS19A and on using
cbis-11852_system_health_validation.sh
DN09271553 1-0 © 2021 Nokia 175

CBIS - Acceptance Test Procedures - 12-2021

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

CBIS - Acceptance Test Procedures - 12-2021

Uploaded by

Copyright:

Available Formats

CloudBand Infrastructure

CBIS Acceptance Test Procedures

Copyright © 2021 Nokia. Nokia confidential.

Important Notice on Product Safety

2 About This Document.................................................................................................................................... 7

4 CBIS Acceptance Test Cases......................................................................................................................12

4.1.6.3 Basic VXLAN Connectivity Through Router.......................................................................... 64

4.2.4.6 Scale Out Compute (HCI or none-HCI compute)................................................................ 149

5 Appendix 1: Basic CLI................................................................................................................................164

6 Appendix 2: Artifact Files for ATP Tests................................................................................................. 174

Discovery Center: https://doc.networks.nokia.com/default Portfolio > Select a product > Products

Disovery Center Documentation:

Discovery Issue 1-0 24 June 2021

DN09271553 1-0 © 2021 Nokia 6

2 About This Document

• R20 GA - Scratch and Upgrade installation

2.1 Intended Audience

2.2 Technical Support

2.3 Related Information

No. Document title DN

1 CBIS Guide to Documentation DN1000020022

2 CBIS Glossary DN1000020033

3 CBIS Release Notes DN09260007

4 CBIS Privacy Considerations DN09281088

5 CBIS Product Description DN09259905

6 CBIS Upgrade From R19A MP3 to CBIS R20 SP2 DN1000047791

7 CBIS Manager DN09259917

8 CBIS Installation Manual DN1000036376

9 CBIS Operations Manual DN09259995

10 CBIS Security Hardening Manual DN09260019

11 CBIS Legal Notices relating to FOSS DN09260022

DN09271553 1-0 © 2021 Nokia 7

No. Document title DN

12 CBIS Acceptance Test Procedure (ATP) DN09271553

13 CBIS New Hardware Certification Tests DN09277365

14 CBIS Guide to Networking and Storage Performance DN1000036009

15 CBIS Networking and Storage Performance Benchmarking DN1000008298

16 CBIS FOSS Data - extracted from TALKO site DN1000047880

17 CBIS FOSS Data (zip) - extracted from TALKO site

18 CBIS Mixed Hardware Detailed Tests DN1000047235

19 CBIS Zabbix Metrics KPIs - excel file DN09273684

2.4 Supported Hardware

2.5 Important Note About Copy and Paste

DN09271553 1-0 © 2021 Nokia 8

3.3 10 Test Execution Tips

• Import of an Installation Deploy JSON prepared in advance.

DN09271553 1-0 © 2021 Nokia 9

3.5 Test Outcomes

3.6 CLI Deprecation - Important information

DN09271553 1-0 © 2021 Nokia 10

DN09271553 1-0 © 2021 Nokia 11

4 CBIS Acceptance Test Cases

4.1 CBIS Functional Test Cases

4.1.1 CBIS Automated Installation and Configuration (with/without Nuage SDN)

Test Execution 1. Before the installation:

DN09271553 1-0 © 2021 Nokia 12

4.1.2 Verify CBIS Health

Description CBIS Health Check.

Objective Verify that CBIS is healthy.

Estimated 0.5 Hour

Prerequisite/s The CBIS Cluster is installed successfully.

3. Check the Nova services status by running the command:

source ~/overcloudrc && openstack compute service list

DN09271553 1-0 © 2021 Nokia 13