Troubleshooting vCMP guests

Emmanuel T. Jiki, Sr. NSE

[Jan. 2016]
 Objectives
• Introduction – concepts

• CMP & Flexible Resource Allocation

• Hypervisor Configuration

• Guest Configuration

• Troubleshooting & Best Practices

• Useful Solution links

• Q/A

© F5 Networks, Inc 2
 Terminology
Virtual Clustered Multiprocessing vCMP
• vCMP Host
• vCMP Guest
• Virtual Machine
• BIG-IP Cluster
• Virtual Disk
• Cluster IP address
• vCMP daemon
• Virtual management network

© F5 Networks, Inc 3
Introduction to vCMP
Concepts
 Introduction - What is vCMP?
• Virtualized Clustered Multiprocessing • Multiple virtual BIG-IP instance
(vCMP)
 Allocated hardware resources
 vCMP Host = HyperVisor
o CPUs
 vCMP Guest = virtual BIG-IP instance
o Memory

• Software Version (BIG-IP v11.0.0+) o Storage Space – virtual disks

© F5 Networks, Inc 5
 Introduction
Key points
• Guests inherit license from host
• Allocate adequate disk space to support multi-slot software installation
• Provision vCMP as dedicated on the host
• Avoid configuring module objects or failover on the host
• Avoid forcing Guest “offline” when re-setting “device trust”

© F5 Networks, Inc 6
 vCMP Hardware and Software support summary
vCMP is currently only supported from v11.x. Hypervisor and Guest instances can run
different supported versions of software
VIPRION CHASSIS BIG-IP APPLIANCE
• 4480/4800 Chassis and 4340N/4300 • 10200V/10250V
Blades
• 7200V/7200V
• 2200/2400 Chassis and 2250/2150/2100
Blades • 5200V/5250V

© F5 Networks, Inc 7
 vCMP Virtual Disks
• Each blade has own copy
• Maximum Virtual Disk size: 100GB
• Extended automatically when needed
• Disk space is ‘oversubscribed‘
• Virtual Disks not automatically deleted

© F5 Networks, Inc 8
Flexible Resource Allocation
• Simple Striping (11.0.0 -11.3.0)

• Flexible Allocation (11.4 and later)

11.0.0 – 11.3.0 Resource Allocation
• Simple Striping
 It restricts the number of slots to either
“single” or "all”
 The number of cores assigned to each guest
is a static, platform-dependent value

© F5 Networks, Inc 10
 Flexible Resource Allocation v11.4 and above

• Core distribution across slots

• Number of cores / slot

VM VM VM
• Maximum number of slots
VM VM VM • Minimum number of slots
VM VM VM • Allowed slots
VM VM
• Template List

© F5 Networks, Inc 11
 Flexible vCMP Resource Allocation – 11.4+
• After the Guest is up and running, Slots/Cores can be added dynamically from the Host.
This will not interrupt current traffic being processed through the Guest.

• The new client's requests will be assigned to the additional new TMM instances after
the Guest has been updated.

© F5 Networks, Inc 12
Hypervisor Configuration
• Provisioning

• Cluster Configuration

• Network Configuration

• Device groups
 Hypervisor - Provisioning

• The Hypervisor
can ONLY be
provisioned as
Dedicated for
vCMP
• All other
licensed
modules will be
available to be
provisioned
within the
Guests
themselves

© F5 Networks, Inc 14
 Hypervisor – Cluster Configuration (Viprion Only)
• Setup IP addresses for each Slot (vCMP and non-vCMP)

© F5 Networks, Inc 15
 Hypervisor – Network Configuration
• The hypervisor should not have any Self IP addresses, or
routes configured, as all Layer3+ traffic will be handled
by the Guests

© F5 Networks, Inc 16
 Hypervisor – Device Groups
• Do not setup any device groups or trusts at the Hypervisor Level.

© F5 Networks, Inc 17
Guest Configuration
Requested state

Guest state

HA Availability
 Guest Creation – Requested State

Configured Provisioned Deployed

• Allows you to define and • Allows you to define and • Allows you to define and
setup the Guest without setup the Guest, setup the Guest,
assigning any resources • Assigning cores, allocating • Assigning cores, allocating
disk and memory. disk and memory.

• Does not start the Guest • Does not start the Guest • Starts the Guest

© F5 Networks, Inc 19
 Guest Creation – Guest Status
• Displays Guests HA Status, Active Volume, Current Version, HA
Failures, Provisioned Modules.
 Hypervisor and Guests need to be running 11.6 or newer code for this
feature

© F5 Networks, Inc 20
 Guest Configuration – High Availability
• Treat just like any other LTM appliance
 Setup Device Trust with Peer
 Setup Device Group with Peer
 Guests have access to VLANs
 Guests running 11.4+ have access to the status of physical trunks or
interfaces.
o HA groups and VLAN failsafe can now be configured at the Guest level.

© F5 Networks, Inc 21
Troubleshooting
• Failure scenarios

• Restarting failed guests

• Verification commands

• Resource contention
 Troubleshooting – Failure scenarios
What can go wrong with a working vCMP configuration?
• F5 recommended best practices not followed during deployment
• Mis-configuration
• Redundant configuration
• Guest fails to start
• Blade failure
• Software and hotfix installation failures
• Resource contention issues

© F5 Networks, Inc 23
 Troubleshooting – Status Information
• Brief status information of guests
tmsh show vcmp guest

• Prepend with “watch <–n seconds>”

• Appending “all-properties” provides CPU Utilization, Throughput in addition to

status information

© F5 Networks, Inc 24
 Troubleshooting – Restarting failed Guest
• A guest may fail to start on a heavily loaded system

• vcmpd retries 3 times

• Workaround 1: vretry

• Workaround 2: Move to Configured then back to Provisioned/Deployed

• N/B: Don’t delete a Guest in transition

© F5 Networks, Inc 25
 Troubleshooting – LTM log analysis
• Filter for vcmpd in ltm logs
[root@VIP4800-R78-S28:/S1-green-P:Active:Standalone] config # grep vcmpd /var/log/ltm
Jan 24 03:32:06 slot2/VIP4800-R78-S28 alert vcmpd[8540]: 01510011:1: vCMP guest
guest2 powered off.
Jan 24 03:32:06 slot2/VIP4800-R78-S28 info vcmpd[8540]: 01510007:6: Guest
(guest2): VS_RUNNING->VS_STOPPING
Jan 24 03:32:06 slot2/VIP4800-R78-S28 notice vcmpd[8540]: 01510006:5: Guest
(guest2): Stopping VM.

© F5 Networks, Inc 26
 Troubleshooting - Commands
• Essential for Guests deployed in Isolated mode

• Use vconsole command to console to Guest from Host:

Option Command sequence

tmsh prompt: run util vconsole guest_name slot_number

system prompt: vconsole guest_name slot_number

• To return to Host prompt:

 Type ctrl-] to get to telnet prompt
 Type q to exit telnet session

© F5 Networks, Inc 27
 Troubleshooting - Resource contention
• vCMP Guests share common physical storage

• Avoid running Modules with high level of logging and intense disk activity

• LTM, GTM and LTM+GTM combo.

• NEVER Provision the Hypervisor in appliance mode. Doing so renders the

Chassis unusable from a vCMP standpoint

© F5 Networks, Inc 28
vCMP Best Practices
Network Configuration

HA Failover
 Best Practice - Network configuration
When deploying vCMP configurations:
• Interfaces, VLANS and trunks configured on the host (hypervisor) system
• Create dedicated HA VLANs on the host but self IP addresses on the guests
• Create RD and VLANs residing in non default partition on the guest not host
• Configure multiple blade interfaces into trunk and assign to VLAN
• Don’t configure HA (Traffic & device groups) on the hypervisor
• Similarly configure failsafe (gateway, network) on the guests
• Set up NTP on the hosts (Essential)
 Verify NTP server reachable – # ntpq –c peer

© F5 Networks, Inc 30
 Best Practice – Other considerations
Miscellaneous:
• Gracefully shutdown all guests prior to upgrading host system
• Upgrade Guests one at a time for multi-slot guests
• Routinely delete unwanted virtual disks on the system
• Ensure initial Guest .iso files are retained on the host system
• Allow adequate boot up time for fully bladed VIPRION with Guests deployed
• Additional RAM per Guest can be obtained by adding more cores

© F5 Networks, Inc 31
 Best Practice - HA Failover
• Guest sync + HA within chassis – not supported

• Guest sync + HA between chassis – is supported

• Guest sync+ HA failover via Device and Traffic groups

• Host sync + HA failover – is not supported

• HA Group ( Layer2) supported

• Avoid VLAN failsafe on Guests sharing same VLAN

© F5 Networks, Inc 32
 Best Practice - vCMP Guest High Availability
vCMP Hosts do not failover

vCMP Guest vCMP Guest

Guest 1a Guest 1b
Guest 2a Guest 2b
Guest 3a Guest 3b

© F5 Networks, Inc 33
Summary and useful solution links
• Solution links

• Recap
 Useful Solution Links
• SOL14218: vCMP guest memory/CPU core allocation matrix
• SOL15372: Overview of the vconsole utility
• SOL17511: TMM may leak memory when a vCMP guest allocates
only one CPU per blade
• SOL14088: vCMP host and compatible guest version matrix
• SOL14727: BIG-IP vCMP hosts and guests configuration options
• SOL15930: Overview of vCMP configuration considerations
• SOL16556: vCMP guests may experience excessive TMM
redirects after entering multiple provisioning states

© F5 Networks, Inc 35
 Useful Solution Links
• SOL14843: Reallocating the disk resource when provisioning
multiple modules on the BIG-IP VE system and BIG-IP vCMP
guests
• SOL15757: Rebuilding device trust for VIPRION devices in HA
device groups
• SOL14392: Configuring a cluster member IP address for all slots
on a VIPRION platform
• SOL15915: Reallocating disk resources when upgrading a vCMP
system

© F5 Networks, Inc 36
 Recap
• Things to take away from this presentation
• Relationship between vCMP Host and Guest configuration and setup
• vCMP best practice
• vCMP troubleshooting tips

If I can be of further assistance please contact me:

Emmanuel T. Jiki | e.jiki@f5.com