COMMVAULT PROPRIETARY AND CONFIDENTIAL INFORMATION - INTERNAL AND PARTNER UNDER NDA USE ONLY- DO NOT DISTRIBUTE

Hyperscale XTM – Deployment

course
Module – Operations
Sept 2023

[Link] | 888.746.3849
© 2023 Commvault. See here for information about our trademarks and patents.
COMMVAULT PROPRIETARY AND CONFIDENTIAL INFORMATION - INTERNAL AND PARTNER UNDER NDA USE ONLY- DO NOT DISTRIBUTE

Notices and Disclaimers

Commvault, Commvault and logo, the "C hexagon” logo, Commvault Systems, Metallic, Metallic and logo,
the “Wave” logo, Commvault HyperScale X, HyperScale X, Recovery Reserve, and ThreatWise are
trademarks or registered trademarks of Commvault Systems, Inc. (“Commvault) The unauthorized use of
any Commvault trademark is strictly prohibited.

Other company and product names mentioned herein may be trademarks of their respective owners.
References to any third-party products, services, or websites should not be considered an endorsement
by Commvault. Some examples are for illustration only and are fictitious.

All right, title, and interest, including all intellectual property rights in and to this document and to any
related subject matter (collectively “Ownership Rights”) are owned and expressly reserved by Commvault.
No Ownership Rights are granted to you.

This document is intended for distribution to and personal reference use solely by Commvault customers;
all use of Commvault Solutions, including this document, is governed by Commvault’s Master Terms &
Conditions (currently available at [Link] which
are incorporated herein in their entirety.

This document is provided “as is.” Information in this document, including any specifications, URLs, or
other references, is subject to change without notice.
See [Link]/IP for more information about our trademarks, patents, and other IP rights.

Confidentiality

This document contains information that is confidential and proprietary to Commvault. Without limiting
rights under copyright or otherwise, this information is provided with the express understanding that it will
be held in strict confidence and that no part of this document will be disclosed, used, reproduced, stored,
or transmitted, in whole or in part, for any purpose other than as expressly approved or provided by
Commvault in writing.

©1999-2023 Commvault

2
COMMVAULT PROPRIETARY AND CONFIDENTIAL INFORMATION - INTERNAL AND PARTNER UNDER NDA USE ONLY- DO NOT DISTRIBUTE

Table of Contents
HyperScale XTM ................................................................................................................................................... 4

Learning Objectives .......................................................................................................................................... 5

HyperScale Dashboard .................................................................................................................................... 6

HyperScale Platform Alerts .......................................................................Error! Bookmark not defined.

Maintenance Procedures ............................................................................................................................... 9

Updates and Upgrades .................................................................................................................................. 10

Startign and Stopping a CVFS Node or Cluster .....................................................................................12

Identifying the Active CommServe®......................................................................................................... 14

CommServe® Virtual Machine Management ......................................................................................... 15

CommServe® Virtual Machine Management ......................................................................................... 16

Replacing Disks ................................................................................................................................................. 17

Quiz ....................................................................................................................................................................... 19

Reasons to Add Notes to an Existing Storage Pool ............................................................................ 20

Adding a node ....................................................................................................................................................21

Adding an Additional DDB for Cloud Workloads ................................................................................. 24

Wrap Up.............................................................................................................................................................. 25

3
COMMVAULT PROPRIETARY AND CONFIDENTIAL INFORMATION - INTERNAL AND PARTNER UNDER NDA USE ONLY- DO NOT DISTRIBUTE

HyperScale XTM

HyperScale XTM
Deployment Course

Module: Operations

© Commvault 2023

Commvault Proprietary and Confidential Information Internal and Partner Under NDA Use Only - Do Not Distribute.

Notes:

Hello, and welcome to the Commvault HyperScale XTM Operations Module.

4
COMMVAULT PROPRIETARY AND CONFIDENTIAL INFORMATION - INTERNAL AND PARTNER UNDER NDA USE ONLY- DO NOT DISTRIBUTE

Learning Objectives

Learning Objectives

• Understand the HyperScale

Dashboard and alerting • Familiarize yourself with typical
maintenance procedures and
• Understand how and when to
operations
perform maintenance
procedures exclusive to • Identify how to assess the
After completing this HyperScale XTM health of a HyperScale XTM
module, you will: cluster
• Understand how to replace
disks within the HyperScale • Understand necessary
XTM Appliance precautions to be taken when
administering a HyperScale
• Understand how to add a
XTM environment
single node or multiple nodes
• Understand how to administer
a hosted CommServe

© Commvault 2023
Commvault Proprietary and Confidential Information Internal and Partner Under NDA Use Only - Do Not Distribute.

Notes:

In this module, we’re going to discuss the exclusive HyperScale dashboard that is available for HyperScale
users. We’ll go into further depth on the maintenance procedures that may be required when administering
the HyperScale XTM solution. We’ll also cover the necessary procedures for adding more nodes to an
existing cluster and how to administer the CommServe® for those that have it hosted on their HyperScale
XTM Appliance. This module’s intent is to provide a clear view of the care that is required when managing a
HyperScale XTM cluster and means on how to assess the need for that care.

5
COMMVAULT PROPRIETARY AND CONFIDENTIAL INFORMATION - INTERNAL AND PARTNER UNDER NDA USE ONLY- DO NOT DISTRIBUTE

HyperScale Dashboard

HyperScale
Dashboard
View into all HyperScale XTM clusters
within a CommCell®

Requires zero setup to be used after

deployment

Granular views available from the dashboard for:

Status and health Hardware Security and Software Health System Mount Update Status Storage Pools by
for clusters and monitoring Ransomware status Point Utilization utilization
nodes

©©Commvault
Commvault2023
2023
CommvaultProprietary
Commvault Proprietaryand
andConfidential
ConfidentialInformation
InformationInternal
Internaland
andPartner
PartnerUnder
UnderNDA
NDAUse
UseOnly
Only- -Do
DoNot
NotDistribute.
Distribute.

Notes:

Command CenterTM provides a significant amount of information at a glance through it’s various
dashboards that are available immediately after logging in. The HyperScale Dashboard is a unique view
into the HyperScale hardware that provides both a macro and micro view of the environment. Through
various reporting hooks that are connected to each of the figures on the dashboard, a deeper view is
available for monitoring individual hardware components with the HyperScale XTM solution, checking out
the security and ransomware status of any cluster within an environment, the health of the individual
components of the Commvault® File system, and many more. The HyperScale XTM Dashboard requires
zero setup and is automatically available once the presence of a HyperScale XTM cluster is detected
within a CommCell®.

6
COMMVAULT PROPRIETARY AND CONFIDENTIAL INFORMATION - INTERNAL AND PARTNER UNDER NDA USE ONLY- DO NOT DISTRIBUTE

Reference Architecture HyperScale Hardware Alerts

Notes:

Complimenting the HyperScale XTM Dashboard, we have two set of alerts that are recommended to be
configured. Similar to the HyperScale Appliance alerts we just mentioned in our Post Deployment
module, we have Reference Architecture Hardware alerts. They function the same way where they will
monitor the state of the PSU, the fans, the disk drives, and the network cards. Unlike the Appliance
alerts though, Commvault® will not be notified of the issue as the state of the hardware is the
responsibility of the customer’s.

We also have HyperScale Platform Alerts which are responsible for monitoring various individual
functions for the Commvault® file system along with it’s relevant subsystems and workspaces. This
includes but is not limited to, the status of the critical services for the Commvault® file system, available

7
COMMVAULT PROPRIETARY AND CONFIDENTIAL INFORMATION - INTERNAL AND PARTNER UNDER NDA USE ONLY- DO NOT DISTRIBUTE

space in storage locations that are necessary for HyperScale XTM to function properly, network traffic
issues, and monitoring the status of Commvault® Ransomware Protection on the HyperScale nodes.

These alerts are enabled by default when deploying on CPR2023E, but for environments that are
upgrading to Commvault Platform Release 2023E, users will need to enable these alerts manually. For
more information on the alerts, search for ‘Hardware Alert’ within our documentation portal.

8
COMMVAULT PROPRIETARY AND CONFIDENTIAL INFORMATION - INTERNAL AND PARTNER UNDER NDA USE ONLY- DO NOT DISTRIBUTE

Maintenance Procedures

Maintenance Procedures

1 for Updates 1 for Shutdown 1 for CS VM 1 for Data disk 1 for metadata
and Upgrades of Clusters management replacements disk
replacements

© Commvault 2023
Commvault Proprietary and Confidential Information Internal and Partner Under NDA Use Only - Do Not Distribute.

Notes:

Having the HyperScale XTM solution opens up a series of administrative and maintenance features that
will need to be followed while maintaining a HyperScale environment. These maintenance procedures
will be each discussed in clarity and will be demoed as well in order to help illustrate what needs to be
done.

9
COMMVAULT PROPRIETARY AND CONFIDENTIAL INFORMATION - INTERNAL AND PARTNER UNDER NDA USE ONLY- DO NOT DISTRIBUTE

Updates and Upgrades

10
COMMVAULT PROPRIETARY AND CONFIDENTIAL INFORMATION - INTERNAL AND PARTNER UNDER NDA USE ONLY- DO NOT DISTRIBUTE

Notes:

HyperScale XTM clusters should always be maintained with the latest set of security releases and
storage file system updates. These updates arrive within a monthly cadence as Platform Versions for
HyperScale XTM. Platform versions follow a standard naming convention where the first number is
indicative of the HyperScale generation of hardware that the update is for, followed by a year-month
format. So for example, Commvault® Platform Version 2.2308, is the Commvault® Platform release
for the second generation of HyperScale, HyperScale XTM. 2308 signals to the user that it’s August’s
2023 set of operating system and storage platform updates.

When deploying a new HyperScale XTM cluster in a CommCell®, one of the nodes that are deployed,
will be chosen automatically as a software cache where these platform updates are hosted in order to
perform this nature of update. From Command CenterTM, either or both the Commvault® File System
and operating system, can be updated in one of two ways.

First we have disruptive updates which perform the necessary update procedure by taking down the
cluster’s storage and Commvault® MediaAgent services for a maintenance window where all operations
to the cluster that’s being upgraded, are completely interrupted. While this may seem as if this is
undesirable, a disruptive update does offer the advantage of completing much faster than it’s alternative
form of updating for users on Commvault Platform Release 2023E or higher.

Non-disruptive updates provides a staggered rolling upgrade process where the entirety of the cluster
isn’t taken down for updates, but instead, is taken down one node at a time. This form of updating
cluster allows jobs to continue to be submitted and completed but should be noted that they will subject
possible interruptions that may occur during the upgrade operation. Data protection or recovery
operations may go pending be placed in a waiting state, or possibly even fail outright, as servers are
upgraded individually. Additionally, unlike the faster disruptive procedure, updates that are performed
non-disruptively, may take up to four hours per node.

Both of these operations can be performed from the Server Groups section in Command CenterTM and
selecting Upgrade Software. As operating system upgrades in general, require a reboot, we will always
want to opt for a reboot to be performed. Selecting operating system updates will just update RHEL with
necessary patches and security fixes that are available. Selecting storage updates will take care of the
Commvault® File System alone. Selecting both will update the Commvault® File System first followed
by the Operating System.

11
COMMVAULT PROPRIETARY AND CONFIDENTIAL INFORMATION - INTERNAL AND PARTNER UNDER NDA USE ONLY- DO NOT DISTRIBUTE

Startign and Stopping a CVFS Node or Cluster

12
COMMVAULT PROPRIETARY AND CONFIDENTIAL INFORMATION - INTERNAL AND PARTNER UNDER NDA USE ONLY- DO NOT DISTRIBUTE

Notes:

Although CommServe® LiveSync on HyperScale XTM Appliances has its own exclusive administrative
requirements, all HyperScale XTM environments may run into scenarios where either, individual nodes or the
entire cluster may need to be brought down in order to turn off any number of HyperScale XTM nodes.

We’ll first cover the process for shutting down and starting the Commvault® file system services for an
individual node.

Then we’ll cover the process for bringing a Commvault® File system cluster down and then back up.

When performing any kind of work directly with the Distributed Storage file system, a non-root user account
must be used. To do this, use the substitute user command to switch to admin first.

Since the commands we’ll be using involve both the FQDN (Fully Qualified Domain Name) of the node and
the CVFS cluster name, make sure to get both. If you’re unsure of the CVFS cluster name, we can use the
CVFS command line, hv_deploy, to print this.

To avoid having to enter credentials through this process, we can use public key authentication to connect to
other nodes instead of passwords. We can use this in CVFS since we have passwordless SSH (say each
letter) enabled by exporting the CVFS public key.

Using hv_deploy (say hv deploy) we can use stop node, specify the hostname of the node we want to stop,
and qualify that with the cluster name. This begins a process of shutting down the Commvault file system
services in the correct order. The opposite can be done to start the node as well. Similar to before, CVFS will
start up in the correct designated order.

The hv_deploy command line can be used to stop the cluster which will then proceed to stop the services on
each node in the cluster specified. The same can be done to start the cluster by just qualifying the command
with the cluster to start.

These commands do take some time to run, so please keep that in mind.

These are mandatory steps that *must* be followed in the event that nodes or the cluster need to be powered
off.

13
COMMVAULT PROPRIETARY AND CONFIDENTIAL INFORMATION - INTERNAL AND PARTNER UNDER NDA USE ONLY- DO NOT DISTRIBUTE

Identifying the Active CommServe®

Identifying the Active CommServe®

For HyperScale appliances running CommServe® LiveSync VMs © Commvault 2023

Commvault Proprietary and Confidential Information Internal and Partner Under NDA Use Only - Do Not Distribute.

Notes:

• CommServe® LiveSync is a feature set that provides a high availability solution available for all
Commvault® customers.

• Using CommServe® LiveSync requires the ability to be able to identify which CommServe® is
currently active.

• This is key in administering failover in eligible HyperScale XTM Appliance based environments
which we’ll cover further next.

14
COMMVAULT PROPRIETARY AND CONFIDENTIAL INFORMATION - INTERNAL AND PARTNER UNDER NDA USE ONLY- DO NOT DISTRIBUTE

CommServe® Virtual Machine Management

CommServe® Primary CommServe®

Virtual Machine
Management

CS

Standby CommServe ®

Managed through Interacted with virsh

KVM commands

Only for HyperScale XTM Appliances with CommServe® VMs

©©Commvault
Commvault2023
2023
CommvaultProprietary
Commvault Proprietaryand
andConfidential
ConfidentialInformation
InformationInternal
Internaland
andPartner
PartnerUnder
UnderNDA
NDAUse
UseOnly
Only- -Do
DoNot
NotDistribute.
Distribute.

Notes:

Whether it’s a primary, standby, or both --- CommServe® VMs on the HyperScale XTM appliance are
managed through KVM (pronounce each letter individually), the virtualization module native to the Linux
Kernel.

The CommServe® VMs are able to be interacted with a small bank of virsh (read it like ‘versh’) commands
that are available on Commvault’s documentation portal.

In this example pictured, we can see both nodes 3 and 2 are hosting both a primary CommServe® and a
standby CommServe® respectively.

The names of these virtual machines do not change regardless of which CommServe® is active.

15
COMMVAULT PROPRIETARY AND CONFIDENTIAL INFORMATION - INTERNAL AND PARTNER UNDER NDA USE ONLY- DO NOT DISTRIBUTE

CommServe® Virtual Machine Management

CommServe® Virtual Machine Management

OFF

ON
• Recommended to be followed if HyperScale • Perform LiveSync failover if necessary
node requires maintenance • Easy to do with virsh commands
• Monitor CommServe® LiveSync status when
failing over for completion

Turning CommServe® VMs on and off © Commvault 2023

Commvault Proprietary and Confidential Information Internal and Partner Under NDA Use Only - Do Not Distribute.

Notes:

Say we wanted to shut off the node that is hosting the standby CommServe®, node 2.

With our active CommServe® VM, on node 3, we will not need to go through any CommServe®
LiveSync Failover procedures but be aware that it may be necessary in other situations, for example, if
the active node happened to be on the node we’re shutting down.

Shutting down the CommServe® is simple with the 'virsh shutdown' command and qualifying it with the
name of the CommServe® VM.

After the command is issued, the status of the CommServe® VM can be checked by using the
command, virsh list --all until it reports that it is shutdown.

In order to bring it to turn the CommServe VM back on, just issue the opposite command, virsh start with
the VM name.

Similar to before, with virsh list, we can see when the VM is back online. Shutting down and starting the
VM does not take very long.

It is recommended to verify LiveSync status after bringing any nodes back online in order to ensure that
replication is functioning properly.

16
COMMVAULT PROPRIETARY AND CONFIDENTIAL INFORMATION - INTERNAL AND PARTNER UNDER NDA USE ONLY- DO NOT DISTRIBUTE

Replacing Disks

17
COMMVAULT PROPRIETARY AND CONFIDENTIAL INFORMATION - INTERNAL AND PARTNER UNDER NDA USE ONLY- DO NOT DISTRIBUTE

Notes:

An inevitability in any data center is disk failure. As the expansion of any cluster moves forward, so do
the number of disks, increasing the probability of a disk failure. HyperScale XTM provides a utility that’s
accessible through Command CenterTM or Metallic® which can make the replacement process
painless. Identifying the bad disk can be signaled one of various ways depending on the hardware that
HyperScale XTM is running on. Typically, this is directly signaled to any datacenter administrator
through orange or amber lights on the slot of the respective disk.

When the disk is removed from the server that contains the bad drive, a new disk can be placed right
back in to begin the replacement process. Importantly though, make sure that the new disk is
unformatted prior to beginning the replacement process as this procedure will perform this for you. This
is the only supported method for replacing disks for HyperScale XTM and straying from this procedure
might make the disk ineligible for replacement.

HyperScale XTM also features the ability to replace metadata drives that host the deduplication
database and index cache or the metadata drive for the Commvault® file system itself. This process
involves some preconsiderations and a series of command line operations and that must be ran within
your BMC console. More on this can be found on our documentation portal by searching for ‘metadata
drives’

18
COMMVAULT PROPRIETARY AND CONFIDENTIAL INFORMATION - INTERNAL AND PARTNER UNDER NDA USE ONLY- DO NOT DISTRIBUTE

Quiz

Quiz

Ransomware protection enables ‘selinux’ on

the HyperScale XTM nodes, which requires a
Why should HyperScale XTM reboot to take in effect. Failure to do so may
nodes be placed into risk HyperScale XTM data.
maintenance when enabling
Commvault® ransomware
protection?

© Commvault 2023

Commvault Proprietary and Confidential Information Internal and Partner Under NDA Use Only - Do Not Distribute.

Notes:

Pop quiz – When replacing a hard drive on HyperScale XTM, the disk must be kept in what state prior
to replacement?

The disk must be unformatted as the replacement procedure will format the drive automatically for you.

19
COMMVAULT PROPRIETARY AND CONFIDENTIAL INFORMATION - INTERNAL AND PARTNER UNDER NDA USE ONLY- DO NOT DISTRIBUTE

Reasons to Add Notes to an Existing Storage Pool

Reasons to Add Nodes to an

Existing Storage Pool

Performance Space

More CPU, more memory, more *Storage can scale up to 5 PBs on

disks, yields improved throughput an on-demand basis with as little as
to be balanced to nodes. a single node, to as **many needed.

*All added nodes must have exact same storage of existing nodes. ©©Commvault
Commvault2023
2023
** Up to twelve at a time. CommvaultProprietary
Commvault Proprietaryand
andConfidential
ConfidentialInformation
InformationInternal
Internaland
andPartner
PartnerUnder
UnderNDA
NDAUse
UseOnly
Only- -Do
DoNot
NotDistribute.
Distribute.

Notes:

Nodes may need to be added for a variety of reasons, but these are the most common ones.

The HyperScale XTM solution performs faster as more resources are added to the cluster. With access
to more streams, more storage to disperse deduplication related operations, and more network
operations, workloads can be dispersed more evenly to avoid overloading the HyperScale X cluster.

Naturally, if space is an issue, our HyperScale XTM solution can grow up to multiple petabytes within a
storage pool with one, two, or as many nodes as needed with our easy-to-use installer. As long as the
hardware is identical for all the nodes being added to the existing storage pool, this is a simple
procedure that we’ll cover in a demo next.

20
COMMVAULT PROPRIETARY AND CONFIDENTIAL INFORMATION - INTERNAL AND PARTNER UNDER NDA USE ONLY- DO NOT DISTRIBUTE

Adding a node

21
COMMVAULT PROPRIETARY AND CONFIDENTIAL INFORMATION - INTERNAL AND PARTNER UNDER NDA USE ONLY- DO NOT DISTRIBUTE

22
COMMVAULT PROPRIETARY AND CONFIDENTIAL INFORMATION - INTERNAL AND PARTNER UNDER NDA USE ONLY- DO NOT DISTRIBUTE

Notes:

Before starting this procedure, at least one node in the existing cluster will need to be configured and
updated as a remote software cache. If using Reference Architecture nodes, remember that those
nodes need to be imaged as well. While the deployed nodes may already be on the latest software and
operating system updates, this procedure loads all the necessary software for this procedure to work
properly. Please review this documentation found on our documentation portal for more information.

Once that’s all sorted out, on the new node that’s being added to the cluster, use the same script to
invoke the installer that we’ve used earlier, hsxsetup. This should bring up the simple or advanced
networking setup screen if using HyperScale XTM Appliance, but since this is Reference Architecture in
this demo, we are being taken directly to the advanced network setup to manually define our network
bonds, something we’ve seen earlier in this module. Since we’re adding 12 nodes in this demo, we’ll
select ‘No’ until we’ve configured the remaining nodes.

Now that all our nodes have been configured, selecting ‘Yes, continue’ will take us to the installer
selection that has been shown before.

Selecting Add nodes to an existing cluster will drop users into the installer to select the nodes that are
available to add to the environment. Since we've configured twelve nodes to be added, we're going to
select all of them to be added as part of the expansion. The networking information configured for the
node or multiple nodes along with their networking status, will be displayed for validation purposes.

With networking squared away, an existing node’s fully qualified domain name that is already in the
cluster, and it’s root password are required to be entered in order to perform the necessary background
work to join the cluster.

All that’s left at this point is to enter the CommServe® information that these nodes will be added to and
the automated portion of this installation is ready to begin. The automated portion of the add node
installer is relatively identical to the regular installer we’ve seen earlier with just some additional phases
that are unique to the add nodes procedure. This process allows you to add up to twelve nodes at a time
to a storage pool - in as little as thirty to forty-five minutes. Then, the additional storage and resources
will be available to use, immediately.

23
COMMVAULT PROPRIETARY AND CONFIDENTIAL INFORMATION - INTERNAL AND PARTNER UNDER NDA USE ONLY- DO NOT DISTRIBUTE

Adding an Additional DDB for Cloud Workloads

Adding an Additional
DDB for Cloud
Workloads
• Recommended for outbound cloud auxiliary copy

• Configure Cloud Storage In Command CenterTM

• Specify a Deduplication Database to be located

on /ws/ddb/cloud/ on any of the nodes

• Extra space available on logical volume hosting

Commvault® metadata

© Commvault 2023
Commvault Proprietary and Confidential Information Internal and Partner Under NDA Use Only - Do Not Distribute.

Notes:

All HyperScale XTM Appliances and Reference Architecture models can host deduplication databases
for auxiliary copies that target a cloud storage target.

In order to take advantage of the space available on any HyperScale appliance or reference
architecture node, create cloud storage as you need.

When selecting a location for your DDB, create a folder, if one doesn't exist already, called cloud for
example, and place the DDB there.

This DDB will then be hosted on the NVMe storage available on the HyperScale XTM node.

24
COMMVAULT PROPRIETARY AND CONFIDENTIAL INFORMATION - INTERNAL AND PARTNER UNDER NDA USE ONLY- DO NOT DISTRIBUTE

Wrap Up

Wrap Up

1 2 3
How to assess the health of Maintenance procedures How to properly manage a
a HyperScale XTM cluster available to HyperScale XTM CommServe® server VM
users when it is hosted on a
HyperScale XTM server

© Commvault 2023
Commvault Proprietary and Confidential Information Internal and Partner Under NDA Use Only - Do Not Distribute.

Notes:

In this module, we’ve illustrated various means within Command CenterTM on how to assess the health
of a HyperScale XTM cluster. We’ve also covered the various maintenance procedures that are
available to HyperScale XTM users. We wrapped up by showing a demo of how to properly manage a
CommServe® server VM when it is hosted on a HyperScale XTM server.

25
COMMVAULT PROPRIETARY/CONFIDENTIAL – FOR COMMVAULT PARTNERS UNDER NDA USE – NOT TO BE FURTHER DISTRIBUTED

Thank You

© Commvault 2023

Commvault Proprietary and Confidential Information Internal and Partner Under NDA Use Only - Do Not Distribute.

[Link] | 888.746.3849
© 2023 Commvault. See here for information about our trademarks and patents.