Lenovo de Series Storage Best Practices With Veeam Backup Replication

Lenovo ThinkSystem DE Series Storage Best
Practices with Veeam Backup & Replication 11
September 2021
In p artnership with
Lenovo.com/storage
Lenovo White Papers DE Series Storage Best Practices with Veeam Backup & Replication 11
Contents
1 Summary ................................................................................................................................................ 3
1.1 Audience ....................................................................................................................................... 3
1.2 Purpose ......................................................................................................................................... 3
2 Solution Overview .................................................................................................................................. 3
2.1 DE Series as Veeam Backup & Replication Repository ............................................................... 3
3 Deployment Considerations ................................................................................................................... 4
3.1 Capacity Planning ......................................................................................................................... 4
3.2 Compute ........................................................................................................................................ 6
3.3 Connectivity ................................................................................................................................... 7
3.4 Architecture ................................................................................................................................... 7
3.5 Optimization ................................................................................................................................ 12
3.6 Backup Resiliency ....................................................................................................................... 14
4 DE Series Configuration ...................................................................................................................... 15
4.1 DE Series Configuration Guidelines for VBR 11 Backup Repositories ...................................... 15
4.2 Host Connectivity ........................................................................................................................ 16
5 Veeam Backup & Replication 11 ......................................................................................................... 18
5.1 Sizing Guidelines ........................................................................................................................ 18
5.2 Backup Server ............................................................................................................................. 18
5.3 Proxy ........................................................................................................................................... 18
5.4 Repository ................................................................................................................................... 18
6 Lab Environment .................................................................................................................................. 18
6.1 Lab Hardware .............................................................................................................................. 19
6.2 Lab Software ............................................................................................................................... 19
6.3 Lab Environment Configuration................................................................................................... 19
6.4 Tuning ......................................................................................................................................... 20
7 Appendix - Lab Test Statistics ............................................................................................................. 33
© 2021 Lenovo. All Rights Reserved. 2

1 Summary
The combination of Veeam software and Lenovo ThinkSystem DE Series storage arrays yields a simple,
reliable, and high-performance backup and recovery system capable of meeting the most stringent
enterprise demands spanning workloads across virtual, physical, unstructured (NAS) and cloud data. This
paper will focus on optimizing the deployment of DE Series storage as a primary Veeam Backup and
Replication (VBR) repository.
1.1 Audience
Pre-sales systems engineers, consultants, solution architects, backup administrators, storage operators
and data center managers. Working knowledge of Veeam Backup & Replication and DE Series storage is
recommended.
1.2 Purpose
This document aims to familiarize the reader with the design considerations relative to the deployment of
DE Series storage as a Veeam backup repository and makes recommendations for optimizing configuration
to meet your organization’s data protection requirements.
2 Solution Overview
This section overviews the VBR with DE Series architecture highlighting DE Series as the primary backup
repository and as a source for secondary backup storage. While Lenovo-specific infrastructure is depicted
as backup source e.g. VM, NAS, physical backup for illustration purposes, VBR and DE Series storage are
compatible with any valid Veeam backup source.
2.1 DE Series as Veeam Backup & Replication Repository

Figure 1 represents a high-level architecture of a DE Series as primary backup storage for a Veeam
deployment along with secondary / tertiary backup targets for backup redundancy.

Figure 1. DE Series primary backup storage with secondary / tertiary on-premises or offsite backup targets
3 Deployment Considerations
While this section outlines the primary considerations for the deployment of VBR and DE Series storage, it
should not be considered a comprehensive best practices treatment. For full coverage regarding the most
current VBR best practices refer to - https://bp.veeam.com/vbr.
3.1 Capacity Planning

Foundational to any backup system deployment is the estimation of required storage to meet desired
retention policy goals. A number of tools are publicly available for this calculation including –
• RPS tool (simple) - http://rps.dewin.me/

• Veeam best practices site calculator (advanced) - https://vse.veeambp.com/#/

For example, using the RPS tool an environment consisting of 15TB of source backup data with a
2 week/14-day daily backup retention target, a conservative 10% daily change rate and a weekly synthetic
full would yield in range of 28TB of DE Series storage required for the environment, as shown in Figure 2.
Figure 2. RPS storage estimation example
Any number of factors could materially affect this estimation, including but not limited to the following
considerations:
• Instead of a 14-day retention policy, a 30-, 60- or 90-day retention requirement would significantly
increase storage requirements
• Requiring a weekly active full backup would considerably increase the required storage

• In the case where backups older than one week were being offloaded to a secondary S3 target,
primary backup storage requirements would be decreased
Additional information around such variances can be found in the optimization section below.
3.2 Compute
The foundational compute components for any VBR deployment are as follows (see Figure 3):
• Proxy – source data mover responsible for reading the backup source (VM, NAS, etc.) and
performing preliminary backup data stream processing e.g. compression, etc. Windows or Linux or
a combination of the two platforms may be deployed in a proxy role.
• Repository – target data mover responsible for writing backup data to storage repository such as
DE Series. Windows or Linux or a combination of the two platforms may be deployed to execute
the repository compute role.
• Backup server – responsible for scheduling, orchestration, management, etc. of the backup /
disaster recovery system. Windows-only.
Figure 3. Veeam Compute Components
At install time all the above components are deployed to the Windows server hosting the VBR instance.
While an all-in-one single server / scale up approach is suitable for smaller environments (<500 VMs,
agents), a scale out strategy is recommended for medium to large environments. See the architecture
section below for additional detail.

3.3 Connectivity
While VBR is tolerant of “poor” IP connectivity and does work at virtually any underlying networking speed,
10Gbps or higher networking speed between the Veeam components is recommended for production
deployments.
Another frequently overlooked connectivity requirement for VBR is name resolution. VBR managed servers
(vCenter, proxies, etc.) added by name should be resolvable to IPV4 addresses. This is particularly
important for adding vSphere or Hyper-V clusters in which case VBR automatically retrieves the names of
the hosts comprising the cluster from the cluster management layer and if unable to resolve the cluster host
names, backup will not function.
If the VBR server and or external proxies or repositories are connected to multiple networks preference can
be specified for backup traffic. For additional information, refer to the following documentation:
https://helpcenter.veeam.com/docs/backup/vsphere/select_backup_network.html?ver=110.
VBR can also enforce backup traffic throttling to insure QOS for other applications. See
https://helpcenter.veeam.com/docs/backup/vsphere/setting_network_traffic_throttling.html?ver=110 for
more detail.
Another QOS function supported by VBR is the ability to throttle read rate from production storage based
on read latency. If source read latency exceeds a millisecond threshold set by the administrator additional
load (jobs/tasks) will be suspended until production storage appears healthy again. For additional
information, refer to the following documentation:
https://helpcenter.veeam.com/docs/backup/vsphere/io_settings.html?ver=110
If DE Series storage is connected via Fibre channel or SAS the repository compute component of the VBR
infrastructure would need appropriate hardware connectivity.
3.4 Architecture
The compute components outlined above may be combined or disaggregated in any number of ways and
the following exemplifies a subset of these methods.
VBR
As noted in section 3.2 environments exceeding 500 VMs or agent backups are generally good candidates
for a VBR scale out approach as processing roles are separated from the VBR server instance. For smaller
environments a single server “appliance” approach shown in Figure 4 is viable.

Figure 4. VBR Single Server “appliance” Architecture
As processing requirements or source workload volumes increase the source proxy and repository compute
roles can be offloaded to other VBR-managed servers. The proxy and repository roles can be split,
combined, virtualized, or otherwise offloaded from the VBR server in a way that is most rational for a given
environment. As previously noted, proxy and repository roles are not limited to the Windows platform and
may also be hosted by Linux OS’s.
If a physical server scale-out approach is preferred, the “combo” or proxy+repository server strategy may
be employed. This technique combines source proxy and repository compute on physical servers, as shown
in Figure 5. While VBR is represented external to the “combo” servers, it could of course run on one of the
hosts with the remainder of the servers in the scale out “cluster” hosting proxy/repository-only roles.

Figure 5. Combo Server Scale Out
Alternately, deploying the source proxy as a VM is often a good approach particularly in a predominantly
virtualized environment. Deploying proxies as VMs enable facilities such as VMware hotadd transport to
mitigate hypervisor host backup operation networking impact, as illustrated in Figure 6

Figure 6. Virtualized Scale Out Proxy
VBR Server Placement

As Veeam is an entirely software-based solution, deployment options are considerable and there are any
number of ways to successfully instantiate a VBR system. In terms of precisely where VBR is deployed, it
goes without saying that it should never be deployed on a domain controller, SQL Server or other production
Windows host but rather should be installed on a Windows host dedicated to VBR.
Furthermore, from a best practices perspective, an “out-of-band” approach is preferred, that is Veeam
should be deployed so as not to have production system dependencies such as AD domain membership
(unless required for Exchange, SQL, etc. item-level recovery). In this way Veeam can affect restore
operations without dependency on any other production infrastructure in the event of a catastrophic event.
The question as to whether to virtualize VBR is another frequent point of debate, obviously this implies an
“in-band” approach since VBR would be running on the same infrastructure that it is protecting however on
the “pro” side one could argue that VBR is benefitted by capabilities such as management flexibility (scaling,
etc.), migration ease and high availability afforded by the virtualization layer. The obvious downside,
however, is that if the entire virtualization cluster is lost then so is VBR.

The bottom line relative to deploying VBR as a VM is that yes is it possible, yes it does work correctly, yes
it’s supported and frequently implemented in this manner however an out-of-band VBR deployment is
preferred.
Proxy Placement
In terms of proxy compute location in a production infrastructure the primary consideration is proximity to
the source workload. If multiple proxy server instances are available to service active backups then VBR
will attempt to choose the most proximate proxy however it will employ all available proxies if backup
concurrency requires. Proxy affinity may be explicitly defined to prevent unwanted WAN traversal or
network “hair pinning”. While the proxy is required to read the full data stream from the backup source the
compression it applies prior to transmission to the repository is generally 2:1 so that the communication
between the proxy and repository will be roughly half the amount of source/upstream data being read.
The other primary consideration for proxy placement is the required backup transport method. As
mentioned in the section above virtualizing the proxy instance presents the possibility of leveraging VMware
hot-add for direct virtual disk access from the proxy. Alternately if the backup source data is located on
Fibre Channel storage a physical proxy with appropriate HBA’s would be required for advanced transport
access i.e. Direct SAN or backup from Storage Snapshots over Fibre Channel. Note that VBR will fall back
to network (NBD) mode transport access if advanced transports are unavailable. For additional information
on backup transport modes refer to the following documentation:
https://helpcenter.veeam.com/docs/backup/vsphere/transport_modes.html?ver=110
Repository Server Placement

As with the proxy the repository server should be in as near proximity to the backup storage as possible
otherwise repository server placement is entirely flexible.
DE Series Storage
In a manner similar to the way data processing elements can be scaled out, storage scale-out is achieved
via a VBR “Scale-out Backup Repository” (SOBR). A SOBR aggregates multiple storage extents into a
single logical storage construct for VBR backup storage. Storage extents can be added, migrated, or
removed dynamically without incurring administrative overhead associated with reconfiguring backup job
target storage in the case where backup jobs point to a discrete single storage extent. In a DE Series
environment, extents may consist of LUNs from a single or multiple DE Series arrays connect to single or
multiple repository compute elements. SOBR logic distributes backup job load across the available extents,
as shown in Figure 7.

Figure 7. SOBR Storage Scale-out
For further treatment of SOBR capabilities, refer to the following documentation:

https://helpcenter.veeam.com/docs/backup/vsphere/backup_repository_sobr.html?ver=110
3.5 Optimization
In addition to the macro-level architectural considerations addressed in the preceding sections, there are
many other related technology integrations and tuning capabilities that may be considered for the VBR and
DE Series deployment. A subset of these includes the following elements:
• Storage snapshot integration – For many enterprise production storage solutions including
Lenovo ThinkSystem DM Series (ONTAP) storage, VBR is able to leverage native storage
snapshot orchestration to improve backup processing efficiency for NAS & VM workloads as well
as increase RPO for VM workloads.
• ReFS/XFS – VBR repositories can leverage the built-in block cloning capabilities available in
Windows ReFS and Linux XFS filesystems either of which are fully compatible with DE Series
storage (note that this was selected in the section 3.1 storage simulation example). This

dramatically improves merge and synthetic full backup operations and effectively facilitates
“spaceless” synthetic full’s. For more information refer to the following documentation:
https://helpcenter.veeam.com/docs/backup/vsphere/backup_repository_block_cloning.html?ver=1
10
• Capacity Tiering – While not a direct benefit of a VBR+DE Series deployment, the ability to tier
older backup chains or immediately copy DE Series repository backups to on-premises or cloud-
hosted S3 storage facilitates more cost-effective long-term storage, as well as significantly
enhancing backup redundancy and resiliency.
• Backup Operations – Organizations frequently seek to minimize the daily backup window desiring
the fastest possible daily backup cycle but without sacrificing storage density. The following
architecture attempts to address in the most cost effective and high-performance manner. In this
scenario the SOBR policy routes daily incremental backups to an all-flash DE Series storage extent
minimizing required backup window time while utilizing a spinning disk DE Series array for full
backup/merge operations, as shown in Figure 8.
Figure 8. VBR / DE Series Architecture for Optimized Incremental Backup

• Block Size – By default, VBR backup jobs leverage a 1MB source block (pre-compression) which
is generally the best choice from an overall performance standpoint. The available block size
options are as follows:
o Local Target (large blocks) – 4MB

o Local Target – 1MB
o LAN Target – 512K
o WAN Target – 256K
Block size choice involves a number of trade-off’s which must be assessed relative to deployment
requirements. The smaller the block size the more effective deduplication becomes hence lowering
overall storage requirements but at the expense of additional repository resource requirements
(RAM/CPU) to accommodate processing the increased deduplication metadata. Larger blocks
generally net higher performance but at the expense of fragmentation and reduced deduplication
efficacy. Block size choices can also have a material impact on S3 storage costs from an API
perspective. Since most public cloud S3 storage providers charge for API calls (GET, PUT), a “LAN
Target”-based job would incur 8x the API calls vs. a “Local Target (large blocks)”-based job.
For additional block size selection information, refer to the following documentation:
https://bp.veeam.com/vbr/VBP/3_Build_structures/B_Veeam_Components/B_backup_repositorie
s/block.html#block-sizes
3.6 Backup Resiliency

With the increased frequency and destructiveness of ransomware attacks, not merely maintaining backups
but insuring backups are ransomware-resistant has become crucial. Veeam’s longstanding “3-2-1 rule” (3
copies of your backups across at least 2 distinct media types with an offsite backup) is rapidly giving way
to a 3-2-1-1 rule where the additional “1” is an air gapped or immutable backup copy as ransomware is
evolving to detect, delete and otherwise disabling backups as a first order of business. VBR supports an
array of options which includes DE Series storage.
Tape
Veeam’s LTO tape support (including Lenovo LTO tape drives/libraries) has evolved over many years and
has traditionally been the most preferred solution for a fully air-gapped, ransomware-invulnerable backup
storage. Recently VBR has added LTO WORM tape support to ensure that in addition to tapes in storage,
tapes in a library or drive cannot be erased or overwritten.
Object
As mentioned in the prior section, VBR supports backup copy/move to S3 storage and supports a significant
percentage of commercially available S3 storage solutions. For a smaller subset of these, S3 immutability
is also supported. Validated solution listings can be found at https://www.veeam.com/alliance-partner-
technical-programs.html.

Solutions with the “Object Immutability” designation are fully compatible with VBR S3 immutability. An
unofficial list is maintained at https://forums.veeam.com/object-storage-f52/unoffizial-compatibility-list-for-
veeam-cloud-tier-t56956.html
Hardened Linux
New to VBR v11 is support for immutable disk-tier backups which are compatible with DE Series storage.
This approach enables immutability for a designated time period by tagging VBR backup files with an
immutable attribute for Linux filesystems supporting that feature. The XFS filesystem is recommended as
it supports this capability as well as possessing the block cloning advantages mentioned in the prior section.
Figure 9. DE Series Backup Immutability with Veeam Hardened Linux Repository
4 DE Series Configuration
4.1 DE Series Configuration Guidelines for VBR 11 Backup Repositories
Lenovo recommends avoiding thin volumes while setting up DE Series with Veeam.
For optimal performance, Lenovo recommends that you follow these guidelines:
• Use RAID 6 (8+2) volume groups.

• Create multiple volume groups, having an even number when possible so that you can achieve balance
between owning controllers. Make sure of drawer loss protection (DLP) when applicable to reduce the
risk of data unavailability and/or data loss.
• Create a single standard (not thin) volume per volume group.
• When creating the volumes, select a 512KB segment size to match the 512KB transfer size that Veeam
Backup & Replication presents as a sequential write to the system after sequencing I/O unless using
ReFS-formatted filesystems in which case 256K is recommended.
(https://bp.veeam.com/vbr/VBP/3_Build_structures/B_Veeam_Components/B_backup_repositories/bl
ock.html#ntfs-or-refs).

• Run multiple backup jobs to each repository.
Note: The preceding setup can be configured without any hot spares too, to achieve better performance in
case of drive failure. A caveat is that data loss might occur if more than two drives fail at once.
For a large configuration, Lenovo recommends that you follow these guidelines:
• Use dynamic disk pools (DDP) technology to maximize ease of use and for fast rebuild times.
• Create multiple dynamic disk pools (DDP), having an even number when possible so that you can
achieve balance between owning controllers. Make sure of DLP when applicable to reduce the risk of
data unavailability and/or data loss.
• Create a single standard (not thin) volume per DDP.
• When creating the volumes with DDP, the default segment size is 128KB. No additional selection is
required.
• Run multiple backup jobs to each repository.
Note: RAID 10 was also tested. It can also be considered as an option for volume configuration, but it is
not recommended due to a single parity and capacity penalty. Also, write throughput performance
of RAID 6 and DDP is better in comparison to RAID 10.
4.2 Host Connectivity

DE Series arrays provide multiple options for connectivity.
Host Connectivity for DE2000H, DE4000H and DE4000F

The DE2000H, DE4000H and DE4000F controllers have the following base hardware features:
• Dual Ethernet ports for management-related activities

• Either two optical FC/iSCSI or two RJ-45 iSCSI baseboard ports for host connection
• SAS drive expansion ports to attach expansion drive shelves
Note: Adding an optional HIC is only needed if you want to use the SAS protocol, need more than two host
ports per controller, or want to use both FC and iSCSI protocols.
Host Connectivity for DE6000H and DE6000F

The DE6000H and DE6000F controllers have the following base hardware features:
• Dual Ethernet ports for management-related activities

• Dual optical 16Gbps FC or 10Gbps iSCSI baseboard ports for host connection
• Dual 12Gb SAS drive expansion ports to attach expansion drive shelves
Note: Adding the new optional HICs for the DE6000 controller provides faster host interfaces for iSCSI, FC.

Controller Optional Host I/O
DE2000H, DE4000H & 2-port 10Gb iSCSI (Base-T) per controller

DE4000F
• Controllers must match 2-port 12Gb SAS (wide-port) per controller
• The Base-T iSCSI
onboard controller can 4-port 12Gb SAS (wide-port) per controller
only use the 2-port BaseT
HIC 2-port 10GB iSCSI (optical)/16Gb FC per controller
• A software feature pack
can be applied to convert
the FC HIC ports to iSCSI
or to convert iSCSI
HIC ports to FC 4-port 10Gb iSCSI (optical)/16Gb FC per controller
4-port 12Gb SAS (wide-port): uses mini-SAS cables
4-port 32Gb FC (see the Hardware Universe for SFP details: 32Gb
DE6000H & DE6000F
SFP supports 32Gbps and 16Gbps but not 8Gbps or slower)
• Controllers must match
• A software feature pack* 4-port 25Gb iSCSI (see the Hardware Universe for SFP details: must
can be applied to convert set port speed for 10Gbps or 25Gbps using ThinkSystem System
the FC base ports to Manager: the 25Gbps SFP does work for 25Gb and 10Gb speeds, but
iSCSI the port does not change speeds automatically)
Optimal design is based on your existing environment and how the data flows from primary storage through
the Veeam proxy(s) to its final destination on the DE Series array. One design approach is to leverage
12Gb SAS for connectivity from the Veeam server to the DE Series array. This approach provides the target
backup repository (DE Series) to Veeam as a direct connection. Having a direct connection to the DE Series
array prevents having to write the backup file across an existing network.
This configuration might not be ideal for larger environments in which Veeam’s distributed architecture is
implemented and in which multiple proxy servers process backup data. Having a dedicated network for
backup targets might make more sense in those cases. In any case, flexible options allow tailoring for any
environment. Additionally, VBR include built-in bottleneck detection to help optimize the backup data flow
as illustrated in the tuning section later in this document.

5 Veeam Backup & Replication 11

5.1 Sizing Guidelines
This section covers the CPU and memory considerations for the primary compute components outlined in
section 3.2.
For a single server scale-up or “appliance” approach, VBR sizing is cumulative with regard to the individual
components that follow. Compute and RAM for all three are considered additive and the single server will
have to meet the combined requirements.
Similarly for a proxy + repository or “combo” server approach the VBR infrastructure server(s) need to satisfy
compute requirements for the combined proxy and repository roles.
5.2 Backup Server

Recommended Veeam backup server configuration is 1 CPU core (physical or virtual) and 5GB RAM per
10 concurrently running jobs. Concurrent jobs include any executing backup jobs as well as any job with a
continuous schedule such as backup copy jobs and tape jobs. The minimum recommendation for the VBR
server is 4 CPU cores and 4GB RAM.
5.3 Proxy
It is a best practice to plan for 1 physical core or 1 vCPU and 2GB of RAM for each task as defined in the
VBR proxy server tasks for a given proxy compute instance (note for v11 recommended RAM per task is
200MB which is not yet reflected in the Veeam best practices guide). Each task processes 1 VM disk at a
time, and CPU/RAM resources are used for inline data deduplication, compression, encryption, and other
features that are running on the proxy itself.
5.4 Repository
The recommended amount of CPU for a repository is 1 core per concurrent job and 4GB RAM per core that
processes data on a repository server. At least 2 cores allow for the operating system to be more
responsive.
For further details, refer to the design section in the Veeam best practices guide:
https://bp.veeam.com/vbr/VBP/2_Design_Structures/.
6 Lab Environment
The lab environment leveraged in the testing below however modest affords ample opportunity to
demonstrate scale-up and scale-out for both VBR and DE Series. In this section we will progress through
a series of optimization steps beginning with out-of-the box defaults. The primary goal will be to maximize
throughput / backup speed and to achieve at least 1GB/s sustained backup throughput from the VBR
components to the DE Series storage.

6.1 Lab Hardware
Storage
DE4000F – 12 800GB SSD’s
Compute
SR630 V2 (x2) – 2 Intel Xeon Gold 32 Core processors, 256GB RAM, 960GB SSD, 3TB SATA
HDD, 2 10Gb NIC’s,
Networking
10Gb between all components
6.2 Lab Software

• Veeam Backup & Replication version 11.0.0.837 P20210525
o Windows Server 2019 (VBR server, repositories)
o Ubuntu Linux 20.04.2 LTS (proxies, test / backup source VM’s)
• DE Series
o ThinkSystem SAN OS Version 08.63.00.02.005-LEN
o Management Software - 11.63.54.9018
o Controller Firmware - 08.63.00.02
6.3 Lab Environment Configuration

The lab environment consists of two vCenter-managed ESXi hosts with local HDD and SSD. For the initial
configuration the HDD datastore on each ESXi instance hosts 15 100GB Linux VMs (with 3 attached virtual
disks). In addition, a Windows 2019 VBR VM serving all three primary compute roles (server, proxy,
repository) with specifications only slightly exceeding the requirements listed at:
https://helpcenter.veeam.com/docs/backup/vsphere/system_requirements.html?ver=110 (in this case
4 cores and 8GB RAM) resides on one of the ESXi hosts.
A 5TB iSCSI LUN backed by a single disk pool is presented from the DE Series storage. Multipath IO is
enabled on the VBR server via the DE Series Windows DPM driver
(https://datacentersupport.lenovo.com/us/en/products/storage/lenovo-storage/thinksystem-
de4000f/7y76/7y76cto1ww/j300gmk0/downloads/driver-
list/component?name=Software%20and%20Utilities).
The VBR server mounts the DE Series LUN to drive “V:” formatted ReFS / 64K block size per Veeam best
practices:(https://bp.veeam.com/vbr/VBP/3_Build_structures/B_Veeam_Components/B_backup_repositor

ies/block.html#ntfs-or-refs). This storage is configured as a standalone backup repository extent leveraging

“Per-VM” backup files in VBR.
VBR defines a single backup job with all 30 VMs included targeting the DE Series repository.
Initial configuration is shown in Figure 10.
Figure 10. Initial Test Environment Configuration
Each backup iteration that follows represents a full backup run. Backups are purged after each test for consistency.
6.4 Tuning
The Veeam backup process is a relatively straightforward proposition in that it is a matter of available
processing resources, including the following performance characteristics:
• Available backup source (in this case VM’s hosted in VMware datastores) read IOPs
• Available backup storage (repository) write IOPs

• Available parallel processing capacity

o Proxy tasks
o Repository tasks
• Available networking bandwidth between components
The testing will employ platform native tools to identify potential remedies for sub-optimal performance
including the built-in Veeam backup job bottleneck analysis statistics, VMware host-level, DE Series and
OS-level monitoring.
Refer to the appendices for VBR job and DE Series performance data for each of the test iterations.
Default Configuration
The default configuration yielded around 80 MB/s in sustained performance (charts) over a two-hour
window (job was not allowed to run to completion). No significant bottlenecks were observed during this
run.
Optimization I
From the data in the charts for the initial configuration its evident that at most only two VM’s are being
processed concurrently. While it does vary somewhat per source workload type, the parallelism that is
employed by the backup proxy during backup operations for VMware workloads is a per-VM disk type of
parallel processing. The number of parallel proxy tasks is established in the proxy properties, as shown in
Figure 11.
Figure 11. Proxy Task Setting

As noted in section 5.3 Each Veeam proxy task “slot” attaches to a processor virtual or physical core on
the proxy server hence any cores present on the virtual or physical hardware not required for base OS
operation are eligible for assignment for usage as proxy resources for source backup operations. As tasks
are scaled up on a given proxy server so are RAM requirements. For each task added to the available pool
a corresponding 2GB of RAM increase is also required.
In the test environment the single VBR server is also hosting repository compute where a similar “task”
concept also applies. In the repository case, a “task” is either a backup job in the case where all the
workload backups in a job are being writing to a single backup file or in the case of per-VM backups (true
for our test environment) each “task” equals a concurrent VM backup. Additional RAM requirements for
repository compute are 4GB per additive repository task/core.
As a first step to scale up our single VBR server VM, the virtual core count will be raised to 16 with proxy
and repository tasks increasing from 2 to 8 respectively. This ‘nets an additional 36GB of RAM above the
previously allocated 8GB required for the VBR server which we will round up to an even 64GB.
After applying these VBR server VM configuration updates and increasing task count for the proxy and
repository, backup performance was increased more than 3.5x (from around 80MB/s to 320MB/s) over the
out of the box configuration from the prior scenario (charts). VBR job bottleneck analysis does not exhibit
any substantial limitations. The job was terminated after 1 hour of elapsed time.
Optimization II
The prior test exposed a significant discrepancy between backup speed for the VM’s on the two ESXi hosts.
The primary processing limiting factor was that half the VM’s were being processed with network (NBD)
transport mode. Simply stated the VM’s not local to the physical server hosting the VBR VM were processed
over the network while VM’s co-located on the same physical ESXi host as VBR are able to leverage the
disk hotadd transport rather than reading source VM information over the network.
To mitigate this issue we could simply continue to scale VBR up (additional tasks) however to demonstrate
Veeam’s modularity / scale out capabilities and to take full advantage of the hotadd transport, the proxy
role will be delegated to two new Linux VM’s distributed across both hosts.
For the test two 2 Ubuntu Linux VM’s (4 vCPU, 8GB RAM) were deployed. By disabling the VMware proxy
role from the VBR server all the proxy processing will be effectively offloaded to the two new Linux VM’s
resulting in the test environment configuration, shown in Figure 12.

Figure 12. Scale Out Proxy Compute to Linux VM's per Host
Both Linux VM’s were added to the VBR console as managed hosts and then each were assigned a proxy
role. The prior test assigned 8 total proxy tasks to the VBR-hosted proxy and each new proxy will be
assigned 4 tasks to maintain consistency from the prior test. This change yielded nominal improvement
from roughly 320MB/s in the prior test to a sustained 400MB/s+ (charts) for this iteration. The job was
terminated after roughly one hour.
Optimization III
No substantial bottleneck has yet surfaced (below) indicating the underlying compute, networking and
storage infrastructure can handle higher load, as shown in Figure 13.

Figure 13. VBR Bottleneck Analysis
To double the task parallelism, the Linux proxy VM specifications were increased to 8 vCPU x 16GB RAM,
with each proxy instance being assigned 8 task slots. VBR repository task count was also increased to 16.
After making the above changes only 8 concurrent tasks continued to be observed. Note that the Veeam
default maximum number of VM snapshots per VMware datastore is 4 hence we were still processing 4
VM’s concurrently per proxy/HDD datastore, as shown in Figure 14.
Figure 14 Default datastore snapshot limit indicator
To override the default limit a VBR server registry modification must be implemented. To increase the
number of datastore snapshots allowed create a DWORD value “MaxSnapshotsPerDatastore” in
HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication. The limit was arbitrarily
increased to 16 to prevent the number of datastore snapshots from limiting concurrency. This allowed the
job to run at a full 16 concurrent tasks.
Even though the task concurrency was doubled the results were similar (charts) to the prior testing scenario
(for this test run the backup job was allowed to run to completion hence the lower average MB/s as the job
processed progressively fewer tasks as the job wound down).
The Veeam bottleneck analysis emphatically points to the source storage as the primary choke point, as
shown in Figure 15.

Figure 15. Optimization III VBR Bottleneck Analysis
Optimization IV
As previously indicated an SSD-backed datastore exists on each physical ESXi host in our test environment
so to manually “load balance” the VM distribution across the two datastores (HDD, SSD) per host, 6 (all the
available SSD storage would accommodate) of the 15 VM’s on each host were migrated to the host’s local
SSD datastore.
The “MaxSnapShotsPerDatastore” registry value was reduced to 6 to better “load balance” the distribution
of backups across all datastores. Each Linux proxy was scaled up to 16 vCPU x 32 GB RAM and tasks
were increased to 12 per proxy. The VBR server was similarly scaled to 24 vCPU x 96 GB RAM and the
repository tasks were increased to 24.
As the job streams increased, 660MB/s+ performance was observed roughly halfway through the job,
however, as additional streams were added, diminishing returns were observed as Windows-reported disk
latency increased dramatically (see below). Note the DE Series storage continued to report low (< 15ms)
and 70%+ “headroom” during the job (see Figure 16).

Figure 16. VBR Repository Disk Usage
While certainly not saturated, network usage was frequently spiking to the full 10Gbps on the VBR server
VM (see Figure 17).

Figure 17. VBR Server Network Traffic
Overall performance significantly improved however from the roughly 400MB/s in the prior two scenarios to
well over 600MB/s sustained, 531MB/s overall average in this iteration (charts).
Optimization V
Veeam job analysis from the prior scenario clearly points to the backup storage processing as the primary
bottleneck. This is likely due to VMware / Windows in-guest iSCSI environmental limitations as no
limitations were observed on the DE Series storage. As processing parallelism looks to have reached its
limits on the backup source side, backup target processing can now scale with Veeam “Scale-out Backup
Repository” SOBR in combination with the replacement of the DE Series disk pool with more granularly
controlled volume groups. To this point, all our backup storage communication has flowed exclusively from
the VBR server VM to the DE Series Controller A which owns the storage pool (see Figure 18).

Figure 18. DE Series Controller Traffic
To attempt to alleviate the storage/target end bottleneck and in accordance with previously stated best
practices, our DE Series repository will be scrapped for a new volume-based RAID 6 rather than disk pool-
based LUN. A new volume group will be created with two volumes split across the two DE Series controllers.
A new Veeam repository-only server will be created on one of the ESXi hosts with the VBR server retaining
its repository compute role as well (though this could of course be relocated to a separate server). The new
repository server will attach the volume from Controller A and the VBR host will attach the volume from
Controller B. These two new individual repositories will be aggregated into a Veeam SOBR and our backup
job will be reconfigured to point to the SOBR for backup storage.
With the above changes, our new configuration is shown in Figures 19, 20 and 21.

Figure 19. Scale-out Repository

Figure 20 DE Series Volume Group
Figure 21. DE Series Repository Volumes
As seen below these changes have more evenly distributed the repository write load and effectively reach
the stated 1GB/s backup processing goal, as shown in Figure 22.

Figure 22. Backup Job statistics
Additionally, both DE Series controllers are now actively processing backups (see Figure 23).

Figure 23. DE Series Physical View
While the VBR job processing average did not ever exceed 1GB/s, >1GiB/s was sustained (chart) to the
DE Series backup repository.
Performance optimizations could have also been applied at the networking layer, e.g. NIC teaming, jumbo
frames, etc. however this infrastructure layer did not ever appear to be a primary performance impediment.

Ultimately, the primary bottleneck was lack of available read IOPS to the VMware datastores, however, the
1GB/s goal was achieved.
7 Appendix - Lab Test Statistics

Default Configuration


Optimization I


Optimization II


Optimization III


Optimization IV


Optimization V


LENOVO PROVIDES THIS PUBLICATION “AS IS” WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED,
INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS
FOR A PARTICULAR PURPOSE. This information could include technical inaccuracies or typographical errors. Changes may be
made to the information herein; these changes will be incorporated in new editions of the publication. Lenovo may make improvements
and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice.
Any performance data contained herein was determined in a controlled environment; therefore, the results obtained in other operating
environments may vary significantly. Some measurements may have been made on development-level systems, and there is no
guarantee that these measurements will be the same on generally available systems. Furthermore, some measurements may have
been estimated through extrapolation. Actual results may vary. Users of this document should verify the applicable data for their
specific environment.
Any references in this publication to non-Lenovo Web sites are provided for convenience only and do not in any manner serve as an
endorsement of those Web sites. The materials at those Web sites are not part of the materials for this Lenovo product, and use of
those Web sites is at your own risk.
Lenovo, the Lenovo logo, System x, and ThinkServer, are trademarks of Lenovo in the United States, other countries, or both.
Veeam is a trademark of Veeam Software in the United State, other countries, or both.
Microsoft, Windows Storage Server 2012, Windows Server 2012, and the Windows Logo are trademarks of Microsoft Corporation in
the United States, other countries, or both.

Lenovo de Series Storage Best Practices With Veeam Backup Replication

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Lenovo de Series Storage Best Practices With Veeam Backup Replication

Uploaded by

Copyright:

Available Formats

Lenovo ThinkSystem DE Series Storage Best

Practices with Veeam Backup & Replication 11

© 2021 Lenovo. All Rights Reserved. 2

2.1 DE Series as Veeam Backup & Replication Repository

© 2021 Lenovo. All Rights Reserved. 3

3.1 Capacity Planning

• RPS tool (simple) - http://rps.dewin.me/

© 2021 Lenovo. All Rights Reserved. 4

Figure 2. RPS storage estimation example

© 2021 Lenovo. All Rights Reserved. 5

Figure 3. Veeam Compute Components

© 2021 Lenovo. All Rights Reserved. 6

© 2021 Lenovo. All Rights Reserved. 7

Figure 4. VBR Single Server “appliance” Architecture

© 2021 Lenovo. All Rights Reserved. 8

Figure 5. Combo Server Scale Out

© 2021 Lenovo. All Rights Reserved. 9

Figure 6. Virtualized Scale Out Proxy

VBR Server Placement

© 2021 Lenovo. All Rights Reserved. 10

Repository Server Placement

© 2021 Lenovo. All Rights Reserved. 11

Figure 7. SOBR Storage Scale-out

For further treatment of SOBR capabilities, refer to the following documentation:

© 2021 Lenovo. All Rights Reserved. 12

Figure 8. VBR / DE Series Architecture for Optimized Incremental Backup

© 2021 Lenovo. All Rights Reserved. 13

o Local Target (large blocks) – 4MB

3.6 Backup Resiliency

© 2021 Lenovo. All Rights Reserved. 14

Figure 9. DE Series Backup Immutability with Veeam Hardened Linux Repository

• Use RAID 6 (8+2) volume groups.

© 2021 Lenovo. All Rights Reserved. 15

• Run multiple backup jobs to each repository.

4.2 Host Connectivity

Host Connectivity for DE2000H, DE4000H and DE4000F

• Dual Ethernet ports for management-related activities

Host Connectivity for DE6000H and DE6000F

• Dual Ethernet ports for management-related activities

© 2021 Lenovo. All Rights Reserved. 16

Controller Optional Host I/O

DE2000H, DE4000H & 2-port 10Gb iSCSI (Base-T) per controller

4-port 12Gb SAS (wide-port): uses mini-SAS cables

© 2021 Lenovo. All Rights Reserved. 17

5 Veeam Backup & Replication 11

5.2 Backup Server

© 2021 Lenovo. All Rights Reserved. 18

6.1 Lab Hardware

6.2 Lab Software

6.3 Lab Environment Configuration

© 2021 Lenovo. All Rights Reserved. 19

ies/block.html#ntfs-or-refs). This storage is configured as a standalone backup repository extent leveraging

Initial configuration is shown in Figure 10.

Figure 10. Initial Test Environment Configuration

© 2021 Lenovo. All Rights Reserved. 20

• Available parallel processing capacity

Figure 11. Proxy Task Setting

© 2021 Lenovo. All Rights Reserved. 21

© 2021 Lenovo. All Rights Reserved. 22

© 2021 Lenovo. All Rights Reserved. 23

Figure 13. VBR Bottleneck Analysis

Figure 14 Default datastore snapshot limit indicator

© 2021 Lenovo. All Rights Reserved. 24