Openshift Container Platform 4.15 Installing en Us

OpenShift Container Platform 4.
15
Installing
Installing and configuring OpenShift Container Platform clusters
Last Updated: 2024-04-25

OpenShift Container Platform 4.15 Installing
Installing and configuring OpenShift Container Platform clusters
Legal Notice
Copyright © 2024 Red Hat, Inc.
The text of and illustrations in this document are licensed by Red Hat under a Creative Commons
Attribution–Share Alike 3.0 Unported license ("CC-BY-SA"). An explanation of CC-BY-SA is
available at
http://creativecommons.org/licenses/by-sa/3.0/
. In accordance with CC-BY-SA, if you distribute this document or an adaptation of it, you must
provide the URL for the original version.
Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert,
Section 4d of CC-BY-SA to the fullest extent permitted by applicable law.
Red Hat, Red Hat Enterprise Linux, the Shadowman logo, the Red Hat logo, JBoss, OpenShift,
Fedora, the Infinity logo, and RHCE are trademarks of Red Hat, Inc., registered in the United States
and other countries.
Linux ® is the registered trademark of Linus Torvalds in the United States and other countries.
Java ® is a registered trademark of Oracle and/or its affiliates.
XFS ® is a trademark of Silicon Graphics International Corp. or its subsidiaries in the United States
and/or other countries.
MySQL ® is a registered trademark of MySQL AB in the United States, the European Union and
other countries.
Node.js ® is an official trademark of Joyent. Red Hat is not formally related to or endorsed by the
official Joyent Node.js open source or commercial project.
The OpenStack ® Word Mark and OpenStack logo are either registered trademarks/service marks
or trademarks/service marks of the OpenStack Foundation, in the United States and other
countries and are used with the OpenStack Foundation's permission. We are not affiliated with,
endorsed or sponsored by the OpenStack Foundation, or the OpenStack community.
All other trademarks are the property of their respective owners.
Abstract
This document provides information about installing OpenShift Container Platform and details
about some configuration processes.
Table of Contents
Table of Contents
.CHAPTER
. . . . . . . . . . 1.. .OPENSHIFT
. . . . . . . . . . . . .CONTAINER
. . . . . . . . . . . . .PLATFORM
. . . . . . . . . . . . .INSTALLATION
. . . . . . . . . . . . . . . .OVERVIEW
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .73
..............
1.1. ABOUT OPENSHIFT CONTAINER PLATFORM INSTALLATION 73
1.1.1. About the installation program 73
1.1.2. About Red Hat Enterprise Linux CoreOS (RHCOS) 74
1.1.3. Glossary of common terms for OpenShift Container Platform installing 74
1.1.4. Installation process 76
The installation process with the Assisted Installer 77
The installation process with Agent-based infrastructure 77
The installation process with installer-provisioned infrastructure 77
The installation process with user-provisioned infrastructure 78
Installation process details 78
1.1.5. Verifying node state after installation 80
Installation scope 81
1.1.6. OpenShift Local overview 81
1.2. SUPPORTED PLATFORMS FOR OPENSHIFT CONTAINER PLATFORM CLUSTERS 82
.CHAPTER
. . . . . . . . . . 2.
. . SELECTING
. . . . . . . . . . . . .A
. . CLUSTER
. . . . . . . . . . .INSTALLATION
. . . . . . . . . . . . . . . .METHOD
. . . . . . . . . .AND
. . . . .PREPARING
. . . . . . . . . . . . .IT
. . .FOR
. . . . USERS
. . . . . . . . . . . . . . . . . .85
..............
2.1. SELECTING A CLUSTER INSTALLATION TYPE 85
2.1.1. Do you want to install and manage an OpenShift Container Platform cluster yourself? 85
2.1.2. Have you used OpenShift Container Platform 3 and want to use OpenShift Container Platform 4? 86
2.1.3. Do you want to use existing components in your cluster? 86
2.1.4. Do you need extra security for your cluster? 87
2.2. PREPARING YOUR CLUSTER FOR USERS AFTER INSTALLATION 87
2.3. PREPARING YOUR CLUSTER FOR WORKLOADS 87
2.4. SUPPORTED INSTALLATION METHODS FOR DIFFERENT PLATFORMS 88
. . . . . . . . . . . 3.
CHAPTER . . CLUSTER
. . . . . . . . . . .CAPABILITIES
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .94
..............
3.1. SELECTING CLUSTER CAPABILITIES 94
3.2. OPTIONAL CLUSTER CAPABILITIES IN OPENSHIFT CONTAINER PLATFORM 4.15 96
3.2.1. Bare-metal capability 96
Purpose 96
3.2.2. Build capability 97
Purpose 97
3.2.3. Cloud credential capability 97
Purpose 97
3.2.4. Cluster storage capability 97
Purpose 97
Notes 98
3.2.5. Console capability 98
Purpose 98
3.2.6. CSI snapshot controller capability 98
Purpose 98
3.2.7. DeploymentConfig capability 98
Purpose 98
3.2.8. Insights capability 99
Purpose 99
Notes 99
3.2.9. Machine API capability 99
Purpose 99
3.2.10. Marketplace capability 99
Purpose 99
1
3.2.11. Operator Lifecycle Manager capability 100

Purpose 100
3.2.12. Node Tuning capability 100
Purpose 100
3.2.13. OpenShift samples capability 101
Purpose 101
3.2.14. Cluster Image Registry capability 101
Purpose 101
Project 102
3.3. ADDITIONAL RESOURCES 102
.CHAPTER
. . . . . . . . . . 4.
. . .DISCONNECTED
. . . . . . . . . . . . . . . . . INSTALLATION
. . . . . . . . . . . . . . . . .MIRRORING
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .103
...............
4.1. ABOUT DISCONNECTED INSTALLATION MIRRORING 103
4.1.1. Creating a mirror registry 103
4.1.2. Mirroring images for a disconnected installation 103
4.2. CREATING A MIRROR REGISTRY WITH MIRROR REGISTRY FOR RED HAT OPENSHIFT 103
4.2.1. Prerequisites 103
4.2.2. Mirror registry for Red Hat OpenShift introduction 104
4.2.2.1. Mirror registry for Red Hat OpenShift limitations 104
4.2.3. Mirroring on a local host with mirror registry for Red Hat OpenShift 105
4.2.4. Updating mirror registry for Red Hat OpenShift from a local host 106
4.2.5. Mirroring on a remote host with mirror registry for Red Hat OpenShift 107
4.2.6. Updating mirror registry for Red Hat OpenShift from a remote host 108
4.2.7. Replacing mirror registry for Red Hat OpenShift SSL/TLS certificates 108
4.2.8. Uninstalling the mirror registry for Red Hat OpenShift 109
4.2.9. Mirror registry for Red Hat OpenShift flags 110
4.2.10. Mirror registry for Red Hat OpenShift release notes 111
4.2.10.1. Mirror registry for Red Hat OpenShift 1.3.11 111
4.2.10.12.1. New features 113
4.2.10.12.2. Bug fixes 114
4.2.10.15.1. Bug fixes 115
4.2.10.16.1. New features 115
2
Table of Contents
4.2.10.22.1. Bug fixes 116

4.2.10.23.1. New features 116
4.2.10.23.2. Bug fixes 116
4.2.11. Troubleshooting mirror registry for Red Hat OpenShift 116
4.2.12. Additional resources 117
4.3. MIRRORING IMAGES FOR A DISCONNECTED INSTALLATION 117
4.3.2. About the mirror registry 118
4.3.3. Preparing your mirror host 119
4.3.3.1. Installing the OpenShift CLI by downloading the binary 119
Installing the OpenShift CLI on Linux 119
Installing the OpenShift CLI on Windows 120
Installing the OpenShift CLI on macOS 120
4.3.4. Configuring credentials that allow images to be mirrored 121
4.3.5. Mirroring the OpenShift Container Platform image repository 123
4.3.6. The Cluster Samples Operator in a disconnected environment 126
4.3.6.1. Cluster Samples Operator assistance for mirroring 126
4.3.7. Mirroring Operator catalogs for use with disconnected clusters 127
4.3.7.1. Prerequisites 128
4.3.7.2. Extracting and mirroring catalog contents 128
4.3.7.2.1. Mirroring catalog contents to registries on the same network 128
4.3.7.2.2. Mirroring catalog contents to airgapped registries 130
4.3.7.3. Generated manifests 132
4.3.7.4. Postinstallation requirements 133
4.3.8. Next steps 133
4.4. MIRRORING IMAGES FOR A DISCONNECTED INSTALLATION USING THE OC-MIRROR PLUGIN 134
4.4.1. About the oc-mirror plugin 134
4.4.2. oc-mirror compatibility and support 135
4.4.3. About the mirror registry 135
4.4.5. Preparing your mirror hosts 136
4.4.5.1. Installing the oc-mirror OpenShift CLI plugin 137
4.4.5.2. Configuring credentials that allow images to be mirrored 137
4.4.6. Creating the image set configuration 140
4.4.7. Mirroring an image set to a mirror registry 142
4.4.7.1. Mirroring an image set in a partially disconnected environment 142
4.4.7.1.1. Mirroring from mirror to mirror 142
4.4.7.2. Mirroring an image set in a fully disconnected environment 143
4.4.7.2.1. Mirroring from mirror to disk 143
4.4.7.2.2. Mirroring from disk to mirror 145
4.4.8. Configuring your cluster to use the resources generated by oc-mirror 145
4.4.9. Keeping your mirror registry content updated 146
4.4.9.1. About updating your mirror registry content 147
Adding new and updated images 147
Pruning images 147
4.4.9.2. Updating your mirror registry content 148
4.4.10. Performing a dry run 149
4.4.11. Including local OCI Operator catalogs 150
4.4.12. Image set configuration parameters 152
4.4.13. Image set configuration examples 159
Use case: Including the shortest OpenShift Container Platform update path 159
3
Use case: Including all versions of OpenShift Container Platform from a minimum to the latest version for
multi-architecture releases 159
Use case: Including Operator versions from a minimum to the latest 160
Use case: Including the Nutanix CSI Operator 161
Use case: Including the default Operator channel 161
Use case: Including an entire catalog (all versions) 162
Use case: Including an entire catalog (channel heads only) 162
Use case: Including arbitrary images and helm charts 162
4.4.14. Command reference for oc-mirror 163
. . . . . . . . . . . 5.
CHAPTER . . INSTALLING
. . . . . . . . . . . . . .ON
. . . .ALIBABA
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .166
...............
5.1. PREPARING TO INSTALL ON ALIBABA CLOUD 166
5.1.2. Requirements for installing OpenShift Container Platform on Alibaba Cloud 166
5.1.3. Registering and Configuring Alibaba Cloud Domain 166
5.1.4. Supported Alibaba regions 167
5.2. CREATING THE REQUIRED ALIBABA CLOUD RESOURCES 167
5.2.1. Creating the required RAM user 167
5.2.2. Configuring the Cloud Credential Operator utility 172
5.3. INSTALLING A CLUSTER QUICKLY ON ALIBABA CLOUD 173
5.3.2. Internet access for OpenShift Container Platform 174
5.3.3. Generating a key pair for cluster node SSH access 174
5.3.4. Obtaining the installation program 176
5.3.5. Creating the installation configuration file 177
5.3.6. Generating the required installation manifests 179
5.3.7. Creating credentials for OpenShift Container Platform components with the ccoctl tool 179
5.3.8. Deploying the cluster 181
5.3.9. Installing the OpenShift CLI by downloading the binary 183
5.3.10. Logging in to the cluster by using the CLI 185
5.3.11. Logging in to the cluster by using the web console 185
5.3.12. Telemetry access for OpenShift Container Platform 186
5.4. INSTALLING A CLUSTER ON ALIBABA CLOUD WITH CUSTOMIZATIONS 186
5.4.4.1. Creating the installation configuration file 190
5.4.4.2. Generating the required installation manifests 192
5.4.4.3. Creating credentials for OpenShift Container Platform components with the ccoctl tool 192
5.4.4.4. Sample customized install-config.yaml file for Alibaba Cloud 195
5.4.4.5. Configuring the cluster-wide proxy during installation 196
4
Table of Contents

5.5. INSTALLING A CLUSTER ON ALIBABA CLOUD WITH NETWORK CUSTOMIZATIONS 203
5.5.5. Network configuration phases 206
5.5.6. Cluster Network Operator configuration 212
5.5.6.1. Cluster Network Operator configuration object 212
defaultNetwork object configuration 213
Configuration for the OVN-Kubernetes network plugin 214
kubeProxyConfig object configuration (OpenShiftSDN container network interface only) 219
5.5.7. Specifying advanced network configuration 220
5.5.8. Configuring hybrid networking with OVN-Kubernetes 221
5.6. INSTALLING A CLUSTER ON ALIBABA CLOUD INTO AN EXISTING VPC 227
5.6.2. Using a custom VPC 228
5.6.2.1. Requirements for using your VPC 228
5.6.2.2. VPC validation 229
5.6.2.3. Division of permissions 229
5.6.2.4. Isolation between clusters 229
5.6.5.4. Configuring the Cloud Credential Operator utility 235
5.6.5.5. Creating credentials for OpenShift Container Platform components with the ccoctl tool 237
5

5.7. INSTALLATION CONFIGURATION PARAMETERS FOR ALIBABA CLOUD 244
5.7.1. Available installation configuration parameters for Alibaba Cloud 245
5.7.1.1. Required configuration parameters 245
5.7.1.2. Network configuration parameters 246
5.7.1.3. Optional configuration parameters 248
5.7.1.4. Additional Alibaba Cloud configuration parameters 254
5.8. UNINSTALLING A CLUSTER ON ALIBABA CLOUD 259
5.8.1. Removing a cluster that uses installer-provisioned infrastructure 259
. . . . . . . . . . . 6.
CHAPTER . . .INSTALLING
. . . . . . . . . . . . .ON
. . . .AWS
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261
...............
6.1. PREPARING TO INSTALL ON AWS 261
6.1.2. Requirements for installing OpenShift Container Platform on AWS 261
6.1.3. Choosing a method to install OpenShift Container Platform on AWS 261
6.1.3.1. Installing a cluster on a single node 261
6.1.3.2. Installing a cluster on installer-provisioned infrastructure 261
6.1.3.3. Installing a cluster on user-provisioned infrastructure 262
6.2. CONFIGURING AN AWS ACCOUNT 262
6.2.1. Configuring Route 53 263
6.2.1.1. Ingress Operator endpoint configuration for AWS Route 53 263
6.2.2. AWS account limits 264
6.2.3. Required AWS permissions for the IAM user 266
6.2.4. Creating an IAM user 274
6.2.5. IAM Policies and AWS authentication 275
6.2.5.1. Default permissions for IAM instance profiles 276
6.2.5.2. Specifying an existing IAM role 277
6.2.5.3. Using AWS IAM Analyzer to create policy templates 278
6.2.6. Supported AWS Marketplace regions 279
6.2.7. Supported AWS regions 279
6.2.7.1. AWS public regions 279
6.2.7.2. AWS GovCloud regions 280
6.2.7.3. AWS SC2S and C2S secret regions 280
6.2.7.4. AWS China regions 280
6.3. INSTALLING A CLUSTER QUICKLY ON AWS 281
6.4. INSTALLING A CLUSTER ON AWS WITH CUSTOMIZATIONS 291
6
Table of Contents

6.4.4. Obtaining an AWS Marketplace image 293
6.4.6.1. Minimum resource requirements for cluster installation 297
6.4.6.2. Tested instance types for AWS 297
6.4.6.3. Tested instance types for AWS on 64-bit ARM infrastructures 298
6.4.6.4. Sample customized install-config.yaml file for AWS 298
6.4.8. Alternatives to storing administrator-level secrets in the kube-system project 305
6.4.8.1. Manually creating long-term credentials 305
6.4.8.2. Configuring an AWS cluster to use short-term credentials 307
6.4.8.2.1. Configuring the Cloud Credential Operator utility 307
6.4.8.2.2. Creating AWS resources with the Cloud Credential Operator utility 310
6.4.8.2.2.1. Creating AWS resources with a single command 311
6.4.8.2.2.2. Creating AWS resources individually 313
6.4.8.2.3. Incorporating the Cloud Credential Operator utility manifests 315
6.5. INSTALLING A CLUSTER ON AWS WITH NETWORK CUSTOMIZATIONS 320
7

6.5.11. Configuring an Ingress Controller Network Load Balancer on a new AWS cluster 353
6.6. INSTALLING A CLUSTER ON AWS IN A RESTRICTED NETWORK 359
6.6.2. About installations in restricted networks 360
6.6.2.1. Additional limits 360
6.6.3. About using a custom VPC 361
Option 1: Create VPC endpoints 362
Option 2: Create a proxy without VPC endpoints 362
Option 3: Create a proxy with VPC endpoints 362
6.6.11. Disabling the default OperatorHub catalog sources 390
6.7. INSTALLING A CLUSTER ON AWS INTO AN EXISTING VPC 391
8
Table of Contents

6.7.2.5. Optional: AWS security groups 396
6.7.2.6. Modifying trust policy when installing into a shared VPC 397
6.7.6.6. Applying existing AWS security groups to the cluster 408
6.8. INSTALLING A PRIVATE CLUSTER ON AWS 426
6.8.2. Private clusters 426
6.8.2.1. Private clusters in AWS 427
6.8.2.1.1. Limitations 427
6.8.7. Manually creating the installation configuration file 435
9

6.9. INSTALLING A CLUSTER ON AWS INTO A GOVERNMENT REGION 460
6.9.2. AWS government regions 461
6.9.3. Installation requirements 461
10
Table of Contents

6.10. INSTALLING A CLUSTER ON AWS INTO A SECRET OR TOP SECRET REGION 496
6.10.2. AWS secret regions 497
6.10.7. Uploading a custom RHCOS AMI in AWS 504
6.10.17. Next steps 533
11
6.11. INSTALLING A CLUSTER ON AWS CHINA 533

6.11.7. Uploading a custom RHCOS AMI in AWS 542
6.11.16. Next steps 570
6.12. INSTALLING A CLUSTER ON USER-PROVISIONED INFRASTRUCTURE IN AWS BY USING
CLOUDFORMATION TEMPLATES 571
6.12.3. Requirements for a cluster with user-provisioned infrastructure 572
6.12.3.1. Required machines for cluster installation 572
6.12.3.5. Certificate signing requests management 575
12
Table of Contents
6.12.4. Required AWS infrastructure components 575

6.12.4.1. Other infrastructure components 576
6.12.4.2. Cluster machines 583
6.12.4.3. Required AWS permissions for the IAM user 584
6.12.8. Creating the installation files for AWS 595
6.12.8.1. Optional: Creating a separate /var partition 595
6.12.8.4. Creating the Kubernetes manifest and Ignition config files 601
6.12.9. Extracting the infrastructure name 603
6.12.10. Creating a VPC in AWS 604
6.12.10.1. CloudFormation template for the VPC 606
6.12.11. Creating networking and load balancing components in AWS 612
6.12.11.1. CloudFormation template for the network and load balancers 616
6.12.12. Creating security group and roles in AWS 624
6.12.12.1. CloudFormation template for security objects 626
6.12.13. Accessing RHCOS AMIs with stream metadata 637
6.12.14. RHCOS AMIs for the AWS infrastructure 638
6.12.14.1. AWS regions without a published RHCOS AMI 641
6.12.14.2. Uploading a custom RHCOS AMI in AWS 641
6.12.15. Creating the bootstrap node in AWS 643
6.12.15.1. CloudFormation template for the bootstrap machine 648
6.12.16. Creating the control plane machines in AWS 652
6.12.16.1. CloudFormation template for control plane machines 657
6.12.17. Creating the worker nodes in AWS 663
6.12.17.1. CloudFormation template for worker machines 666
6.12.18. Initializing the bootstrap sequence on AWS with user-provisioned infrastructure 668
6.12.21. Approving the certificate signing requests for your machines 672
6.12.22. Initial Operator configuration 674
6.12.22.1. Image registry storage configuration 675
6.12.22.1.1. Configuring registry storage for AWS with user-provisioned infrastructure 676
6.12.22.1.2. Configuring storage for the image registry in non-production clusters 677
6.12.23. Deleting the bootstrap resources 677
6.12.24. Creating the Ingress DNS Records 678
6.12.25. Completing an AWS installation on user-provisioned infrastructure 680
6.12.29. Next steps 683
6.13. INSTALLING A CLUSTER ON AWS IN A RESTRICTED NETWORK WITH USER-PROVISIONED
INFRASTRUCTURE 683
13

6.13.5. Required AWS infrastructure components 688
6.13.5.1. Other infrastructure components 688
6.13.5.2. Cluster machines 696
6.13.5.3. Required AWS permissions for the IAM user 696
6.13.7. Creating the installation files for AWS 706
6.13.8. Extracting the infrastructure name 715
6.13.9. Creating a VPC in AWS 716
6.13.10. Creating networking and load balancing components in AWS 724
6.13.10.1. CloudFormation template for the network and load balancers 727
6.13.11. Creating security group and roles in AWS 735
6.13.11.1. CloudFormation template for security objects 738
6.13.12. Accessing RHCOS AMIs with stream metadata 749
6.13.13. RHCOS AMIs for the AWS infrastructure 750
6.13.14. Creating the bootstrap node in AWS 753
6.13.14.1. CloudFormation template for the bootstrap machine 757
6.13.15. Creating the control plane machines in AWS 761
6.13.15.1. CloudFormation template for control plane machines 766
6.13.16. Creating the worker nodes in AWS 772
6.13.16.1. CloudFormation template for worker machines 775
6.13.17. Initializing the bootstrap sequence on AWS with user-provisioned infrastructure 777
6.13.20.1. Disabling the default OperatorHub catalog sources 782
6.13.20.2.1. Configuring registry storage for AWS with user-provisioned infrastructure 783
6.13.21. Deleting the bootstrap resources 784
6.13.22. Creating the Ingress DNS Records 785
6.13.23. Completing an AWS installation on user-provisioned infrastructure 787
6.13.27. Next steps 790
6.14. INSTALLING A CLUSTER ON AWS WITH COMPUTE NODES ON AWS LOCAL ZONES 790
14
Table of Contents
6.14.1. Infrastructure prerequisites 790

6.14.2. About AWS Local Zones and edge compute pool 791
6.14.2.1. Cluster limitations in AWS Local Zones 791
6.14.2.2. About edge compute pools 792
6.14.3. Installation prerequisites 793
6.14.3.1. Opting in to an AWS Local Zones 793
6.14.3.2. Internet access for OpenShift Container Platform 794
6.14.3.3. Obtaining an AWS Marketplace image 795
6.14.3.5. Obtaining the installation program 797
6.14.3.6. Generating a key pair for cluster node SSH access 798
6.14.4. Preparing for the installation 800
6.14.4.4. Examples of installation configuration files with edge compute pools 803
6.14.4.5. Customizing the cluster network MTU 804
6.14.5. Cluster installation options for an AWS Local Zones environment 805
6.14.6. Install a cluster quickly in AWS Local Zones 805
6.14.6.1. Modifying an installation configuration file to use AWS Local Zones 806
6.14.7. Installing a cluster in an existing VPC that has Local Zone subnets 807
6.14.7.1. Creating a VPC in AWS 808
6.14.7.3. Creating subnets in Local Zones 816
6.14.7.4. CloudFormation template for the VPC subnet 817
6.14.7.5. Modifying an installation configuration file to use AWS Local Zones subnets 819
6.14.8. Optional: AWS security groups 820
6.14.9. Optional: Assign public IP addresses to edge compute nodes 820
6.14.11. Verifying the status of the deployed cluster 823
6.14.11.1. Logging in to the cluster by using the CLI 823
6.14.11.2. Logging in to the cluster by using the web console 824
6.14.11.3. Verifying nodes that were created with edge compute pool 825
6.15. INSTALLING A CLUSTER ON AWS WITH COMPUTE NODES ON AWS WAVELENGTH ZONES 826
6.15.1. Infrastructure prerequisites 826
6.15.2. About AWS Wavelength Zones and edge compute pool 827
6.15.2.1. Cluster limitations in AWS Wavelength Zones 827
6.15.2.2. About edge compute pools 828
6.15.3. Installation prerequisites 829
6.15.3.1. Opting in to an AWS Wavelength Zones 829
6.15.3.3. Obtaining an AWS Marketplace image 831
6.15.4. Preparing for the installation 836
15

6.15.4.4. Examples of installation configuration files with edge compute pools 839
6.15.5. Cluster installation options for an AWS Wavelength Zones environment 839
6.15.6. Install a cluster quickly in AWS Wavelength Zones 840
6.15.6.1. Modifying an installation configuration file to use AWS Wavelength Zones 840
6.15.7. Installing a cluster in an existing VPC that has Wavelength Zone subnets 841
6.15.7.1. Creating a VPC in AWS 842
6.15.7.3. Creating a VPC carrier gateway 850
6.15.7.4. CloudFormation template for the VPC Carrier Gateway 852
6.15.7.5. Creating subnets in Wavelength Zones 853
6.15.7.6. CloudFormation template for the VPC subnet 855
6.15.7.7. Modifying an installation configuration file to use AWS Wavelength Zones subnets 857
6.15.8. Optional: Assign public IP addresses to edge compute nodes 857
6.15.10. Verifying the status of the deployed cluster 860
6.15.10.2. Logging in to the cluster by using the web console 861
6.15.10.3. Verifying nodes that were created with edge compute pool 862
6.16. INSTALLING A CLUSTER ON AWS WITH COMPUTE NODES ON AWS OUTPOSTS 863
6.17. INSTALLING A THREE-NODE CLUSTER ON AWS 863
6.17.1. Configuring a three-node cluster 864
6.18. UNINSTALLING A CLUSTER ON AWS 865
6.18.2. Deleting Amazon Web Services resources with the Cloud Credential Operator utility 865
6.18.3. Deleting a cluster with a configured AWS Local Zone infrastructure 867
6.19. INSTALLATION CONFIGURATION PARAMETERS FOR AWS 868
6.19.1. Available installation configuration parameters for AWS 868
6.19.1.4. Optional AWS configuration parameters 878
. . . . . . . . . . . 7.
CHAPTER . . INSTALLING
. . . . . . . . . . . . . .ON
. . . .AZURE
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 884
................
7.1. PREPARING TO INSTALL ON AZURE 884
7.1.2. Requirements for installing OpenShift Container Platform on Azure 884
7.1.3. Choosing a method to install OpenShift Container Platform on Azure 884
7.2. CONFIGURING AN AZURE ACCOUNT 885
7.2.1. Azure account limits 885
7.2.2. Configuring a public DNS zone in Azure 888
7.2.3. Increasing Azure account limits 888
7.2.4. Recording the subscription and tenant IDs 889
7.2.5. Supported identities to access Azure resources 891
7.2.5.1. Required Azure roles 891
7.2.5.2. Required Azure permissions for installer-provisioned infrastructure 891
16
Table of Contents
7.2.5.3. Using Azure managed identities 899

7.2.5.4. Creating a service principal 899
7.2.6. Supported Azure Marketplace regions 900
7.2.7. Supported Azure regions 901
Supported Azure public regions 901
Supported Azure Government regions 902
7.3. ENABLING USER-MANAGED ENCRYPTION FOR AZURE 902
7.3.1. Preparing an Azure Disk Encryption Set 903
7.4. INSTALLING A CLUSTER QUICKLY ON AZURE 905
7.5. INSTALLING A CLUSTER ON AZURE WITH CUSTOMIZATIONS 914
7.5.4. Using the Azure Marketplace offering 916
7.5.6.2. Tested instance types for Azure 923
7.5.6.3. Tested instance types for Azure on 64-bit ARM infrastructures 923
7.5.6.4. Enabling trusted launch for Azure VMs 924
7.5.6.5. Enabling confidential VMs 924
7.5.6.6. Sample customized install-config.yaml file for Azure 926
7.5.7. Configuring user-defined tags for Azure 930
7.5.8. Querying user-defined tags for Azure 932
7.5.10.2. Configuring an Azure cluster to use short-term credentials 937
7.5.10.2.2. Creating Azure resources with the Cloud Credential Operator utility 939
17
7.6. INSTALLING A CLUSTER ON AZURE WITH NETWORK CUSTOMIZATIONS 945

7.7. INSTALLING A CLUSTER ON AZURE INTO AN EXISTING VNET 982
7.7.2. About reusing a VNet for your OpenShift Container Platform cluster 982
7.7.2.1. Requirements for using your VNet 982
7.7.2.1.1. Network security group requirements 984
18
Table of Contents

7.7.11. Next steps 1010
7.8. INSTALLING A PRIVATE CLUSTER ON AZURE 1010
7.8.2.1. Private clusters in Azure 1011
7.8.2.2. User-defined outbound routing 1012
Private cluster with network address translation 1013
Private cluster with Azure Firewall 1013
Private cluster with a proxy configuration 1013
Private cluster with no internet access 1013
7.8.10. Optional: Preparing a private Microsoft Azure cluster for a private image registry 1039
19
7.8.14. Next steps 1043

7.9. INSTALLING A CLUSTER ON AZURE INTO A GOVERNMENT REGION 1044
7.9.2. Azure government regions 1044
7.9.3.1. Private clusters in Azure 1045
7.9.13. Next steps 1067
7.10. INSTALLING A CLUSTER ON AZURE IN A RESTRICTED NETWORK WITH USER-PROVISIONED
INFRASTRUCTURE 1067
7.10.2. Configuring your Azure project 1069
7.10.2.1. Azure account limits 1069
7.10.2.2. Configuring a public DNS zone in Azure 1072
7.10.2.3. Increasing Azure account limits 1072
7.10.2.5. Required Azure roles 1073
7.10.2.6. Required Azure permissions for user-provisioned infrastructure 1074
7.10.2.8. Supported Azure regions 1082
20
Table of Contents

7.10.5. Creating the installation files for Azure 1092
7.10.5.4. Exporting common variables for ARM templates 1099
7.10.6. Creating the Azure resource group 1103
7.10.7. Uploading the RHCOS cluster image and bootstrap Ignition config file 1104
7.10.8. Example for creating DNS zones 1106
7.10.9. Creating a VNet in Azure 1106
7.10.9.1. ARM template for the VNet 1107
7.10.10. Deploying the RHCOS cluster image for the Azure infrastructure 1109
7.10.10.1. ARM template for image storage 1110
7.10.11. Networking requirements for user-provisioned infrastructure 1113
7.10.11.1. Network connectivity requirements 1113
7.10.12. Creating networking and load balancing components in Azure 1114
7.10.12.1. ARM template for the network and load balancers 1115
7.10.13. Creating the bootstrap machine in Azure 1120
7.10.13.1. ARM template for the bootstrap machine 1121
7.10.14. Creating the control plane machines in Azure 1125
7.10.14.1. ARM template for control plane machines 1126
7.10.15. Wait for bootstrap completion and remove bootstrap resources in Azure 1130
7.10.16. Creating additional worker machines in Azure 1131
7.10.16.1. ARM template for worker machines 1132
7.10.20. Adding the Ingress DNS records 1140
7.10.21. Completing an Azure installation on user-provisioned infrastructure 1142
7.11. INSTALLING A CLUSTER ON AZURE USING ARM TEMPLATES 1143
7.11.3. Configuring your Azure project 1144
7.11.3.1. Azure account limits 1144
7.11.3.2. Configuring a public DNS zone in Azure 1147
7.11.3.3. Increasing Azure account limits 1147
7.11.3.5. Recording the subscription and tenant IDs 1148
7.11.3.6. Supported identities to access Azure resources 1150
7.11.3.7. Required Azure permissions for user-provisioned infrastructure 1150
7.11.3.8. Using Azure managed identities 1156
7.11.3.10. Supported Azure regions 1158
21

7.11.8. Creating the installation files for Azure 1167
7.11.12. Creating a VNet in Azure 1181
7.11.13. Deploying the RHCOS cluster image for the Azure infrastructure 1184
7.11.15. Creating networking and load balancing components in Azure 1189
7.11.16. Creating the bootstrap machine in Azure 1195
7.11.17. Creating the control plane machines in Azure 1200
7.11.18. Wait for bootstrap completion and remove bootstrap resources in Azure 1204
7.11.19. Creating additional worker machines in Azure 1205
7.11.24. Completing an Azure installation on user-provisioned infrastructure 1217
7.12. INSTALLING A CLUSTER ON AZURE IN A RESTRICTED NETWORK 1218
Restricted cluster with Azure Firewall 1219
22
Table of Contents

7.12.12. Next steps 1250
7.13. INSTALLING A THREE-NODE CLUSTER ON AZURE 1250
7.13.2. Next steps 1251
7.14. UNINSTALLING A CLUSTER ON AZURE 1251
7.14.2. Deleting Microsoft Azure resources with the Cloud Credential Operator utility 1252
7.15. INSTALLATION CONFIGURATION PARAMETERS FOR AZURE 1253
7.15.1. Available installation configuration parameters for Azure 1253
7.15.1.4. Additional Azure configuration parameters 1262
. . . . . . . . . . . 8.
. . . . . . . . . . . . .ON
. . . .AZURE
. . . . . . . STACK
. . . . . . . . HUB
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1280
................
8.1. PREPARING TO INSTALL ON AZURE STACK HUB 1280
8.1.2. Requirements for installing OpenShift Container Platform on Azure Stack Hub 1280
8.1.3. Choosing a method to install OpenShift Container Platform on Azure Stack Hub 1280
8.2. CONFIGURING AN AZURE STACK HUB ACCOUNT 1281
8.2.1. Azure Stack Hub account limits 1281
8.2.2. Configuring a DNS zone in Azure Stack Hub 1282
8.2.3. Required Azure Stack Hub roles 1283
8.2.4. Creating a service principal 1283
8.3. INSTALLING A CLUSTER ON AZURE STACK HUB WITH AN INSTALLER-PROVISIONED
INFRASTRUCTURE 1286
23
8.3.4. Uploading the RHCOS cluster image 1288

8.3.6.1. Sample customized install-config.yaml file for Azure Stack Hub 1291
8.3.7. Manually manage cloud credentials 1293
8.3.8. Configuring the cluster to use an internal CA 1295
8.3.14. Next steps 1300
8.4. INSTALLING A CLUSTER ON AZURE STACK HUB WITH NETWORK CUSTOMIZATIONS 1300
8.4.4. Uploading the RHCOS cluster image 1303
8.4.7. Manually manage cloud credentials 1308
8.4.8. Configuring the cluster to use an internal CA 1310
8.4.18. Next steps 1326
8.5. INSTALLING A CLUSTER ON AZURE STACK HUB USING ARM TEMPLATES 1326
8.5.3. Configuring your Azure Stack Hub project 1328
8.5.3.1. Azure Stack Hub account limits 1328
8.5.3.2. Configuring a DNS zone in Azure Stack Hub 1330
8.5.3.4. Required Azure Stack Hub roles 1330
24
Table of Contents
8.5.6. Creating the installation files for Azure Stack Hub 1335
8.5.6.1. Manually creating the installation configuration file 1335
8.5.10. Creating a VNet in Azure Stack Hub 1351
8.5.11. Deploying the RHCOS cluster image for the Azure Stack Hub infrastructure 1352
8.5.13. Creating networking and load balancing components in Azure Stack Hub 1354
8.5.14. Creating the bootstrap machine in Azure Stack Hub 1356
8.5.15. Creating the control plane machines in Azure Stack Hub 1358
8.5.16. Wait for bootstrap completion and remove bootstrap resources in Azure Stack Hub 1359
8.5.17. Creating additional worker machines in Azure Stack Hub 1360
8.5.22. Completing an Azure Stack Hub installation on user-provisioned infrastructure 1367
8.6. INSTALLATION CONFIGURATION PARAMETERS FOR AZURE STACK HUB 1368
8.6.1. Available installation configuration parameters for Azure Stack Hub 1368
8.6.1.4. Additional Azure Stack Hub configuration parameters 1377
8.7. UNINSTALLING A CLUSTER ON AZURE STACK HUB 1380
. . . . . . . . . . . 9.
. . . . . . . . . . . . .ON
. . . .GCP
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1382
................
9.1. PREPARING TO INSTALL ON GCP 1382
9.1.2. Requirements for installing OpenShift Container Platform on GCP 1382
9.1.3. Choosing a method to install OpenShift Container Platform on GCP 1382
9.2. CONFIGURING A GCP PROJECT 1383
9.2.1. Creating a GCP project 1383
9.2.2. Enabling API services in GCP 1384
25
9.2.3. Configuring DNS for GCP 1385

9.2.4. GCP account limits 1385
9.2.5. Creating a service account in GCP 1387
9.2.5.1. Required GCP roles 1388
9.2.5.2. Required GCP permissions for installer-provisioned infrastructure 1389
9.2.5.3. Required GCP permissions for shared VPC installations 1395
9.2.6. Supported GCP regions 1396
9.3. INSTALLING A CLUSTER QUICKLY ON GCP 1397
9.4. INSTALLING A CLUSTER ON GCP WITH CUSTOMIZATIONS 1406
9.4.5.2. Tested instance types for GCP 1412
9.4.5.3. Tested instance types for GCP on 64-bit ARM infrastructures 1412
9.4.5.4. Using custom machine types 1412
9.4.5.5. Enabling Shielded VMs 1413
9.4.5.6. Enabling Confidential VMs 1414
9.4.5.7. Sample customized install-config.yaml file for GCP 1415
9.4.6. Managing user-defined labels and tags for GCP 1420
9.4.6.1. Configuring user-defined labels and tags for GCP 1422
9.4.6.2. Querying user-defined labels and tags for GCP 1423
9.4.8.2. Configuring a GCP cluster to use short-term credentials 1428
9.4.8.2.2. Creating GCP resources with the Cloud Credential Operator utility 1431
9.4.9. Using the GCP Marketplace offering 1434
9.4.13. Next steps 1437
26
Table of Contents
9.5. INSTALLING A CLUSTER ON GCP WITH NETWORK CUSTOMIZATIONS 1438

9.5.15. Next steps 1472
9.6. INSTALLING A CLUSTER ON GCP IN A RESTRICTED NETWORK 1473
9.6.5.8. Create an Ingress Controller with global access on GCP 1486
27

9.6.12. Next steps 1502
9.7. INSTALLING A CLUSTER ON GCP INTO AN EXISTING VPC 1502
9.7.13. Next steps 1531
9.8. INSTALLING A CLUSTER ON GCP INTO A SHARED VPC 1531
28
Table of Contents

9.8.5. Creating the installation files for GCP 1534
9.8.5.4. Sample customized install-config.yaml file for shared VPC installation 1537
9.8.11. Next steps 1553
9.9. INSTALLING A PRIVATE CLUSTER ON GCP 1553
9.9.2.1. Private clusters in GCP 1554
29

9.9.14. Next steps 1583
9.10. INSTALLING A CLUSTER ON USER-PROVISIONED INFRASTRUCTURE IN GCP BY USING DEPLOYMENT
MANAGER TEMPLATES 1583
9.10.2. Certificate signing requests management 1584
9.10.4. Configuring your GCP project 1584
9.10.4.1. Creating a GCP project 1584
9.10.4.2. Enabling API services in GCP 1585
9.10.4.3. Configuring DNS for GCP 1586
9.10.4.4. GCP account limits 1586
9.10.4.5. Creating a service account in GCP 1588
9.10.4.7. Required GCP permissions for user-provisioned infrastructure 1590
9.10.4.8. Supported GCP regions 1597
9.10.4.9. Installing and configuring CLI tools for GCP 1599
9.10.7. Exporting common variables 1612
9.10.7.1. Extracting the infrastructure name 1612
9.10.7.2. Exporting common variables for Deployment Manager templates 1612
9.10.8. Creating a VPC in GCP 1613
9.10.8.1. Deployment Manager template for the VPC 1614
9.10.9.1. Setting the cluster node hostnames through DHCP 1615
9.10.10. Creating load balancers in GCP 1617
9.10.10.1. Deployment Manager template for the external load balancer 1618
9.10.10.2. Deployment Manager template for the internal load balancer 1619
9.10.11. Creating a private DNS zone in GCP 1621
9.10.11.1. Deployment Manager template for the private DNS 1622
9.10.12. Creating firewall rules in GCP 1623
9.10.12.1. Deployment Manager template for firewall rules 1624
9.10.13. Creating IAM roles in GCP 1626
9.10.13.1. Deployment Manager template for IAM roles 1628
9.10.14. Creating the RHCOS cluster image for the GCP infrastructure 1628
9.10.15. Creating the bootstrap machine in GCP 1629
9.10.15.1. Deployment Manager template for the bootstrap machine 1631
30
Table of Contents
9.10.16. Creating the control plane machines in GCP 1633

9.10.16.1. Deployment Manager template for control plane machines 1635
9.10.17. Wait for bootstrap completion and remove bootstrap resources in GCP 1637
9.10.18. Creating additional worker machines in GCP 1638
9.10.18.1. Deployment Manager template for worker machines 1640
9.10.22. Optional: Adding the ingress DNS records 1646
9.10.23. Completing a GCP installation on user-provisioned infrastructure 1648
9.10.25. Next steps 1650
9.11. INSTALLING A CLUSTER INTO A SHARED VPC ON GCP USING DEPLOYMENT MANAGER TEMPLATES
1650
9.11.2. Certificate signing requests management 1651
9.11.4. Configuring the GCP project that hosts your cluster 1652
9.11.4.4.1. Required GCP roles 1655
9.11.6. Configuring the GCP project that hosts your shared VPC network 1661
9.11.6.2. Creating a VPC in GCP 1662
9.11.6.2.1. Deployment Manager template for the VPC 1664
31

9.11.22. Adding the ingress DNS records 1708
9.11.23. Adding ingress firewall rules 1710
9.11.23.1. Creating cluster-wide firewall rules for a shared VPC in GCP 1710
9.11.26. Next steps 1714
9.12. INSTALLING A CLUSTER ON GCP IN A RESTRICTED NETWORK WITH USER-PROVISIONED
INFRASTRUCTURE 1714
9.12.4. Configuring your GCP project 1716
9.12.4.7. Required GCP permissions for user-provisioned infrastructure 1721
32
Table of Contents
9.12.8. Creating a VPC in GCP 1745
9.12.8.1. Deployment Manager template for the VPC 1746
9.12.22. Optional: Adding the ingress DNS records 1776
9.12.25. Next steps 1780
9.13. INSTALLING A THREE-NODE CLUSTER ON GCP 1780
9.13.2. Next steps 1782
9.14. INSTALLATION CONFIGURATION PARAMETERS FOR GCP 1782
9.14.1. Available installation configuration parameters for GCP 1782
9.14.1.4. Additional Google Cloud Platform (GCP) configuration parameters 1792
9.15. UNINSTALLING A CLUSTER ON GCP 1825
9.15.2. Deleting Google Cloud Platform resources with the Cloud Credential Operator utility 1826
. . . . . . . . . . . 10.
CHAPTER . . . INSTALLING
. . . . . . . . . . . . . .ON
. . . .IBM
. . . . CLOUD
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1828
................
10.1. PREPARING TO INSTALL ON IBM CLOUD 1828
10.1.2. Requirements for installing OpenShift Container Platform on IBM Cloud 1828
10.1.3. Choosing a method to install OpenShift Container Platform on IBM Cloud 1828
10.1.4. Next steps 1829
33
10.2. CONFIGURING AN IBM CLOUD ACCOUNT 1829

10.2.2. Quotas and limits on IBM Cloud 1829
Virtual Private Cloud (VPC) 1829
Application load balancer 1829
Floating IP address 1829
Virtual Server Instances (VSI) 1830
Block Storage Volumes 1830
10.2.3. Configuring DNS resolution 1831
10.2.3.1. Using IBM Cloud Internet Services for DNS resolution 1831
10.2.3.2. Using IBM Cloud DNS Services for DNS resolution 1832
10.2.4. IBM Cloud IAM Policies and API Key 1833
10.2.4.1. Required access policies 1833
10.2.4.2. Access policy assignment 1834
10.2.4.3. Creating an API key 1834
10.2.5. Supported IBM Cloud regions 1835
10.2.6. Next steps 1835
10.3. CONFIGURING IAM FOR IBM CLOUD 1835
10.3.3. Next steps 1837
10.4. USER-MANAGED ENCRYPTION FOR IBM CLOUD 1837
10.4.1. Next steps 1838
10.5. INSTALLING A CLUSTER ON IBM CLOUD WITH CUSTOMIZATIONS 1838
10.5.5. Exporting the API key 1842
10.5.6.2. Sample customized install-config.yaml file for IBM Cloud 1844
10.5.7. Manually creating IAM 1847
10.5.12. Next steps 1853
10.6. INSTALLING A CLUSTER ON IBM CLOUD WITH NETWORK CUSTOMIZATIONS 1854
34
Table of Contents

10.6.15. Next steps 1878
10.7. INSTALLING A CLUSTER ON IBM CLOUD INTO AN EXISTING VPC 1878
10.7.13. Next steps 1896
10.8. INSTALLING A PRIVATE CLUSTER ON IBM CLOUD 1896
10.8.3. Private clusters in IBM Cloud 1897
10.8.3.1. Limitations 1897
35

10.8.15. Next steps 1914
10.9. INSTALLING A CLUSTER ON IBM CLOUD IN A RESTRICTED NETWORK 1915
10.9.2.1. Required internet access and an installation host 1916
10.9.2.2. Access to a mirror registry 1916
10.9.2.3. Access to IBM service endpoints 1916
10.9.3.4. Allowing endpoint gateway traffic 1918
10.9.6. Downloading the RHCOS cluster image 1921
10.9.12. Post installation 1936
10.9.12.2. Installing the policy resources into the cluster 1936
10.9.14. Next steps 1937
10.10. INSTALLATION CONFIGURATION PARAMETERS FOR IBM CLOUD 1938
10.10.1. Available installation configuration parameters for IBM Cloud 1938
10.10.1.4. Additional IBM Cloud configuration parameters 1947
10.11. UNINSTALLING A CLUSTER ON IBM CLOUD 1953
. . . . . . . . . . . 11.
. . . . . . . . . . . . . ON
. . . .NUTANIX
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1955
................
36
Table of Contents
11.1. PREPARING TO INSTALL ON NUTANIX 1955

11.1.1. Nutanix version requirements 1955
11.1.2. Environment requirements 1955
11.1.2.1. Required account privileges 1955
11.1.2.2. Cluster limits 1957
11.1.2.3. Cluster resources 1957
11.1.2.4. Networking requirements 1957
11.1.2.4.1. Required IP Addresses 1958
11.1.2.4.2. DNS records 1958
11.2. FAULT TOLERANT DEPLOYMENTS USING MULTIPLE PRISM ELEMENTS 1960
11.2.1. Failure domain requirements 1960
11.2.2. Installation method and failure domain configuration 1960
11.3. INSTALLING A CLUSTER ON NUTANIX 1961
11.3.3. Internet access for Prism Central 1962
11.3.6. Adding Nutanix root CA certificates to your system trust 1965
11.3.7.1. Sample customized install-config.yaml file for Nutanix 1967
11.3.7.2. Configuring failure domains 1970
11.3.9. Configuring IAM for Nutanix 1975
11.3.10. Adding config map and secret resources required for Nutanix CCM 1978
11.3.12. Configuring the default storage container 1980
11.3.15. Next steps 1980
11.4. INSTALLING A CLUSTER ON NUTANIX IN A RESTRICTED NETWORK 1980
11.4.4. Adding Nutanix root CA certificates to your system trust 1984
11.4.5. Downloading the RHCOS cluster image 1984
11.4.6.1. Sample customized install-config.yaml file for Nutanix 1988
11.4.6.2. Configuring failure domains 1991
11.4.8. Configuring IAM for Nutanix 1996
11.4.10. Post installation 2000
37

11.4.10.2. Installing the policy resources into the cluster 2000
11.4.10.3. Configuring the default storage container 2001
11.4.13. Next steps 2002
11.5. INSTALLING A THREE-NODE CLUSTER ON NUTANIX 2002
11.5.2. Next steps 2002
11.6. UNINSTALLING A CLUSTER ON NUTANIX 2002
11.7. INSTALLATION CONFIGURATION PARAMETERS FOR NUTANIX 2003
11.7.1. Available installation configuration parameters for Nutanix 2003
11.7.1.4. Additional Nutanix configuration parameters 2012
. . . . . . . . . . . 12.
. . . . . . . . . . . . . .ON
. . . BARE
. . . . . . .METAL
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2018
................
12.1. PREPARING FOR BARE METAL CLUSTER INSTALLATION 2018
12.1.2. Planning a bare metal cluster for OpenShift Virtualization 2018
12.1.3. NIC partitioning for SR-IOV devices (Technology Preview) 2018
12.1.4. Choosing a method to install OpenShift Container Platform on bare metal 2019
12.2. INSTALLING A USER-PROVISIONED CLUSTER ON BARE METAL 2020
12.2.3.4. Requirements for baremetal clusters on vSphere 2024
12.2.3.5. Networking requirements for user-provisioned infrastructure 2024
12.2.3.5.1. Setting the cluster node hostnames through DHCP 2024
12.2.3.5.2. Network connectivity requirements 2024
NTP configuration for user-provisioned infrastructure 2026
12.2.3.6. User-provisioned DNS requirements 2026
12.2.3.6.1. Example DNS configuration for user-provisioned clusters 2028
12.2.3.7. Load balancing requirements for user-provisioned infrastructure 2030
12.2.3.7.1. Example load balancer configuration for user-provisioned clusters 2032
12.2.4. Preparing the user-provisioned infrastructure 2034
12.2.5. Validating DNS resolution for user-provisioned infrastructure 2036
12.2.9.1. Sample install-config.yaml file for bare metal 2044
38
Table of Contents
12.2.9.3. Configuring a three-node cluster 2048

12.2.10. Creating the Kubernetes manifest and Ignition config files 2049
12.2.11. Installing RHCOS and starting the OpenShift Container Platform bootstrap process 2051
12.2.11.1. Installing RHCOS by using an ISO image 2052
12.2.11.2. Installing RHCOS by using PXE or iPXE booting 2055
12.2.11.3. Advanced RHCOS installation configuration 2060
12.2.11.3.1. Using advanced networking options for PXE and ISO installations 2060
12.2.11.3.2. Disk partitioning 2061
12.2.11.3.2.1. Creating a separate /var partition 2062
12.2.11.3.2.2. Retaining existing partitions 2064
12.2.11.3.3. Identifying Ignition configs 2065
12.2.11.3.4. Default console configuration 2065
12.2.11.3.5. Enabling the serial console for PXE and ISO installations 2066
12.2.11.3.6. Customizing a live RHCOS ISO or PXE install 2066
12.2.11.3.7. Customizing a live RHCOS ISO image 2067
12.2.11.3.7.1. Modifying a live install ISO image to enable the serial console 2067
12.2.11.3.7.2. Modifying a live install ISO image to use a custom certificate authority 2068
12.2.11.3.7.3. Modifying a live install ISO image with customized network settings 2069
12.2.11.3.8. Customizing a live RHCOS PXE environment 2071
12.2.11.3.8.1. Modifying a live install PXE environment to enable the serial console 2071
12.2.11.3.8.2. Modifying a live install PXE environment to use a custom certificate authority 2072
12.2.11.3.8.3. Modifying a live install PXE environment with customized network settings 2072
12.2.11.3.9. Advanced RHCOS installation reference 2074
12.2.11.3.9.1. Networking and bonding options for ISO installations 2074
Configuring DHCP or static IP addresses 2075
Configuring an IP address without a static hostname 2075
Specifying multiple network interfaces 2076
Configuring default gateway and route 2076
Disabling DHCP on a single interface 2076
Combining DHCP and static IP configurations 2076
Configuring VLANs on individual interfaces 2076
Providing multiple DNS servers 2077
Bonding multiple network interfaces to a single interface 2077
Bonding multiple SR-IOV network interfaces to a dual port NIC interface 2077
Using network teaming 2078
12.2.11.3.9.2. coreos-installer options for ISO and PXE installations 2078
12.2.11.3.9.3. coreos.inst boot options for ISO or PXE installations 2083
12.2.11.4. Enabling multipathing with kernel arguments on RHCOS 2084
12.2.12. Waiting for the bootstrap process to complete 2086
12.2.15.1. Image registry removed during installation 2091
12.2.15.2.1. Configuring registry storage for bare metal and other manual installations 2092
12.2.15.2.3. Configuring block registry storage for bare metal 2094
12.2.16. Completing installation on user-provisioned infrastructure 2095
12.2.18. Next steps 2098
12.3. INSTALLING A USER-PROVISIONED BARE METAL CLUSTER WITH NETWORK CUSTOMIZATIONS
2098
39

12.3.13. Creating the Ignition config files 2133
40
Table of Contents

12.3.18.3. Configuring block registry storage for bare metal 2175
12.3.21. Next steps 2179
12.4. INSTALLING A USER-PROVISIONED BARE METAL CLUSTER ON A RESTRICTED NETWORK 2179
12.4.10. Configuring chrony time service 2208
41

12.4.15.2.1. Changing the image registry’s management state 2250
12.4.18. Next steps 2256
12.5. SCALING A USER-PROVISIONED CLUSTER WITH THE BARE METAL OPERATOR 2257
12.5.1. About scaling a user-provisioned cluster with the Bare Metal Operator 2257
12.5.1.1. Prerequisites for scaling a user-provisioned cluster 2257
12.5.1.2. Limitations for scaling a user-provisioned cluster 2257
12.5.2. Configuring a provisioning resource to scale user-provisioned clusters 2257
12.5.3. Provisioning new hosts in a user-provisioned cluster by using the BMO 2258
12.5.4. Optional: Managing existing hosts in a user-provisioned cluster by using the BMO 2261
42
Table of Contents
12.5.5. Removing hosts from a user-provisioned cluster by using the BMO 2262
12.6. INSTALLATION CONFIGURATION PARAMETERS FOR BARE METAL 2264
12.6.1. Available installation configuration parameters for bare metal 2264
.CHAPTER
. . . . . . . . . . 13.
. . . INSTALLING
. . . . . . . . . . . . . .ON-PREMISE
. . . . . . . . . . . . . .WITH
. . . . . .ASSISTED
. . . . . . . . . . .INSTALLER
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2274
.................
13.1. INSTALLING AN ON-PREMISE CLUSTER USING THE ASSISTED INSTALLER 2274
13.1.1. Using the Assisted Installer 2274
13.1.2. API support for the Assisted Installer 2275
.CHAPTER
. . . . . . . . . . 14.
. . . INSTALLING
. . . . . . . . . . . . . .AN
. . . ON-PREMISE
. . . . . . . . . . . . . . .CLUSTER
. . . . . . . . . . WITH
. . . . . . THE
. . . . .AGENT-BASED
. . . . . . . . . . . . . . . . INSTALLER
. . . . . . . . . . . . . . . . . . . . . . 2276
.................
14.1. PREPARING TO INSTALL WITH THE AGENT-BASED INSTALLER 2276
14.1.1. About the Agent-based Installer 2276
14.1.2. Understanding Agent-based Installer 2276
14.1.2.1. Agent-based Installer workflow 2277
14.1.2.2. Recommended resources for topologies 2278
14.1.3. About FIPS compliance 2279
14.1.4. Configuring FIPS through the Agent-based Installer 2280
14.1.5. About networking 2280
14.1.5.1. DHCP 2281
14.1.5.2. Static networking 2281
14.1.6. Requirements for a cluster using the platform "none" option 2283
14.1.6.1. Platform "none" DNS requirements 2283
14.1.6.1.1. Example DNS configuration for platform "none" clusters 2285
14.1.6.2. Platform "none" Load balancing requirements 2287
14.1.6.2.1. Example load balancer configuration for platform "none" clusters 2289
14.1.7. Example: Bonds and VLAN interface node network configuration 2290
14.1.8. Example: Bonds and SR-IOV dual-nic node network configuration 2292
14.1.9. Sample install-config.yaml file for bare metal 2295
14.1.10. Validation checks before agent ISO creation 2298
14.1.10.1. ZTP manifests 2298
14.1.11. About root device hints 2298
14.1.12. Next steps 2299
14.2. UNDERSTANDING DISCONNECTED INSTALLATION MIRRORING 2299
14.2.1. Mirroring images for a disconnected installation through the Agent-based Installer 2300
14.2.2. About mirroring the OpenShift Container Platform image repository for a disconnected registry 2300
14.2.2.1. Configuring the Agent-based Installer to use mirrored images 2301
14.3. INSTALLING AN OPENSHIFT CONTAINER PLATFORM CLUSTER WITH THE AGENT-BASED INSTALLER
2302
14.3.2. Installing OpenShift Container Platform with the Agent-based Installer 2302
14.3.2.1. Downloading the Agent-based Installer 2302
14.3.2.2. Creating the preferred configuration inputs 2303
14.3.2.3. Optional: Creating additional manifest files 2306
14.3.2.4. Optional: Using ZTP manifests 2308
14.3.2.5. Optional: Encrypting the disk 2309
14.3.2.6. Creating and booting the agent image 2310
14.3.2.7. Verifying that the current installation host can pull release images 2310
14.3.2.8. Tracking and verifying installation progress 2312
14.3.3. Sample GitOps ZTP custom resources 2314
43
14.3.4. Gathering log data from a failed Agent-based installation 2316

14.4. PREPARING PXE ASSETS FOR OPENSHIFT CONTAINER PLATFORM 2317
14.4.2. Downloading the Agent-based Installer 2318
14.4.3. Creating the preferred configuration inputs 2318
14.4.4. Creating the PXE assets 2322
14.4.5. Manually adding IBM Z agents 2323
14.4.5.1. Adding IBM Z agents with z/VM 2323
14.4.5.2. Adding IBM Z(R) agents with RHEL KVM 2324
14.5. PREPARING AN AGENT-BASED INSTALLED CLUSTER FOR THE MULTICLUSTER ENGINE FOR
KUBERNETES OPERATOR 2325
14.5.2. Preparing an Agent-based cluster deployment for the multicluster engine for Kubernetes Operator while
disconnected 2325
14.5.3. Preparing an Agent-based cluster deployment for the multicluster engine for Kubernetes Operator while
connected 2327
14.6. INSTALLATION CONFIGURATION PARAMETERS FOR THE AGENT-BASED INSTALLER 2332
14.6.1. Available installation configuration parameters 2332
14.6.1.4. Additional bare metal configuration parameters for the Agent-based Installer 2342
14.6.1.5. Additional VMware vSphere configuration parameters 2345
14.6.1.6. Deprecated VMware vSphere configuration parameters 2348
14.6.2. Available Agent configuration parameters 2350
. . . . . . . . . . . 15.
. . . . . . . . . . . . . .ON
. . . .A. SINGLE
. . . . . . . . .NODE
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2354
.................
15.1. PREPARING TO INSTALL ON A SINGLE NODE 2354
15.1.2. About OpenShift on a single node 2354
15.1.3. Requirements for installing OpenShift on a single node 2354
15.2. INSTALLING OPENSHIFT ON A SINGLE NODE 2356
15.2.1. Installing single-node OpenShift using the Assisted Installer 2356
15.2.1.1. Generating the discovery ISO with the Assisted Installer 2356
15.2.1.2. Installing single-node OpenShift with the Assisted Installer 2357
15.2.2. Installing single-node OpenShift manually 2357
15.2.2.1. Generating the installation ISO with coreos-installer 2357
15.2.2.2. Monitoring the cluster installation using openshift-install 2360
15.2.3. Installing single-node OpenShift on cloud providers 2361
15.2.3.1. Additional requirements for installing single-node OpenShift on a cloud provider 2361
15.2.3.2. Supported cloud providers for single-node OpenShift 2361
15.2.3.3. Installing single-node OpenShift on AWS 2361
15.2.3.4. Installing single-node OpenShift on Azure 2361
15.2.3.5. Installing single-node OpenShift on GCP 2362
15.2.4. Creating a bootable ISO image on a USB drive 2362
15.2.5. Booting from an HTTP-hosted ISO image using the Redfish API 2362
15.2.6. Creating a custom live RHCOS ISO for remote server access 2363
15.2.7. Installing single-node OpenShift with IBM Z and IBM LinuxONE 2365
Hardware requirements 2365
15.2.7.1. Installing single-node OpenShift with z/VM on IBM Z and IBM LinuxONE 2365
44
Table of Contents
15.2.7.2. Installing single-node OpenShift with RHEL KVM on IBM Z and IBM LinuxONE 2370
15.2.8. Installing single-node OpenShift with IBM Power 2373
15.2.8.1. Setting up basion for single-node OpenShift with IBM Power 2374
15.2.8.2. Installing single-node OpenShift with IBM Power 2377
. . . . . . . . . . . 16.
CHAPTER . . . DEPLOYING
. . . . . . . . . . . . . .INSTALLER-PROVISIONED
. . . . . . . . . . . . . . . . . . . . . . . . . . . .CLUSTERS
. . . . . . . . . . . .ON
. . . BARE
. . . . . . .METAL
. . . . . . . . . . . . . . . . . . . . . . . . . . . 2379
.................
16.1. OVERVIEW 2379
16.2. PREREQUISITES 2380
16.2.1. Node requirements 2381
16.2.2. Planning a bare metal cluster for OpenShift Virtualization 2382
16.2.3. Firmware requirements for installing with virtual media 2383
16.2.4. Network requirements 2384
16.2.4.1. Ensuring required ports are open 2384
16.2.4.2. Increase the network MTU 2386
16.2.4.3. Configuring NICs 2386
16.2.4.4. DNS requirements 2387
16.2.4.5. Dynamic Host Configuration Protocol (DHCP) requirements 2388
16.2.4.6. Reserving IP addresses for nodes with the DHCP server 2388
16.2.4.7. Provisioner node requirements 2390
16.2.4.8. Network Time Protocol (NTP) 2390
16.2.4.9. Port access for the out-of-band management IP address 2390
16.2.5. Configuring nodes 2390
Configuring nodes when using the provisioning network 2390
Configuring nodes without the provisioning network 2391
Configuring nodes for Secure Boot manually 2392
16.2.6. Out-of-band management 2392
16.2.7. Required data for installation 2393
16.2.8. Validation checklist for nodes 2393
16.3. SETTING UP THE ENVIRONMENT FOR AN OPENSHIFT INSTALLATION 2394
16.3.1. Installing RHEL on the provisioner node 2394
16.3.2. Preparing the provisioner node for OpenShift Container Platform installation 2394
16.3.3. Checking NTP server synchronization 2396
16.3.4. Configuring networking 2397
16.3.5. Establishing communication between subnets 2398
16.3.6. Retrieving the OpenShift Container Platform installer 2401
16.3.7. Extracting the OpenShift Container Platform installer 2401
16.3.8. Optional: Creating an RHCOS images cache 2402
16.3.9. Setting the cluster node hostnames through DHCP 2404
16.3.10. Configuring the install-config.yaml file 2404
16.3.10.1. Configuring the install-config.yaml file 2404
16.3.10.2. Additional install-config parameters 2407
Hosts 2412
16.3.10.3. BMC addressing 2412
IPMI 2413
Redfish network boot 2413
Redfish APIs 2414
16.3.10.4. BMC addressing for Dell iDRAC 2415
BMC address formats for Dell iDRAC 2415
Redfish virtual media for Dell iDRAC 2415
Redfish network boot for iDRAC 2416
16.3.10.5. BMC addressing for HPE iLO 2417
Redfish virtual media for HPE iLO 2418
45
Redfish network boot for HPE iLO 2418

16.3.10.6. BMC addressing for Fujitsu iRMC 2419
16.3.10.7. Root device hints 2420
16.3.10.8. Optional: Setting proxy settings 2421
16.3.10.9. Optional: Deploying with no provisioning network 2422
16.3.10.10. Optional: Deploying with dual-stack networking 2422
16.3.10.11. Optional: Configuring host network interfaces 2423
16.3.10.12. Configuring host network interfaces for subnets 2425
16.3.10.13. Optional: Configuring address generation modes for SLAAC in dual-stack networks 2426
16.3.10.14. Optional: Configuring host network interfaces for dual port NIC 2428
16.3.10.15. Configuring multiple cluster nodes 2431
16.3.10.16. Optional: Configuring managed Secure Boot 2431
16.3.11. Manifest configuration files 2432
16.3.11.1. Creating the OpenShift Container Platform manifests 2432
16.3.11.2. Optional: Configuring NTP for disconnected clusters 2432
16.3.11.3. Configuring network components to run on the control plane 2435
16.3.11.4. Optional: Deploying routers on worker nodes 2437
16.3.11.5. Optional: Configuring the BIOS 2438
16.3.11.6. Optional: Configuring the RAID 2439
16.3.11.7. Optional: Configuring storage on nodes 2440
16.3.12. Creating a disconnected registry 2441
Prerequisites 2441
16.3.12.1. Preparing the registry node to host the mirrored registry 2441
16.3.12.2. Mirroring the OpenShift Container Platform image repository for a disconnected registry 2442
16.3.12.3. Modify the install-config.yaml file to use the disconnected registry 2445
16.3.13. Validation checklist for installation 2446
16.3.14. Deploying the cluster via the OpenShift Container Platform installer 2446
16.3.15. Following the installation 2446
16.3.16. Verifying static IP address configuration 2446
16.3.17. Preparing to reinstall a cluster on bare metal 2447
16.4. INSTALLER-PROVISIONED POSTINSTALLATION CONFIGURATION 2447
16.4.1. Optional: Configuring NTP for disconnected clusters 2447
16.4.2. Enabling a provisioning network after installation 2450
16.4.3. Services for an external load balancer 2452
16.4.3.1. Configuring an external load balancer 2455
16.5. EXPANDING THE CLUSTER 2460
16.5.1. Preparing the bare metal node 2461
16.5.2. Replacing a bare-metal control plane node 2465
16.5.3. Preparing to deploy with Virtual Media on the baremetal network 2469
16.5.4. Diagnosing a duplicate MAC address when provisioning a new host in the cluster 2471
16.5.5. Provisioning the bare metal node 2472
16.6. TROUBLESHOOTING 2474
16.6.1. Troubleshooting the installation program workflow 2474
16.6.2. Troubleshooting install-config.yaml 2476
16.6.3. Troubleshooting bootstrap VM issues 2477
16.6.3.1. Bootstrap VM cannot boot up the cluster nodes 2478
16.6.3.2. Inspecting logs 2479
16.6.4. Investigating an unavailable Kubernetes API 2480
16.6.5. Troubleshooting a failure to initialize the cluster 2481
16.6.6. Troubleshooting a failure to fetch the console URL 2484
16.6.7. Troubleshooting a failure to add the ingress certificate to kubeconfig 2485
16.6.8. Troubleshooting SSH access to cluster nodes 2486
46
Table of Contents
16.6.9. Cluster nodes will not PXE boot 2486

16.6.10. Installing creates no worker nodes 2487
16.6.11. Troubleshooting the Cluster Network Operator 2487
16.6.12. Unable to discover new bare metal hosts using the BMC 2488
16.6.13. Troubleshooting worker nodes that cannot join the cluster 2489
16.6.14. Cleaning up previous installations 2489
16.6.15. Issues with creating the registry 2490
16.6.16. Miscellaneous issues 2490
16.6.16.1. Addressing the runtime network not ready error 2490
16.6.16.2. Addressing the "No disk found with matching rootDeviceHints" error message 2491
16.6.16.3. Cluster nodes not getting the correct IPv6 address over DHCP 2492
16.6.16.4. Cluster nodes not getting the correct hostname over DHCP 2492
16.6.16.5. Routes do not reach endpoints 2494
16.6.16.6. Failed Ignition during Firstboot 2495
16.6.16.7. NTP out of sync 2495
16.6.17. Reviewing the installation 2497
. . . . . . . . . . . 17.
. . . . . . . . . . . . . .IBM
. . . . CLOUD
. . . . . . . . BARE
. . . . . . .METAL
. . . . . . . .(CLASSIC)
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2498
.................
17.1.1. Setting up IBM Cloud Bare Metal (Classic) infrastructure 2498
Use one data center per cluster 2498
Create public and private VLANs 2498
Ensure subnets have sufficient IP addresses 2498
Configuring NICs 2499
Configuring canonical names 2500
Creating DNS entries 2500
Network Time Protocol (NTP) 2501
Configure a DHCP server 2501
Ensure BMC access privileges 2501
Create bare metal servers 2502
17.2. SETTING UP THE ENVIRONMENT FOR AN OPENSHIFT CONTAINER PLATFORM INSTALLATION 2502
17.2.1. Preparing the provisioner node on IBM Cloud(R) Bare Metal (Classic) infrastructure 2502
17.2.2. Configuring the public subnet 2506
17.2.3. Retrieving the OpenShift Container Platform installer 2508
17.2.4. Extracting the OpenShift Container Platform installer 2509
17.2.5. Configuring the install-config.yaml file 2509
17.2.6. Additional install-config parameters 2511
Hosts 2515
17.2.7. Root device hints 2516
17.2.8. Creating the OpenShift Container Platform manifests 2517
17.2.9. Deploying the cluster via the OpenShift Container Platform installer 2518
17.2.10. Following the installation 2518
. . . . . . . . . . . 18.
. . . . . . . . . . . . . .ON
. . . .IBM
. . . .Z. .AND
. . . . .IBM
. . . . LINUXONE
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2519
................
18.1. PREPARING TO INSTALL ON IBM Z AND IBM LINUXONE 2519
18.1.2. Choosing a method to install OpenShift Container Platform on IBM Z or IBM LinuxONE 2519
18.1.2.1. User-provisioned infrastructure installation of OpenShift Container Platform on IBM Z 2520
18.2. INSTALLING A CLUSTER WITH Z/VM ON IBM Z AND IBM LINUXONE 2521
47

18.2.3.3. Minimum IBM Z system environment 2523
Operating system requirements 2524
IBM Z network connectivity requirements 2524
Disk storage 2524
Storage / Main Memory 2524
18.2.3.4. Preferred IBM Z system environment 2524
Disk storage 2525
18.2.9.1. Sample install-config.yaml file for IBM Z 2544
18.2.12. Configuring NBDE with static IP in an IBM Z or IBM LinuxONE environment 2559
18.2.13.1. Advanced RHCOS installation reference 2564
18.2.13.1.1. Networking and bonding options for ISO installations 2564
48
Table of Contents

18.2.17.1.1. Configuring registry storage for IBM Z 2573
18.2.20. Next steps 2577
18.3. INSTALLING A CLUSTER WITH Z/VM ON IBM Z AND IBM LINUXONE IN A RESTRICTED NETWORK
2578
Disk storage 2581
Disk storage 2582
49
18.3.18. Next steps 2634
18.4. INSTALLING A CLUSTER WITH RHEL KVM ON IBM Z AND IBM LINUXONE 2634
18.4.3. Machine requirements for a cluster with user-provisioned infrastructure 2635
18.4.3.1. Required machines 2635
18.4.3.3. IBM Z network connectivity requirements 2636
18.4.3.4. Host machine resource requirements 2636
18.4.3.6. Minimum resource requirements 2637
18.4.3.8. Preferred resource requirements 2638
50
Table of Contents
18.4.12.1. Installing RHCOS using IBM Secure Execution 2673
18.4.12.2. Configuring NBDE with static IP in an IBM Z or IBM LinuxONE environment 2676
18.4.12.3. Fast-track installation by using a prepackaged QCOW2 disk image 2678
18.4.12.4. Full installation on a new QCOW2 disk image 2679
18.4.12.5.1. Networking options for ISO installations 2680
18.4.19. Next steps 2693
18.5. INSTALLING A CLUSTER WITH RHEL KVM ON IBM Z AND IBM LINUXONE IN A RESTRICTED NETWORK
2693
18.5.4. Machine requirements for a cluster with user-provisioned infrastructure 2695
18.5.4.1. Required machines 2695
18.5.4.3. IBM Z network connectivity requirements 2696
18.5.4.4. Host machine resource requirements 2696
51

18.5.4.6. Minimum resource requirements 2697
18.5.4.8. Preferred resource requirements 2698
18.5.11.1. Installing RHCOS using IBM Secure Execution 2730
18.5.11.2. Configuring NBDE with static IP in an IBM Z or IBM LinuxONE environment 2733
18.5.11.3. Fast-track installation by using a prepackaged QCOW2 disk image 2735
18.5.11.4. Full installation on a new QCOW2 disk image 2736
18.5.11.5.1. Networking options for ISO installations 2737
52
Table of Contents
18.5.17. Next steps 2750

18.6. INSTALLING A CLUSTER IN AN LPAR ON IBM Z AND IBM LINUXONE 2750
Disk storage 2753
Disk storage 2754
53

18.6.20. Next steps 2806
18.7. INSTALLING A CLUSTER IN AN LPAR ON IBM Z AND IBM LINUXONE IN A RESTRICTED NETWORK
2806
Disk storage 2810
Disk storage 2811
54
Table of Contents

18.7.18. Next steps 2861
18.8. INSTALLATION CONFIGURATION PARAMETERS FOR IBM Z AND IBM LINUXONE 2862
. . . . . . . . . . . 19.
. . . . . . . . . . . . . .ON
. . . .IBM
. . . .POWER
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2880
.................
19.1. PREPARING TO INSTALL ON IBM POWER 2880
19.1.2. Choosing a method to install OpenShift Container Platform on IBM Power 2880
19.2. INSTALLING A CLUSTER ON IBM POWER 2880
19.2.3.3. Minimum IBM Power requirements 2882
55

Disk storage for the IBM Power guest virtual machines 2883
Network for the PowerVM guest virtual machines 2883
Storage / main memory 2883
19.2.3.4. Recommended IBM Power system requirements 2883
19.2.9.1. Sample install-config.yaml file for IBM Power 2904
19.2.12.2. Installing RHCOS by using PXE booting 2925
56
Table of Contents

19.2.16.1.1. Configuring registry storage for IBM Power 2936
19.2.19. Next steps 2940
19.3. INSTALLING A CLUSTER ON IBM POWER IN A RESTRICTED NETWORK 2940
19.3.4.3. Minimum IBM Power requirements 2944
19.3.4.4. Recommended IBM Power system requirements 2944
19.3.8.1. Sample install-config.yaml file for IBM Power 2962
57

19.3.11.2. Installing RHCOS by using PXE booting 2984
19.3.15.2.1. Changing the image registry’s management state 2995
19.3.15.2.2. Configuring registry storage for IBM Power 2995
19.3.17. Next steps 3000
19.4. INSTALLATION CONFIGURATION PARAMETERS FOR IBM POWER 3000
. . . . . . . . . . . 20.
CHAPTER . . . .INSTALLING
. . . . . . . . . . . . . ON
. . . . IBM
. . . . .POWER
. . . . . . . .VIRTUAL
. . . . . . . . . .SERVER
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3019
.................
20.1. PREPARING TO INSTALL ON IBM POWER VIRTUAL SERVER 3019
20.1.2. Requirements for installing OpenShift Container Platform on IBM Power Virtual Server 3019
20.1.3. Choosing a method to install OpenShift Container Platform on IBM Power Virtual Server 3019
20.1.5. Next steps 3021
20.2. CONFIGURING AN IBM CLOUD ACCOUNT 3021
20.2.2. Quotas and limits on IBM Power Virtual Server 3021
Virtual Private Cloud 3022
Application load balancer 3022
Transit Gateways 3022
Dynamic Host Configuration Protocol Service 3022
Networking 3022
Virtual Server Instances 3022
20.2.3. Configuring DNS resolution 3022
58
Table of Contents
20.2.4. Using IBM Cloud Internet Services for DNS resolution 3023
20.2.5. IBM Cloud IAM Policies and API Key 3024
20.2.5.1. Pre-requisite permissions 3024
20.2.5.2. Cluster-creation permissions 3024
20.2.5.3. Access policy assignment 3025
20.2.5.4. Creating an API key 3025
20.2.6. Supported IBM Power Virtual Server regions and zones 3026
20.2.7. Next steps 3027
20.3. CREATING AN IBM POWER VIRTUAL SERVER WORKSPACE 3027
20.3.1. Creating an IBM Power Virtual Server workspace 3027
20.3.2. Next steps 3027
20.4. INSTALLING A CLUSTER ON IBM POWER VIRTUAL SERVER WITH CUSTOMIZATIONS 3027
20.4.6.1. Sample customized install-config.yaml file for IBM Power Virtual Server 3032
20.4.12. Next steps 3042
20.5. INSTALLING A CLUSTER ON IBM POWER VIRTUAL SERVER INTO AN EXISTING VPC 3042
20.5.13. Next steps 3059
20.6. INSTALLING A PRIVATE CLUSTER ON IBM POWER VIRTUAL SERVER 3060
59

20.6.3. Private clusters in IBM Power Virtual Server 3061
20.6.3.1. Limitations 3061
20.6.4. Requirements for using your VPC 3061
20.6.15. Next steps 3077
20.7. INSTALLING A CLUSTER ON IBM POWER VIRTUAL SERVER IN A RESTRICTED NETWORK 3077
20.7.14. Next steps 3095
20.8. UNINSTALLING A CLUSTER ON IBM POWER VIRTUAL SERVER 3096
20.9. INSTALLATION CONFIGURATION PARAMETERS FOR IBM POWER VIRTUAL SERVER 3097
60
Table of Contents

. . . . . . . . . . . 21.
. . . . . . . . . . . . . .ON
. . . OPENSTACK
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3117
................
21.1. PREPARING TO INSTALL ON OPENSTACK 3117
21.1.2. Choosing a method to install OpenShift Container Platform on OpenStack 3117
21.1.3. Scanning RHOSP endpoints for legacy HTTPS certificates 3118
21.1.3.1. Scanning RHOSP endpoints for legacy HTTPS certificates manually 3120
21.2. PREPARING TO INSTALL A CLUSTER THAT USES SR-IOV OR OVS-DPDK ON OPENSTACK 3121
21.2.1. Requirements for clusters on RHOSP that use either SR-IOV or OVS-DPDK 3121
21.2.1.1. Requirements for clusters on RHOSP that use SR-IOV 3121
21.2.1.2. Requirements for clusters on RHOSP that use OVS-DPDK 3122
21.2.2. Preparing to install a cluster that uses SR-IOV 3122
21.2.2.1. Creating SR-IOV networks for compute machines 3122
21.2.3. Preparing to install a cluster that uses OVS-DPDK 3123
21.2.4. Next steps 3123
21.3. INSTALLING A CLUSTER ON OPENSTACK WITH CUSTOMIZATIONS 3124
21.3.2. Resource guidelines for installing OpenShift Container Platform on RHOSP 3124
21.3.2.1. Control plane machines 3125
21.3.2.2. Compute machines 3125
21.3.2.3. Bootstrap machine 3126
21.3.2.4.1. Example load balancer configuration for clusters that are deployed with user-managed load
balancers 3128
21.3.4. Enabling Swift on RHOSP 3130
21.3.5. Configuring an image registry with custom storage on clusters that run on RHOSP 3131
21.3.6. Verifying external network access 3133
21.3.7. Defining parameters for the installation program 3134
21.3.8. Setting OpenStack Cloud Controller Manager options 3136
21.3.10.2. Custom subnets in RHOSP deployments 3141
21.3.10.3. Deploying a cluster with bare metal machines 3142
21.3.10.4. Cluster deployment on RHOSP provider networks 3143
21.3.10.4.1. RHOSP provider network requirements for cluster installation 3144
21.3.10.4.2. Deploying a cluster that has a primary interface on a provider network 3145
21.3.10.5. Sample customized install-config.yaml file for RHOSP 3147
21.3.10.6. Optional: Configuring a cluster with dual-stack networking 3149
21.3.10.6.1. Deploying the dual-stack cluster 3149
21.3.10.7. Installation configuration for a cluster on OpenStack with a user-managed load balancer 3152
21.3.12. Enabling access to the environment 3154
21.3.12.1. Enabling access with floating IP addresses 3154
61
21.3.12.2. Completing installation without floating IP addresses 3156

21.3.14. Verifying cluster status 3158
21.3.17. Next steps 3159
21.4. INSTALLING A CLUSTER ON OPENSTACK ON YOUR OWN INFRASTRUCTURE 3160
21.4.4. Downloading playbook dependencies 3163
21.4.5. Downloading the installation playbooks 3163
21.4.8. Creating the Red Hat Enterprise Linux CoreOS (RHCOS) image 3167
21.4.9. Verifying external network access 3168
21.4.12. Creating network resources on RHOSP 3172
21.4.13.1. Custom subnets in RHOSP deployments 3174
21.4.13.2. Sample customized install-config.yaml file for RHOSP 3175
21.4.13.3. Setting a custom subnet for machines 3177
21.4.13.4. Emptying compute machine pools 3178
21.4.13.5. Cluster deployment on RHOSP provider networks 3178
21.4.13.5.1. RHOSP provider network requirements for cluster installation 3180
21.4.13.5.2. Deploying a cluster that has a primary interface on a provider network 3181
21.4.15. Preparing the bootstrap Ignition files 3184
21.4.16. Creating control plane Ignition config files on RHOSP 3186
21.4.17. Updating network resources on RHOSP 3187
21.4.17.1. Deploying a cluster with bare metal machines 3189
21.4.18. Creating the bootstrap machine on RHOSP 3190
21.4.19. Creating the control plane machines on RHOSP 3191
21.4.21. Deleting bootstrap resources from RHOSP 3192
21.4.22. Creating compute machines on RHOSP 3193
21.4.24. Verifying a successful installation 3196
21.4.26. Next steps 3197
21.5. INSTALLING A CLUSTER ON OPENSTACK IN A RESTRICTED NETWORK 3197
62
Table of Contents

21.5.5. Enabling Swift on RHOSP 3200
21.5.7. Setting OpenStack Cloud Controller Manager options 3202
21.5.8. Creating the RHCOS image for restricted network installations 3204
21.5.9.2. Sample customized install-config.yaml file for restricted OpenStack installations 3209
21.5.13. Verifying cluster status 3215
21.5.17. Next steps 3217
21.6. OPENSTACK CLOUD CONTROLLER MANAGER REFERENCE GUIDE 3218
21.6.1. The OpenStack Cloud Controller Manager 3218
21.6.2. The OpenStack Cloud Controller Manager (CCM) config map 3218
21.6.2.1. Load balancer options 3219
21.6.2.2. Options that the Operator overrides 3222
21.7. DEPLOYING ON OPENSTACK WITH ROOTVOLUME AND ETCD ON LOCAL DISK 3224
21.7.1. Deploying RHOSP on local disk 3224
21.8. UNINSTALLING A CLUSTER ON OPENSTACK 3230
21.9. UNINSTALLING A CLUSTER ON RHOSP FROM YOUR OWN INFRASTRUCTURE 3231
21.9.1. Downloading playbook dependencies 3231
21.9.2. Removing a cluster from RHOSP that uses your own infrastructure 3232
21.10. INSTALLATION CONFIGURATION PARAMETERS FOR OPENSTACK 3232
21.10.1.4. Additional Red Hat OpenStack Platform (RHOSP) configuration parameters 3242
21.10.1.5. Optional RHOSP configuration parameters 3245
. . . . . . . . . . . 22.
. . . . . . . . . . . . . ON
. . . .OCI
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3258
.................
22.1. INSTALLING A CLUSTER ON ORACLE CLOUD INFRASTRUCTURE (OCI) BY USING THE ASSISTED
INSTALLER 3258
22.1.1. The Assisted Installer and OCI overview 3258
22.1.2. Creating OCI resources and services 3259
22.1.3. Using the Assisted Installer to generate an OCI-compatible discovery ISO image 3260
22.1.4. Downloading manifest files and deployment resources 3261
22.1.5. Completing the remaining Assisted Installer steps 3263
22.1.6. Verifying a successful cluster installation on OCI 3264
22.1.7. Troubleshooting installation of a cluster on OCI 3265
63
The Ingress Load Balancer in OCI is not at a healthy status 3265

OCI create stack operation fails with an Error: 400-InvalidParameter message 3265
22.2. INSTALLING A CLUSTER ON ORACLE CLOUD INFRASTRUCTURE (OCI) BY USING THE AGENT-BASED
INSTALLER 3266
22.2.1. The Agent-based Installer and OCI overview 3266
22.2.2. Creating OCI infrastructure resources and services 3267
22.2.3. Creating configuration files for installing a cluster on OCI 3269
22.2.4. Configuring your firewall for OpenShift Container Platform 3273
22.2.5. Running a cluster on OCI 3275
22.2.6. Verifying that your Agent-based cluster installation runs on OCI 3277
. . . . . . . . . . . 23.
. . . . . . . . . . . . . ON
. . . .VSPHERE
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3279
.................
23.1. INSTALLATION METHODS 3279
23.1.1. Assisted Installer 3279
23.1.2. Agent-based Installer 3279
23.1.3. Installer-provisioned infrastructure installation 3279
23.1.4. User-provisioned infrastructure installation 3279
23.2. INSTALLER-PROVISIONED INFRASTRUCTURE 3280
23.2.1. vSphere installation requirements 3280
23.2.1.1. VMware vSphere infrastructure requirements 3280
23.2.1.3. VMware vSphere CSI Driver Operator requirements 3283
23.2.1.4. vCenter requirements 3284
Required vCenter account privileges 3284
Minimum required vCenter account privileges 3292
Using OpenShift Container Platform with vMotion 3301
Cluster resources 3302
Cluster limits 3302
Networking requirements 3303
Required IP Addresses 3303
DNS records 3303
Static IP addresses for vSphere nodes 3304
23.2.2. Preparing to install a cluster using installer-provisioned infrastructure 3305
23.2.2.4. Adding vCenter root CA certificates to your system trust 3310
23.2.3. Installing a cluster on vSphere 3311
23.2.3.3. Deploying the cluster 3312
23.2.3.5. Creating registry storage 3316
23.2.3.5.1. Image registry removed during installation 3316
23.2.3.5.2. Image registry storage configuration 3316
23.2.3.5.2.1. Configuring registry storage for VMware vSphere 3316
23.2.3.5.2.2. Configuring block registry storage for VMware vSphere 3318
23.2.3.6. Telemetry access for OpenShift Container Platform 3319
23.2.3.7. Next steps 3319
64
Table of Contents
23.2.4. Installing a cluster on vSphere with customizations 3319

23.2.4.3. VMware vSphere region and zone enablement 3321
23.2.4.4.1. Sample install-config.yaml file for an installer-provisioned VMware vSphere cluster 3325
23.2.4.4.2. Configuring the cluster-wide proxy during installation 3326
23.2.4.4.3. Configuring regions and zones for a VMware vCenter 3328
23.2.4.9. Services for an external load balancer 3336
23.2.4.9.1. Configuring an external load balancer 3339
23.2.4.10. Next steps 3344
23.2.5. Installing a cluster on vSphere with network customizations 3345
23.2.5.4.3. Optional: Deploying with dual-stack networking 3353
23.2.5.5. Network configuration phases 3356
23.2.5.6. Specifying advanced network configuration 3357
23.2.5.7. Cluster Network Operator configuration 3358
23.2.5.7.1. Cluster Network Operator configuration object 3358
23.2.5.13. Configuring network components to run on the control plane 3380
23.2.5.14. Next steps 3382
23.2.6. Installing a cluster on vSphere in a restricted network 3382
23.2.6.2. About installations in restricted networks 3383
23.2.6.2.1. Additional limits 3384
23.2.6.4. Creating the RHCOS image for restricted network installations 3384
65

23.2.6.13. Next steps 3408
23.3. USER-PROVISIONED INFRASTRUCTURE 3409
23.3.1. vSphere installation requirements for user-provisioned infrastructure 3409
23.3.1.1. VMware vSphere infrastructure requirements 3409
23.3.1.2. VMware vSphere CSI Driver Operator requirements 3410
23.3.1.3. Requirements for a cluster with user-provisioned infrastructure 3411
23.3.1.3.1. vCenter requirements 3411
Required vCenter account privileges 3411
Minimum required vCenter account privileges 3419
Using OpenShift Container Platform with vMotion 3424
Cluster resources 3425
Cluster limits 3425
Networking requirements 3426
DNS records 3426
23.3.1.3.2. Required machines for cluster installation 3427
23.3.1.3.3. Minimum resource requirements for cluster installation 3428
23.3.1.3.4. Requirements for encrypting virtual machines 3428
23.3.1.3.5. Certificate signing requests management 3429
23.3.1.3.6. Networking requirements for user-provisioned infrastructure 3429
23.3.1.3.6.1. Setting the cluster node hostnames through DHCP 3430
23.3.1.3.6.2. Network connectivity requirements 3430
Ethernet adaptor hardware address requirements 3431
23.3.1.3.7. User-provisioned DNS requirements 3432
23.3.1.3.7.1. Example DNS configuration for user-provisioned clusters 3434
23.3.1.3.8. Load balancing requirements for user-provisioned infrastructure 3436
23.3.1.3.8.1. Example load balancer configuration for user-provisioned clusters 3438
23.3.2. Preparing to install a cluster using user-provisioned infrastructure 3440
23.3.2.4. Preparing the user-provisioned infrastructure 3445
23.3.2.5. Validating DNS resolution for user-provisioned infrastructure 3446
23.3.3. Installing a cluster on vSphere with user-provisioned infrastructure 3449
66
Table of Contents

23.3.3.4.1. Sample install-config.yaml file for VMware vSphere 3452
23.3.3.7. Installing RHCOS and starting the OpenShift Container Platform bootstrap process 3462
23.3.3.8. Adding more compute machines to a cluster in vSphere 3467
23.3.3.9. Disk partitioning 3468
Creating a separate /var partition 3468
23.3.3.10. Waiting for the bootstrap process to complete 3470
23.3.3.12. Approving the certificate signing requests for your machines 3472
23.3.3.13. Initial Operator configuration 3475
23.3.3.13.2.2. Configuring storage for the image registry in non-production clusters 3477
23.3.3.14. Completing installation on user-provisioned infrastructure 3479
23.3.3.15. Configuring vSphere DRS anti-affinity rules for control plane nodes 3482
23.3.3.17. Next steps 3483
23.3.4. Installing a cluster on vSphere with network customizations 3483
23.3.4.5. Network configuration phases 3494
23.3.4.6. Specifying advanced network configuration 3494
23.3.4.7. Cluster Network Operator configuration 3496
23.3.4.7.1. Cluster Network Operator configuration object 3496
23.3.4.8. Creating the Ignition config files 3503
23.3.4.15.1. Initial Operator configuration 3517
67

23.3.4.19. Next steps 3524
23.3.5. Installing a cluster on vSphere in a restricted network with user-provisioned infrastructure 3524
23.3.5.2. About installations in restricted networks 3525
23.3.5.2.1. Additional limits 3526
23.3.5.7. Configuring chrony time service 3538
23.3.5.15. Initial Operator configuration 3552
23.3.5.15.1. Disabling the default OperatorHub catalog sources 3553
23.3.5.15.2.2. Configuring storage for the image registry in non-production clusters 3555
23.3.5.19. Next steps 3561
23.4. INSTALLING A CLUSTER ON VSPHERE USING THE ASSISTED INSTALLER 3561
23.5. INSTALLING A CLUSTER ON VSPHERE USING THE AGENT-BASED INSTALLER 3562
23.6. INSTALLING A THREE-NODE CLUSTER ON VSPHERE 3562
23.6.2. Next steps 3563
23.7. UNINSTALLING A CLUSTER ON VSPHERE THAT USES INSTALLER-PROVISIONED INFRASTRUCTURE
3563
23.8. USING THE VSPHERE PROBLEM DETECTOR OPERATOR 3564
23.8.1. About the vSphere Problem Detector Operator 3564
23.8.2. Running the vSphere Problem Detector Operator checks 3565
23.8.3. Viewing the events from the vSphere Problem Detector Operator 3566
23.8.4. Viewing the logs from the vSphere Problem Detector Operator 3566
23.8.5. Configuration checks run by the vSphere Problem Detector Operator 3567
23.8.6. About the storage class configuration check 3568
23.8.7. Metrics for the vSphere Problem Detector Operator 3569
68
Table of Contents
23.9. INSTALLATION CONFIGURATION PARAMETERS FOR VSPHERE 3569

23.9.1.7. Optional VMware vSphere machine pool configuration parameters 3589
. . . . . . . . . . . 24.
. . . . . . . . . . . . . ON
. . . . ANY
. . . . . PLATFORM
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3591
................
24.1. INSTALLING A CLUSTER ON ANY PLATFORM 3591
24.1.9.1. Sample install-config.yaml file for other platforms 3613
69

24.1.18. Next steps 3652
.CHAPTER
. . . . . . . . . . 25.
. . . .INSTALLATION
. . . . . . . . . . . . . . . .CONFIGURATION
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3653
.................
25.1. CUSTOMIZING NODES 3653
25.1.1. Creating machine configs with Butane 3653
25.1.1.1. About Butane 3653
25.1.1.2. Installing Butane 3653
25.1.1.3. Creating a MachineConfig object by using Butane 3654
25.1.2. Adding day-1 kernel arguments 3655
25.1.3. Adding kernel modules to nodes 3656
25.1.3.1. Building and testing the kernel module container 3657
25.1.3.2. Provisioning a kernel module to OpenShift Container Platform 3660
25.1.3.2.1. Provision kernel modules via a MachineConfig object 3660
25.1.4. Encrypting and mirroring disks during installation 3662
25.1.4.1. About disk encryption 3662
25.1.4.1.1. Configuring an encryption threshold 3663
25.1.4.2. About disk mirroring 3665
25.1.4.3. Configuring disk encryption and mirroring 3665
25.1.4.4. Configuring a RAID-enabled data volume 3672
25.1.5. Configuring chrony time service 3674
25.2. CONFIGURING YOUR FIREWALL 3676
25.2.1. Configuring your firewall for OpenShift Container Platform 3676
25.3. ENABLING LINUX CONTROL GROUP VERSION 1 (CGROUP V1) 3681
25.3.1. Enabling Linux cgroup v1 during installation 3681
. . . . . . . . . . . 26.
CHAPTER . . . .VALIDATING
. . . . . . . . . . . . . AN
. . . . INSTALLATION
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3683
.................
26.1. REVIEWING THE INSTALLATION LOG 3683
26.2. VIEWING THE IMAGE PULL SOURCE 3683
26.3. GETTING CLUSTER VERSION, STATUS, AND UPDATE DETAILS 3684
26.4. CLUSTERS THAT USE SHORT-TERM CREDENTIALS: VERIFYING THE CREDENTIALS CONFIGURATION
3686
26.5. QUERYING THE STATUS OF THE CLUSTER NODES BY USING THE CLI 3687
26.6. REVIEWING THE CLUSTER STATUS FROM THE OPENSHIFT CONTAINER PLATFORM WEB CONSOLE
3687
26.7. REVIEWING THE CLUSTER STATUS FROM RED HAT OPENSHIFT CLUSTER MANAGER 3688
70
Table of Contents
26.8. CHECKING CLUSTER RESOURCE AVAILABILITY AND UTILIZATION 3689

26.9. LISTING ALERTS THAT ARE FIRING 3691
26.10. NEXT STEPS 3691
. . . . . . . . . . . 27.
CHAPTER . . . .TROUBLESHOOTING
. . . . . . . . . . . . . . . . . . . . . . INSTALLATION
. . . . . . . . . . . . . . . . ISSUES
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3692
.................
27.2. GATHERING LOGS FROM A FAILED INSTALLATION 3692
27.3. MANUALLY GATHERING LOGS WITH SSH ACCESS TO YOUR HOST(S) 3693
27.4. MANUALLY GATHERING LOGS WITHOUT SSH ACCESS TO YOUR HOST(S) 3694
27.5. GETTING DEBUG INFORMATION FROM THE INSTALLATION PROGRAM 3694
27.6. REINSTALLING THE OPENSHIFT CONTAINER PLATFORM CLUSTER 3695
. . . . . . . . . . . 28.
CHAPTER . . . .SUPPORT
. . . . . . . . . . .FOR
. . . . .FIPS
. . . . .CRYPTOGRAPHY
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3696
.................
28.1. FIPS VALIDATION IN OPENSHIFT CONTAINER PLATFORM 3696
28.2. FIPS SUPPORT IN COMPONENTS THAT THE CLUSTER USES 3697
28.2.1. etcd 3697
28.2.2. Storage 3697
28.2.3. Runtimes 3697
28.3. INSTALLING A CLUSTER IN FIPS MODE 3697
71
72
CHAPTER 1. OPENSHIFT CONTAINER PLATFORM INSTALLATION OVERVIEW
CHAPTER 1. OPENSHIFT CONTAINER PLATFORM

INSTALLATION OVERVIEW
1.1. ABOUT OPENSHIFT CONTAINER PLATFORM INSTALLATION

The OpenShift Container Platform installation program offers four methods for deploying a cluster
which are detailed in the following list:
Interactive: You can deploy a cluster with the web-based Assisted Installer. This is an ideal
approach for clusters with networks connected to the internet. The Assisted Installer is the
easiest way to install OpenShift Container Platform, it provides smart defaults, and it performs
pre-flight validations before installing the cluster. It also provides a RESTful API for automation
and advanced configuration scenarios.
Local Agent-based: You can deploy a cluster locally with the Agent-based Installer for
disconnected environments or restricted networks. It provides many of the benefits of the
Assisted Installer, but you must download and configure the Agent-based Installer first.
Configuration is done with a command-line interface. This approach is ideal for disconnected
environments.
Automated: You can deploy a cluster on installer-provisioned infrastructure. The installation

program uses each cluster host’s baseboard management controller (BMC) for provisioning.
You can deploy clusters in connected or disconnected environments.
Full control: You can deploy a cluster on infrastructure that you prepare and maintain, which
provides maximum customizability. You can deploy clusters in connected or disconnected
environments.
Each method deploys a cluster with the following characteristics:
Highly available infrastructure with no single points of failure, which is available by default.
Administrators can control what updates are applied and when.
1.1.1. About the installation program

You can use the installation program to deploy each type of cluster. The installation program generates
the main assets, such as Ignition config files for the bootstrap, control plane, and compute machines.
You can start an OpenShift Container Platform cluster with these three machine configurations,
provided you correctly configured the infrastructure.
The OpenShift Container Platform installation program uses a set of targets and dependencies to
manage cluster installations. The installation program has a set of targets that it must achieve, and each
target has a set of dependencies. Because each target is only concerned with its own dependencies, the
installation program can act to achieve multiple targets in parallel with the ultimate target being a
running cluster. The installation program recognizes and uses existing components instead of running
commands to create them again because the program meets the dependencies.
Figure 1.1. OpenShift Container Platform installation targets and dependencies

73
Figure 1.1. OpenShift Container Platform installation targets and dependencies
1.1.2. About Red Hat Enterprise Linux CoreOS (RHCOS)

Post-installation, each cluster machine uses Red Hat Enterprise Linux CoreOS (RHCOS) as the
operating system. RHCOS is the immutable container host version of Red Hat Enterprise Linux (RHEL)
and features a RHEL kernel with SELinux enabled by default. RHCOS includes the kubelet, which is the
Kubernetes node agent, and the CRI-O container runtime, which is optimized for Kubernetes.
Every control plane machine in an OpenShift Container Platform 4.15 cluster must use RHCOS, which
includes a critical first-boot provisioning tool called Ignition. This tool enables the cluster to configure
the machines. Operating system updates are delivered as a bootable container image, using OSTree as
a backend, that is deployed across the cluster by the Machine Config Operator. Actual operating system
changes are made in-place on each machine as an atomic operation by using rpm-ostree. Together,
these technologies enable OpenShift Container Platform to manage the operating system like it
manages any other application on the cluster, by in-place upgrades that keep the entire platform up to
date. These in-place updates can reduce the burden on operations teams.
If you use RHCOS as the operating system for all cluster machines, the cluster manages all aspects of its
components and machines, including the operating system. Because of this, only the installation
program and the Machine Config Operator can change machines. The installation program uses Ignition
config files to set the exact state of each machine, and the Machine Config Operator completes more
changes to the machines, such as the application of new certificates or keys, after installation.
1.1.3. Glossary of common terms for OpenShift Container Platform installing

The glossary defines common terms that relate to the installation content. Read the following list of
terms to better understand the installation process.
Assisted Installer
An installer hosted at console.redhat.com that provides a web-based user interface or a RESTful API
74
An installer hosted at console.redhat.com that provides a web-based user interface or a RESTful API
for creating a cluster configuration. The Assisted Installer generates a discovery image. Cluster
machines boot with the discovery image, which installs RHCOS and an agent. Together, the Assisted
Installer and agent provide preinstallation validation and installation for the cluster.
Agent-based Installer
An installer similar to the Assisted Installer, but you must download the Agent-based Installer first.
The Agent-based Installer is ideal for disconnected environments.
Bootstrap node
A temporary machine that runs a minimal Kubernetes configuration required to deploy the OpenShift
Container Platform control plane.
Control plane
A container orchestration layer that exposes the API and interfaces to define, deploy, and manage
the lifecycle of containers. Also known as control plane machines.
Compute node
Nodes that are responsible for executing workloads for cluster users. Also known as worker nodes.
Disconnected installation
In some situations, parts of a data center might not have access to the internet, even through proxy
servers. You can still install the OpenShift Container Platform in these environments, but you must
download the required software and images and make them available to the disconnected
environment.
The OpenShift Container Platform installation program
A program that provisions the infrastructure and deploys a cluster.
Installer-provisioned infrastructure
The installation program deploys and configures the infrastructure that the cluster runs on.
Ignition config files
A file that the Ignition tool uses to configure Red Hat Enterprise Linux CoreOS (RHCOS) during
operating system initialization. The installation program generates different Ignition configuration
files to initialize bootstrap, control plane, and worker nodes.
Kubernetes manifests
Specifications of a Kubernetes API object in a JSON or YAML format. A configuration file can
include deployments, config maps, secrets, daemonsets, and so on.
Kubelet
A primary node agent that runs on each node in the cluster to ensure that containers are running in a
pod.
Load balancers
A load balancer serves as the single point of contact for clients. Load balancers for the API distribute
incoming traffic across control plane nodes.
Machine Config Operator
An Operator that manages and applies configurations and updates of the base operating system and
container runtime, including everything between the kernel and kubelet, for the nodes in the cluster.
Operators
The preferred method of packaging, deploying, and managing a Kubernetes application in an
OpenShift Container Platform cluster. An operator takes human operational knowledge and encodes
it into software that is easily packaged and shared with customers.
User-provisioned infrastructure
You can install OpenShift Container Platform on infrastructure that you provide. You can use the
75
You can install OpenShift Container Platform on infrastructure that you provide. You can use the
installation program to generate the assets required to provision the cluster infrastructure, create
the cluster infrastructure, and then deploy the cluster to the infrastructure that you provided.
1.1.4. Installation process

Except for the Assisted Installer, when you install an OpenShift Container Platform cluster, you must
download the installation program from the appropriate Cluster Type page on the OpenShift Cluster
Manager Hybrid Cloud Console. This console manages:
REST API for accounts.
Registry tokens, which are the pull secrets that you use to obtain the required components.
Cluster registration, which associates the cluster identity to your Red Hat account to facilitate
the gathering of usage metrics.
In OpenShift Container Platform 4.15, the installation program is a Go binary file that performs a series
of file transformations on a set of assets. The way you interact with the installation program differs
depending on your installation type. Consider the following installation use cases:
To deploy a cluster with the Assisted Installer, you must configure the cluster settings by using
the Assisted Installer. There is no installation program to download and configure. After you
finish setting the cluster configuration, you download a discovery ISO and then boot cluster
machines with that image. You can install clusters with the Assisted Installer on Nutanix,
vSphere, and bare metal with full integration, and other platforms without integration. If you
install on bare metal, you must provide all of the cluster infrastructure and resources, including
the networking, load balancing, storage, and individual cluster machines.
To deploy clusters with the Agent-based Installer, you can download the Agent-based Installer
first. You can then configure the cluster and generate a discovery image. You boot cluster
machines with the discovery image, which installs an agent that communicates with the
installation program and handles the provisioning for you instead of you interacting with the
installation program or setting up a provisioner machine yourself. You must provide all of the
cluster infrastructure and resources, including the networking, load balancing, storage, and
individual cluster machines. This approach is ideal for disconnected environments.
For clusters with installer-provisioned infrastructure, you delegate the infrastructure

bootstrapping and provisioning to the installation program instead of doing it yourself. The
installation program creates all of the networking, machines, and operating systems that are
required to support the cluster, except if you install on bare metal. If you install on bare metal,
you must provide all of the cluster infrastructure and resources, including the bootstrap
machine, networking, load balancing, storage, and individual cluster machines.
If you provision and manage the infrastructure for your cluster, you must provide all of the
cluster infrastructure and resources, including the bootstrap machine, networking, load
balancing, storage, and individual cluster machines.
For the installation program, the program uses three sets of files during installation: an installation
configuration file that is named install-config.yaml, Kubernetes manifests, and Ignition config files for
your machine types.
IMPORTANT
76
IMPORTANT
You can modify Kubernetes and the Ignition config files that control the underlying
RHCOS operating system during installation. However, no validation is available to
confirm the suitability of any modifications that you make to these objects. If you modify
these objects, you might render your cluster non-functional. Because of this risk,
modifying Kubernetes and Ignition config files is not supported unless you are following
documented procedures or are instructed to do so by Red Hat support.
The installation configuration file is transformed into Kubernetes manifests, and then the manifests are
wrapped into Ignition config files. The installation program uses these Ignition config files to create the
cluster.
The installation configuration files are all pruned when you run the installation program, so be sure to
back up all the configuration files that you want to use again.
IMPORTANT
You cannot modify the parameters that you set during installation, but you can modify
many cluster attributes after installation.
The installation process with the Assisted Installer

Installation with the Assisted Installer involves creating a cluster configuration interactively by using the
web-based user interface or the RESTful API. The Assisted Installer user interface prompts you for
required values and provides reasonable default values for the remaining parameters, unless you change
them in the user interface or with the API. The Assisted Installer generates a discovery image, which you
download and use to boot the cluster machines. The image installs RHCOS and an agent, and the agent
handles the provisioning for you. You can install OpenShift Container Platform with the Assisted
Installer and full integration on Nutanix, vSphere, and bare metal. Additionally, you can install OpenShift
Container Platform with the Assisted Installer on other platforms without integration.
OpenShift Container Platform manages all aspects of the cluster, including the operating system itself.
Each machine boots with a configuration that references resources hosted in the cluster that it joins.
This configuration allows the cluster to manage itself as updates are applied.
If possible, use the Assisted Installer feature to avoid having to download and configure the Agent-
based Installer.
The installation process with Agent-based infrastructure

Agent-based installation is similar to using the Assisted Installer, except that you must initially download
and install the Agent-based Installer . An Agent-based installation is useful when you want the
convenience of the Assisted Installer, but you need to install a cluster in a disconnected environment.
If possible, use the Agent-based installation feature to avoid having to create a provisioner machine
with a bootstrap VM, and then provision and maintain the cluster infrastructure.
The installation process with installer-provisioned infrastructure

The default installation type uses installer-provisioned infrastructure. By default, the installation
program acts as an installation wizard, prompting you for values that it cannot determine on its own and
providing reasonable default values for the remaining parameters. You can also customize the
installation process to support advanced infrastructure scenarios. The installation program provisions
the underlying infrastructure for the cluster.
You can install either a standard cluster or a customized cluster. With a standard cluster, you provide
minimum details that are required to install the cluster. With a customized cluster, you can specify more
details about the platform, such as the number of machines that the control plane uses, the type of
77
virtual machine that the cluster deploys, or the CIDR range for the Kubernetes service network.
If possible, use this feature to avoid having to provision and maintain the cluster infrastructure. In all
other environments, you use the installation program to generate the assets that you require to
provision your cluster infrastructure.
With installer-provisioned infrastructure clusters, OpenShift Container Platform manages all aspects of
the cluster, including the operating system itself. Each machine boots with a configuration that
references resources hosted in the cluster that it joins. This configuration allows the cluster to manage
itself as updates are applied.
The installation process with user-provisioned infrastructure

You can also install OpenShift Container Platform on infrastructure that you provide. You use the
installation program to generate the assets that you require to provision the cluster infrastructure,
create the cluster infrastructure, and then deploy the cluster to the infrastructure that you provided.
If you do not use infrastructure that the installation program provisioned, you must manage and
maintain the cluster resources yourself. The following list details some of these self-managed resources:
The underlying infrastructure for the control plane and compute machines that make up the
cluster
Load balancers
Cluster networking, including the DNS records and required subnets
Storage for the cluster infrastructure and applications
If your cluster uses user-provisioned infrastructure, you have the option of adding RHEL compute
machines to your cluster.
Installation process details

When a cluster is provisioned, each machine in the cluster requires information about the cluster.
OpenShift Container Platform uses a temporary bootstrap machine during initial configuration to
provide the required information to the permanent control plane. The temporary bootstrap machine
boots by using an Ignition config file that describes how to create the cluster. The bootstrap machine
creates the control plane machines that make up the control plane. The control plane machines then
create the compute machines, which are also known as worker machines. The following figure illustrates
this process:
Figure 1.2. Creating the bootstrap, control plane, and compute machines
78
Figure 1.2. Creating the bootstrap, control plane, and compute machines
After the cluster machines initialize, the bootstrap machine is destroyed. All clusters use the bootstrap
process to initialize the cluster, but if you provision the infrastructure for your cluster, you must complete
many of the steps manually.
IMPORTANT
The Ignition config files that the installation program generates contain
certificates that expire after 24 hours, which are then renewed at that time. If the
cluster is shut down before renewing the certificates and the cluster is later
restarted after the 24 hours have elapsed, the cluster automatically recovers the
expired certificates. The exception is that you must manually approve the
pending node-bootstrapper certificate signing requests (CSRs) to recover
kubelet certificates. See the documentation for Recovering from expired control
plane certificates for more information.
Consider using Ignition config files within 12 hours after they are generated,
because the 24-hour certificate rotates from 16 to 22 hours after the cluster is
installed. By using the Ignition config files within 12 hours, you can avoid
installation failure if the certificate update runs during installation.
Bootstrapping a cluster involves the following steps:
1. The bootstrap machine boots and starts hosting the remote resources required for the control
plane machines to boot. If you provision the infrastructure, this step requires manual
intervention.
2. The bootstrap machine starts a single-node etcd cluster and a temporary Kubernetes control
plane.
3. The control plane machines fetch the remote resources from the bootstrap machine and finish
booting. If you provision the infrastructure, this step requires manual intervention.
79
4. The temporary control plane schedules the production control plane to the production control
plane machines.
5. The Cluster Version Operator (CVO) comes online and installs the etcd Operator. The etcd
Operator scales up etcd on all control plane nodes.
6. The temporary control plane shuts down and passes control to the production control plane.
7. The bootstrap machine injects OpenShift Container Platform components into the production
control plane.
8. The installation program shuts down the bootstrap machine. If you provision the infrastructure,
this step requires manual intervention.
9. The control plane sets up the compute nodes.
10. The control plane installs additional services in the form of a set of Operators.
The result of this bootstrapping process is a running OpenShift Container Platform cluster. The cluster
then downloads and configures remaining components needed for the day-to-day operations, including
the creation of compute machines in supported environments.
Additional resources
Red Hat OpenShift Network Calculator
1.1.5. Verifying node state after installation

The OpenShift Container Platform installation completes when the following installation health checks
are successful:
The provisioner can access the OpenShift Container Platform web console.
All control plane nodes are ready.
All cluster Operators are available.
NOTE
After the installation completes, the specific cluster Operators responsible for the worker
nodes continuously attempt to provision all worker nodes. Some time is required before
all worker nodes report as READY. For installations on bare metal, wait a minimum of 60
minutes before troubleshooting a worker node. For installations on all other platforms,
wait a minimum of 40 minutes before troubleshooting a worker node. A DEGRADED
state for the cluster Operators responsible for the worker nodes depends on the
Operators' own resources and not on the state of the nodes.
After your installation completes, you can continue to monitor the condition of the nodes in your cluster.
Prerequisites
The installation program resolves successfully in the terminal.
Procedure
80
1. Show the status of all worker nodes:
$ oc get nodes
Example output
NAME STATUS ROLES AGE VERSION

example-compute1.example.com Ready worker 13m v1.21.6+bb8d50a
example-control1.example.com Ready master 52m v1.21.6+bb8d50a
2. Show the phase of all worker machine nodes:
$ oc get machines -A
Example output
NAMESPACE NAME PHASE TYPE REGION ZONE AGE

openshift-machine-api example-zbbt6-master-0 Running 95m
openshift-machine-api example-zbbt6-worker-0-25bhp Running 49m
openshift-machine-api example-zbbt6-worker-0-8b4c2 Running 49m
openshift-machine-api example-zbbt6-worker-0-jkbqt Running 49m
openshift-machine-api example-zbbt6-worker-0-qrl5b Running 49m
Getting the BareMetalHost resource
Following the installation
Validating an installation
Agent-based Installer
Assisted Installer for OpenShift Container Platform
Installation scope
The scope of the OpenShift Container Platform installation program is intentionally narrow. It is
designed for simplicity and ensured success. You can complete many more configuration tasks after
installation completes.
See Available cluster customizations for details about OpenShift Container Platform
configuration resources.
1.1.6. OpenShift Local overview
OpenShift Local supports rapid application development to get started building OpenShift Container
81
OpenShift Local supports rapid application development to get started building OpenShift Container
Platform clusters. OpenShift Local is designed to run on a local computer to simplify setup and testing,
and to emulate the cloud development environment locally with all of the tools needed to develop
container-based applications.
Regardless of the programming language you use, OpenShift Local hosts your application and brings a
minimal, preconfigured Red Hat OpenShift Container Platform cluster to your local PC without the need
for a server-based infrastructure.
On a hosted environment, OpenShift Local can create microservices, convert them into images, and run
them in Kubernetes-hosted containers directly on your laptop or desktop running Linux, macOS, or
Windows 10 or later.
For more information about OpenShift Local, see Red Hat OpenShift Local Overview .
1.2. SUPPORTED PLATFORMS FOR OPENSHIFT CONTAINER

PLATFORM CLUSTERS
In OpenShift Container Platform 4.15, you can install a cluster that uses installer-provisioned
infrastructure on the following platforms:
Alibaba Cloud
Amazon Web Services (AWS)
Bare metal
Google Cloud Platform (GCP)
IBM Cloud®
Microsoft Azure
Microsoft Azure Stack Hub
Nutanix
Red Hat OpenStack Platform (RHOSP)
The latest OpenShift Container Platform release supports both the latest RHOSP long-life
release and intermediate release. For complete RHOSP release compatibility, see the
OpenShift Container Platform on RHOSP support matrix .
VMware vSphere
For these clusters, all machines, including the computer that you run the installation process on, must
have direct internet access to pull images for platform containers and provide telemetry data to Red
Hat.
IMPORTANT
82
IMPORTANT
After installation, the following changes are not supported:
Mixing cloud provider platforms.
Mixing cloud provider components. For example, using a persistent storage

framework from a another platform on the platform where you installed the
cluster.
In OpenShift Container Platform 4.15, you can install a cluster that uses user-provisioned infrastructure
on the following platforms:
AWS
Azure
Azure Stack Hub
Bare metal
GCP
IBM Power®
IBM Z® or IBM® LinuxONE
RHOSP
The latest OpenShift Container Platform release supports both the latest RHOSP long-life
release and intermediate release. For complete RHOSP release compatibility, see the
OpenShift Container Platform on RHOSP support matrix .
VMware Cloud on AWS
VMware vSphere
Depending on the supported cases for the platform, you can perform installations on user-provisioned
infrastructure, so that you can run machines with full internet access, place your cluster behind a proxy,
or perform a disconnected installation.
In a disconnected installation, you can download the images that are required to install a cluster, place
them in a mirror registry, and use that data to install your cluster. While you require internet access to
pull images for platform containers, with a disconnected installation on vSphere or bare metal
infrastructure, your cluster machines do not require direct internet access.
The OpenShift Container Platform 4.x Tested Integrations page contains details about integration
testing for different platforms.
See Supported installation methods for different platforms for more information about the
types of installations that are available for each supported platform.
See Selecting a cluster installation method and preparing it for users for information about
choosing an installation method and preparing the required resources.
Red Hat OpenShift Network Calculator can help you design your cluster network during both
83
Red Hat OpenShift Network Calculator can help you design your cluster network during both
the deployment and expansion phases. It addresses common questions related to the cluster
network and provides output in a convenient JSON format.
84
CHAPTER 2. SELECTING A CLUSTER INSTALLATION METHOD AND PREPARING IT FOR USERS
CHAPTER 2. SELECTING A CLUSTER INSTALLATION

METHOD AND PREPARING IT FOR USERS
Before you install OpenShift Container Platform, decide what kind of installation process to follow and
verify that you have all of the required resources to prepare the cluster for users.
2.1. SELECTING A CLUSTER INSTALLATION TYPE

Before you install an OpenShift Container Platform cluster, you need to select the best installation
instructions to follow. Think about your answers to the following questions to select the best option.
2.1.1. Do you want to install and manage an OpenShift Container Platform cluster
yourself?
If you want to install and manage OpenShift Container Platform yourself, you can install it on the
following platforms:
Alibaba Cloud
Amazon Web Services (AWS) on 64-bit x86 instances
Amazon Web Services (AWS) on 64-bit ARM instances
Microsoft Azure on 64-bit x86 instances
Microsoft Azure on 64-bit ARM instances
Microsoft Azure Stack Hub
Google Cloud Platform (GCP) on 64-bit x86 instances
Google Cloud Platform (GCP) on 64-bit ARM instances
Red Hat OpenStack Platform (RHOSP)
IBM Cloud®
IBM Z® or IBM® LinuxONE
IBM Z® or IBM® LinuxONE for Red Hat Enterprise Linux (RHEL) KVM
IBM Power®
IBM Power® Virtual Server
Nutanix
VMware vSphere
Bare metal or other platform agnostic infrastructure
You can deploy an OpenShift Container Platform 4 cluster to both on-premise hardware and to cloud
hosting services, but all of the machines in a cluster must be in the same data center or cloud hosting
service.
85
If you want to use OpenShift Container Platform but do not want to manage the cluster yourself, you
have several managed service options. If you want a cluster that is fully managed by Red Hat, you can use
OpenShift Dedicated or OpenShift Online. You can also use OpenShift as a managed service on Azure,
AWS, IBM Cloud®, or Google Cloud. For more information about managed services, see the OpenShift
Products page. If you install an OpenShift Container Platform cluster with a cloud virtual machine as a
virtual bare metal, the corresponding cloud-based storage is not supported.
2.1.2. Have you used OpenShift Container Platform 3 and want to use OpenShift
Container Platform 4?
If you used OpenShift Container Platform 3 and want to try OpenShift Container Platform 4, you need
to understand how different OpenShift Container Platform 4 is. OpenShift Container Platform 4 weaves
the Operators that package, deploy, and manage Kubernetes applications and the operating system
that the platform runs on, Red Hat Enterprise Linux CoreOS (RHCOS), together seamlessly. Instead of
deploying machines and configuring their operating systems so that you can install OpenShift Container
Platform on them, the RHCOS operating system is an integral part of the OpenShift Container Platform
cluster. Deploying the operating system for the cluster machines is part of the installation process for
OpenShift Container Platform. See Differences between OpenShift Container Platform 3 and 4 .
Because you need to provision machines as part of the OpenShift Container Platform cluster installation
process, you cannot upgrade an OpenShift Container Platform 3 cluster to OpenShift Container
Platform 4. Instead, you must create a new OpenShift Container Platform 4 cluster and migrate your
OpenShift Container Platform 3 workloads to them. For more information about migrating, see
Migrating from OpenShift Container Platform 3 to 4 overview . Because you must migrate to OpenShift
Container Platform 4, you can use any type of production cluster installation process to create your new
cluster.
2.1.3. Do you want to use existing components in your cluster?

Because the operating system is integral to OpenShift Container Platform, it is easier to let the
installation program for OpenShift Container Platform stand up all of the infrastructure. These are
called installer provisioned infrastructure installations. In this type of installation, you can provide some
existing infrastructure to the cluster, but the installation program deploys all of the machines that your
cluster initially needs.
You can deploy an installer-provisioned infrastructure cluster without specifying any customizations to
the cluster or its underlying machines to Alibaba Cloud, AWS, Azure, Azure Stack Hub , GCP, Nutanix.
If you need to perform basic configuration for your installer-provisioned infrastructure cluster, such as
the instance type for the cluster machines, you can customize an installation for Alibaba Cloud, AWS,
Azure, GCP, Nutanix.
For installer-provisioned infrastructure installations, you can use an existing VPC in AWS, vNet in Azure ,
or VPC in GCP. You can also reuse part of your networking infrastructure so that your cluster in AWS,
Azure, GCP can coexist with existing IP address allocations in your environment and integrate with
existing MTU and VXLAN configurations. If you have existing accounts and credentials on these clouds,
you can re-use them, but you might need to modify the accounts to have the required permissions to
install OpenShift Container Platform clusters on them.
You can use the installer-provisioned infrastructure method to create appropriate machine instances on
your hardware for vSphere, and bare metal. Additionally, for vSphere, you can also customize additional
network parameters during installation.
If you want to reuse extensive cloud infrastructure, you can complete a user-provisioned infrastructure
installation. With these installations, you manually deploy the machines that your cluster requires during
the installation process. If you perform a user-provisioned infrastructure installation on AWS, Azure,
86
Azure Stack Hub , you can use the provided templates to help you stand up all of the required
components. You can also reuse a shared VPC on GCP. Otherwise, you can use the provider-agnostic
installation method to deploy a cluster into other clouds.
You can also complete a user-provisioned infrastructure installation on your existing hardware. If you
use RHOSP, IBM Z® or IBM® LinuxONE , IBM Z® and IBM® LinuxONE with RHEL KVM , IBM Power, or
vSphere, use the specific installation instructions to deploy your cluster. If you use other supported
hardware, follow the bare metal installation procedure. For some of these platforms, such as vSphere,
and bare metal, you can also customize additional network parameters during installation.
2.1.4. Do you need extra security for your cluster?

If you use a user-provisioned installation method, you can configure a proxy for your cluster. The
instructions are included in each installation procedure.
If you want to prevent your cluster on a public cloud from exposing endpoints externally, you can deploy
a private cluster with installer-provisioned infrastructure on AWS, Azure, or GCP.
If you need to install your cluster that has limited access to the internet, such as a disconnected or
restricted network cluster, you can mirror the installation packages and install the cluster from them.
Follow detailed instructions for user provisioned infrastructure installations into restricted networks for
AWS, GCP, IBM Z® or IBM® LinuxONE , IBM Z® or IBM® LinuxONE with RHEL KVM , IBM Power® ,
vSphere, or bare metal. You can also install a cluster into a restricted network using installer-provisioned
infrastructure by following detailed instructions for AWS, GCP, IBM Cloud® , Nutanix, RHOSP, and
vSphere.
If you need to deploy your cluster to an AWS GovCloud region, AWS China region, or Azure government
region, you can configure those custom regions during an installer-provisioned infrastructure
installation.
You can also configure the cluster machines to use the RHEL cryptographic libraries that have been
submitted to NIST for FIPS 140-2/140-3 Validation during installation.
IMPORTANT
When running Red Hat Enterprise Linux (RHEL) or Red Hat Enterprise Linux CoreOS
(RHCOS) booted in FIPS mode, OpenShift Container Platform core components use the
RHEL cryptographic libraries that have been submitted to NIST for FIPS 140-2/140-3
Validation on only the x86_64, ppc64le, and s390x architectures.
2.2. PREPARING YOUR CLUSTER FOR USERS AFTER INSTALLATION

Some configuration is not required to install the cluster but recommended before your users access the
cluster. You can customize the cluster itself by customizing the Operators that make up your cluster and
integrate you cluster with other required systems, such as an identity provider.
For a production cluster, you must configure the following integrations:
Persistent storage
An identity provider
Monitoring core OpenShift Container Platform components
2.3. PREPARING YOUR CLUSTER FOR WORKLOADS
87
Depending on your workload needs, you might need to take extra steps before you begin deploying
applications. For example, after you prepare infrastructure to support your application build strategy,
you might need to make provisions for low-latency workloads or to protect sensitive workloads . You can
also configure monitoring for application workloads. If you plan to run Windows workloads, you must
enable hybrid networking with OVN-Kubernetes during the installation process; hybrid networking
cannot be enabled after your cluster is installed.
2.4. SUPPORTED INSTALLATION METHODS FOR DIFFERENT

PLATFORMS
You can perform different types of installations on different platforms.
NOTE
Not all installation options are supported for all platforms, as shown in the following
tables. A checkmark indicates that the option is supported and links to the relevant
section.
Table 2.1. Installer-provisioned infrastructure options
Ali A A Az Az Az G G N R Ba Ba vS IB IB IB IB
ba W W ur ur ur C C ut H re re ph M M M M
ba S S e e e P P an O m m er Cl Z® P P
(6 (6 (6 (6 St (6 (6 ix S et et e ou o o
4- 4- 4- 4- ac 4- 4- P al al d® w w
bit bit bi bi k bi bi (6 (6 er er
x8 A t t H t t 4- 4- ® ®
6) R x8 A ub x8 A bi bi Vi
M) 6) R 6) R t t rt
M M x8 A ua
) ) 6) R l
M Se
) rv
er
D ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
ef
au
lt
C ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
us
to
m
88
(6 (6 (6 (6 St (6 (6 ix S et et e ou o o
4- 4- 4- 4- ac 4- 4- P al al d® w w
x8 A t t H t t 4- 4- ® ®
M) 6) R 6) R t t rt
M M x8 A ua
) ) 6) R l
M Se
) rv
er
N ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
et
w
or
k
cu
st
o
mi
za
tio
n
Re ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
str
ict
ed
ne
tw
or
k
Pri ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
va
te
cl
us
te
rs
89
(6 (6 (6 (6 St (6 (6 ix S et et e ou o o
4- 4- 4- 4- ac 4- 4- P al al d® w w
x8 A t t H t t 4- 4- ® ®
M) 6) R 6) R t t rt
M M x8 A ua
) ) 6) R l
M Se
) rv
er
Ex ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
ist
in
g
vir
tu
al
pri
va
te
ne
tw
or
ks
G ✓ ✓
ov
er
n
m
en
t
re
gi
on
s
Se ✓
cr
et
re
gi
on
s
90
(6 (6 (6 (6 St (6 (6 ix S et et e ou o o
4- 4- 4- 4- ac 4- 4- P al al d® w w
x8 A t t H t t 4- 4- ® ®
M) 6) R 6) R t t rt
M M x8 A ua
) ) 6) R l
M Se
) rv
er
C ✓
hi
na
re
gi
on
s
Table 2.2. User-provisioned infrastructure options
Al A A A A A G G N R B B vS IB IB IB IB Pl
ib W W zu zu zu C C ut H ar ar p M M M M at
a S S re re re P P an O e e h Cl Z Z P fo
b (6 (6 (6 (6 St (6 (6 ix S m m er o ® ® o r
a 4- 4- 4- 4- ac 4- 4- P et et e u wi w m
bi bi bi bi k bi bi al al d th er a
t t t t H t t (6 (6 ® R ® g
x8 A x8 A u x8 A 4- 4- H n
6) R 6) R b 6) R bi bi E o
M M M t t L st
) ) ) x8 A K ic
6) R V
M M
)
C ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
us
to
m
91
b (6 (6 (6 (6 St (6 (6 ix S m m er o ® ® o r
a 4- 4- 4- 4- ac 4- 4- P et et e u wi w m
t t t t H t t (6 (6 ® R ® g
x8 A x8 A u x8 A 4- 4- H n
6) R 6) R b 6) R bi bi E o
M M M t t L st
) ) ) x8 A K ic
6) R V
M M
)
N ✓ ✓ ✓
et
w
or
k
c
us
to
mi
za
ti
o
n
R ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
es
tri
ct
e
d
n
et
w
or
k
92
b (6 (6 (6 (6 St (6 (6 ix S m m er o ® ® o r
a 4- 4- 4- 4- ac 4- 4- P et et e u wi w m
t t t t H t t (6 (6 ® R ® g
x8 A x8 A u x8 A 4- 4- H n
6) R 6) R b 6) R bi bi E o
M M M t t L st
) ) ) x8 A K ic
6) R V
M M
)
S ✓ ✓
h
ar
e
d
V
P
C
h
os
te
d
o
ut
si
d
e
of
cl
us
te
r
pr
oj
e
ct
93
CHAPTER 3. CLUSTER CAPABILITIES

Cluster administrators can use cluster capabilities to enable or disable optional components prior to
installation. Cluster administrators can enable cluster capabilities at anytime after installation.
NOTE
Cluster administrators cannot disable a cluster capability after it is enabled.
3.1. SELECTING CLUSTER CAPABILITIES

You can select cluster capabilities by following one of the installation methods that include customizing
your cluster, such as "Installing a cluster on AWS with customizations" or "Installing a cluster on GCP with
customizations".
During a customized installation, you create an install-config.yaml file that contains the configuration
parameters for your cluster.
NOTE
If you customize your cluster by enabling or disabling specific cluster capabilities, you are
responsible for manually maintaining your install-config.yaml file. New OpenShift
Container Platform updates might declare new capability handles for existing
components, or introduce new components altogether. Users who customize their
install-config.yaml file should consider periodically updating their install-config.yaml
file as OpenShift Container Platform is updated.
You can use the following configuration parameters to select cluster capabilities:
capabilities:
baselineCapabilitySet: v4.11 1
additionalEnabledCapabilities: 2
- CSISnapshot
- Console
- Storage
1 Defines a baseline set of capabilities to install. Valid values are None, vCurrent and v4.x. If you
select None, all optional capabilities will be disabled. The default value is vCurrent, which enables
all optional capabilities.
NOTE
v4.x refers to any value up to and including the current cluster version. For example,
valid values for a OpenShift Container Platform 4.12 cluster are v4.11 and v4.12.
2 Defines a list of capabilities to explicitly enable. These will be enabled in addition to the capabilities
specified in baselineCapabilitySet.
NOTE
94
NOTE
In this example, the default capability is set to v4.11. The

additionalEnabledCapabilities field enables additional capabilities over the default
v4.11 capability set.
The following table describes the baselineCapabilitySet values.
Table 3.1. Cluster capabilities baselineCapabilitySet values description
Value Description
vCurrent Specify this option when you want to automatically add new,
default capabilities that are introduced in new releases.
v4.11 Specify this option when you want to enable the default
capabilities for OpenShift Container Platform 4.11. By specifying
v4.11, capabilities that are introduced in newer versions of
OpenShift Container Platform are not enabled. The default
capabilities in OpenShift Container Platform 4.11 are
baremetal, MachineAPI, marketplace, and openshift-
samples.
baremetal, MachineAPI, marketplace, openshift-
samples, Console, Insights, Storage, and CSISnapshot.
samples, Console, Insights, Storage, CSISnapshot, and
NodeTuning.
samples, Console, Insights, Storage, CSISnapshot,
NodeTuning, ImageRegistry , Build, and
DeploymentConfig .
95
Value Description
baremetal, MachineAPI, marketplace,
OperatorLifecycleManager , openshift-samples,
Console, Insights, Storage, CSISnapshot, NodeTuning,
ImageRegistry , Build, CloudCredential, and
DeploymentConfig .
None Specify when the other sets are too large, and you do not need
any capabilities or want to fine-tune via
additionalEnabledCapabilities.
Installing a cluster on AWS with customizations
Installing a cluster on GCP with customizations
3.2. OPTIONAL CLUSTER CAPABILITIES IN OPENSHIFT CONTAINER

PLATFORM 4.15
Currently, cluster Operators provide the features for these optional capabilities. The following
summarizes the features provided by each capability and what functionality you lose if it is disabled.
Cluster Operators reference
3.2.1. Bare-metal capability

Purpose
The Cluster Baremetal Operator provides the features for the baremetal capability.
The Cluster Baremetal Operator (CBO) deploys all the components necessary to take a bare-metal
server to a fully functioning worker node ready to run OpenShift Container Platform compute nodes.
The CBO ensures that the metal3 deployment, which consists of the Bare Metal Operator (BMO) and
Ironic containers, runs on one of the control plane nodes within the OpenShift Container Platform
cluster. The CBO also listens for OpenShift Container Platform updates to resources that it watches
and takes appropriate action.
The bare-metal capability is required for deployments using installer-provisioned infrastructure.

Disabling the bare-metal capability can result in unexpected problems with these deployments.
It is recommended that cluster administrators only disable the bare-metal capability during installations
with user-provisioned infrastructure that do not have any BareMetalHost resources in the cluster.
IMPORTANT
96
IMPORTANT
If the bare-metal capability is disabled, the cluster cannot provision or manage bare-
metal nodes. Only disable the capability if there are no BareMetalHost resources in your
deployment. The baremetal capability depends on the MachineAPI capability. If you
enable the baremetal capability, you must also enable MachineAPI.
Deploying installer-provisioned clusters on bare metal
Preparing for bare metal cluster installation
Bare metal configuration
3.2.2. Build capability

Purpose
The Build capability enables the Build API. The Build API manages the lifecycle of Build and
BuildConfig objects.
IMPORTANT
If the Build capability is disabled, the cluster cannot use Build or BuildConfig resources.
Disable the capability only if Build and BuildConfig resources are not required in the
cluster.
3.2.3. Cloud credential capability

Purpose
The Cloud Credential Operator provides features for the CloudCredential capability.
NOTE
Currently, disabling the CloudCredential capability is only supported for bare-metal

clusters.
The Cloud Credential Operator (CCO) manages cloud provider credentials as Kubernetes custom
resource definitions (CRDs). The CCO syncs on CredentialsRequest custom resources (CRs) to allow
OpenShift Container Platform components to request cloud provider credentials with the specific
permissions that are required for the cluster to run.
By setting different values for the credentialsMode parameter in the install-config.yaml file, the CCO
can be configured to operate in several different modes. If no mode is specified, or the
credentialsMode parameter is set to an empty string ( ""), the CCO operates in its default mode.
About the Cloud Credential Operator
3.2.4. Cluster storage capability

Purpose
The Cluster Storage Operator provides the features for the Storage capability.
97
The Cluster Storage Operator sets OpenShift Container Platform cluster-wide storage defaults. It
ensures a default storageclass exists for OpenShift Container Platform clusters. It also installs
Container Storage Interface (CSI) drivers which enable your cluster to use various storage backends.
IMPORTANT
If the cluster storage capability is disabled, the cluster will not have a default
storageclass or any CSI drivers. Users with administrator privileges can create a default
storageclass and manually install CSI drivers if the cluster storage capability is disabled.
Notes
The storage class that the Operator creates can be made non-default by editing its annotation,
but this storage class cannot be deleted as long as the Operator runs.
3.2.5. Console capability

Purpose
The Console Operator provides the features for the Console capability.
The Console Operator installs and maintains the OpenShift Container Platform web console on a
cluster. The Console Operator is installed by default and automatically maintains a console.
Web console overview
3.2.6. CSI snapshot controller capability

Purpose
The Cluster CSI Snapshot Controller Operator provides the features for the CSISnapshot capability.
The Cluster CSI Snapshot Controller Operator installs and maintains the CSI Snapshot Controller. The
CSI Snapshot Controller is responsible for watching the VolumeSnapshot CRD objects and manages
the creation and deletion lifecycle of volume snapshots.
CSI volume snapshots
3.2.7. DeploymentConfig capability

Purpose
The DeploymentConfig capability enables and manages the DeploymentConfig API.
IMPORTANT
98
IMPORTANT
If you disable the DeploymentConfig capability, the following resources will not be
available in the cluster:
DeploymentConfig resources
The deployer service account
Disable the DeploymentConfig capability only if you do not require DeploymentConfig

resources and the deployer service account in the cluster.
3.2.8. Insights capability

Purpose
The Insights Operator provides the features for the Insights capability.
The Insights Operator gathers OpenShift Container Platform configuration data and sends it to Red
Hat. The data is used to produce proactive insights recommendations about potential issues that a
cluster might be exposed to. These insights are communicated to cluster administrators through
Insights Advisor on console.redhat.com.
Notes
Insights Operator complements OpenShift Container Platform Telemetry.
Using Insights Operator
3.2.9. Machine API capability

Purpose
The machine-api-operator, cluster-autoscaler-operator, and cluster-control-plane-machine-set-
operator Operators provide the features for the MachineAPI capability. You can disable this capability
only if you install a cluster with user-provisioned infrastructure.
The Machine API capability is responsible for all machine configuration and management in the cluster. If
you disable the Machine API capability during installation, you need to manage all machine-related tasks
manually.
Overview of machine management
Machine API Operator
Cluster Autoscaler Operator
Control Plane Machine Set Operator
3.2.10. Marketplace capability

Purpose
The Marketplace Operator provides the features for the marketplace capability.
The Marketplace Operator simplifies the process for bringing off-cluster Operators to your cluster by
99
The Marketplace Operator simplifies the process for bringing off-cluster Operators to your cluster by
using a set of default Operator Lifecycle Manager (OLM) catalogs on the cluster. When the
Marketplace Operator is installed, it creates the openshift-marketplace namespace. OLM ensures
catalog sources installed in the openshift-marketplace namespace are available for all namespaces on
the cluster.
If you disable the marketplace capability, the Marketplace Operator does not create the openshift-
marketplace namespace. Catalog sources can still be configured and managed on the cluster manually,
but OLM depends on the openshift-marketplace namespace in order to make catalogs available to all
namespaces on the cluster. Users with elevated permissions to create namespaces prefixed with
openshift-, such as system or cluster administrators, can manually create the openshift-marketplace
namespace.
If you enable the marketplace capability, you can enable and disable individual catalogs by configuring
the Marketplace Operator.
Red Hat-provided Operator catalogs
3.2.11. Operator Lifecycle Manager capability

Purpose
Operator Lifecycle Manager (OLM) helps users install, update, and manage the lifecycle of Kubernetes
native applications (Operators) and their associated services running across their OpenShift Container
Platform clusters. It is part of the Operator Framework, an open source toolkit designed to manage
Operators in an effective, automated, and scalable way.
If an Operator requires any of the following APIs, then you must enable the OperatorLifecycleManager
capability:
ClusterServiceVersion
CatalogSource
Subscription
InstallPlan
OperatorGroup
IMPORTANT
The marketplace capability depends on the OperatorLifecycleManager capability. You

cannot disable the OperatorLifecycleManager capability and enable the marketplace
capability.
Operator Lifecycle Manager concepts and resources
3.2.12. Node Tuning capability

Purpose
The Node Tuning Operator provides features for the NodeTuning capability.
100
The Node Tuning Operator helps you manage node-level tuning by orchestrating the TuneD daemon
and achieves low latency performance by using the Performance Profile controller. The majority of high-
performance applications require some level of kernel tuning. The Node Tuning Operator provides a
unified management interface to users of node-level sysctls and more flexibility to add custom tuning
specified by user needs.
If you disable the NodeTuning capability, some default tuning settings will not be applied to the control-
plane nodes. This might limit the scalability and performance of large clusters with over 900 nodes or
900 routes.
Using the Node Tuning Operator
3.2.13. OpenShift samples capability

Purpose
The Cluster Samples Operator provides the features for the openshift-samples capability.
The Cluster Samples Operator manages the sample image streams and templates stored in the
openshift namespace.
On initial start up, the Operator creates the default samples configuration resource to initiate the
creation of the image streams and templates. The configuration object is a cluster scoped object with
the key cluster and type configs.samples.
The image streams are the Red Hat Enterprise Linux CoreOS (RHCOS)-based OpenShift Container
Platform image streams pointing to images on registry.redhat.io. Similarly, the templates are those
categorized as OpenShift Container Platform templates.
If you disable the samples capability, users cannot access the image streams, samples, and templates it
provides. Depending on your deployment, you might want to disable this component if you do not need
it.
Configuring the Cluster Samples Operator
3.2.14. Cluster Image Registry capability

Purpose
The Cluster Image Registry Operator provides features for the ImageRegistry capability.
The Cluster Image Registry Operator manages a singleton instance of the OpenShift image registry. It
manages all configuration of the registry, including creating storage.
On initial start up, the Operator creates a default image-registry resource instance based on the
configuration detected in the cluster. This indicates what cloud storage type to use based on the cloud
provider.
If insufficient information is available to define a complete image-registry resource, then an incomplete

resource is defined and the Operator updates the resource status with information about what is
missing.
The Cluster Image Registry Operator runs in the openshift-image-registry namespace and it also
101
The Cluster Image Registry Operator runs in the openshift-image-registry namespace and it also
manages the registry instance in that location. All configuration and workload resources for the registry
reside in that namespace.
In order to integrate the image registry into the cluster’s user authentication and authorization system, a
service account token secret and an image pull secret are generated for each service account in the
cluster.
IMPORTANT
If you disable the ImageRegistry capability or if you disable the integrated OpenShift
image registry in the Cluster Image Registry Operator’s configuration, the service
account token secret and image pull secret are not generated for each service account.
If you disable the ImageRegistry capability, you can reduce the overall resource footprint of OpenShift
Container Platform in resource-constrained environments. Depending on your deployment, you can
disable this component if you do not need it.
Project
cluster-image-registry-operator
Image Registry Operator in OpenShift Container Platform
Automatically generated secrets
3.3. ADDITIONAL RESOURCES

Enabling cluster capabilities after installation
102
CHAPTER 4. DISCONNECTED INSTALLATION MIRRORING
4.1. ABOUT DISCONNECTED INSTALLATION MIRRORING

You can use a mirror registry to ensure that your clusters only use container images that satisfy your
organizational controls on external content. Before you install a cluster on infrastructure that you
provision in a restricted network, you must mirror the required container images into that environment.
To mirror container images, you must have a registry for mirroring.
4.1.1. Creating a mirror registry

If you already have a container image registry, such as Red Hat Quay, you can use it as your mirror
registry. If you do not already have a registry, you can create a mirror registry using the mirror registry for
Red Hat OpenShift.
4.1.2. Mirroring images for a disconnected installation

You can use one of the following procedures to mirror your OpenShift Container Platform image
repository to your mirror registry:
Mirroring images for a disconnected installation
Mirroring images for a disconnected installation using the oc-mirror plugin
4.2. CREATING A MIRROR REGISTRY WITH MIRROR REGISTRY FOR

RED HAT OPENSHIFT
The mirror registry for Red Hat OpenShift is a small and streamlined container registry that you can use
as a target for mirroring the required container images of OpenShift Container Platform for
disconnected installations.
If you already have a container image registry, such as Red Hat Quay, you can skip this section and go
straight to Mirroring the OpenShift Container Platform image repository .
4.2.1. Prerequisites
An OpenShift Container Platform subscription.
Red Hat Enterprise Linux (RHEL) 8 and 9 with Podman 3.4.2 or later and OpenSSL installed.
Fully qualified domain name for the Red Hat Quay service, which must resolve through a DNS
server.
Key-based SSH connectivity on the target host. SSH keys are automatically generated for local
installs. For remote hosts, you must generate your own SSH keys.
2 or more vCPUs.
8 GB of RAM.
About 12 GB for OpenShift Container Platform 4.15 release images, or about 358 GB for
OpenShift Container Platform 4.15 release images and OpenShift Container Platform 4.15 Red
Hat Operator images. Up to 1 TB per stream or more is suggested.
IMPORTANT
103
IMPORTANT
These requirements are based on local testing results with only release images
and Operator images. Storage requirements can vary based on your
organization’s needs. You might require more space, for example, when you
mirror multiple z-streams. You can use standard Red Hat Quay functionality or
the proper API callout to remove unnecessary images and free up space.
4.2.2. Mirror registry for Red Hat OpenShift introduction

For disconnected deployments of OpenShift Container Platform, a container registry is required to carry
out the installation of the clusters. To run a production-grade registry service on such a cluster, you
must create a separate registry deployment to install the first cluster. The mirror registry for Red Hat
OpenShift addresses this need and is included in every OpenShift subscription. It is available for
download on the OpenShift console Downloads page.
The mirror registry for Red Hat OpenShift allows users to install a small-scale version of Red Hat Quay
and its required components using the mirror-registry command line interface (CLI) tool. The mirror
registry for Red Hat OpenShift is deployed automatically with preconfigured local storage and a local
database. It also includes auto-generated user credentials and access permissions with a single set of
inputs and no additional configuration choices to get started.
The mirror registry for Red Hat OpenShift provides a pre-determined network configuration and reports
deployed component credentials and access URLs upon success. A limited set of optional configuration
inputs like fully qualified domain name (FQDN) services, superuser name and password, and custom
TLS certificates are also provided. This provides users with a container registry so that they can easily
create an offline mirror of all OpenShift Container Platform release content when running OpenShift
Container Platform in restricted network environments.
Use of the mirror registry for Red Hat OpenShift is optional if another container registry is already
available in the install environment.
4.2.2.1. Mirror registry for Red Hat OpenShift limitations
The following limitations apply to the mirror registry for Red Hat OpenShift :
The mirror registry for Red Hat OpenShift is not a highly-available registry and only local file
system storage is supported. It is not intended to replace Red Hat Quay or the internal image
registry for OpenShift Container Platform.
The mirror registry for Red Hat OpenShift is only supported for hosting images that are required
to install a disconnected OpenShift Container Platform cluster, such as Release images or Red
Hat Operator images. It uses local storage on your Red Hat Enterprise Linux (RHEL) machine,
and storage supported by RHEL is supported by the mirror registry for Red Hat OpenShift .
NOTE
Because the mirror registry for Red Hat OpenShift uses local storage, you should
remain aware of the storage usage consumed when mirroring images and use
Red Hat Quay’s garbage collection feature to mitigate potential issues. For more
information about this feature, see "Red Hat Quay garbage collection".
Support for Red Hat product images that are pushed to the mirror registry for Red Hat
OpenShift for bootstrapping purposes are covered by valid subscriptions for each respective
product. A list of exceptions to further enable the bootstrap experience can be found on the
104
Self-managed Red Hat OpenShift sizing and subscription guide .
Content built by customers should not be hosted by the mirror registry for Red Hat OpenShift .
Using the mirror registry for Red Hat OpenShift with more than one cluster is discouraged
because multiple clusters can create a single point of failure when updating your cluster fleet. It
is advised to leverage the mirror registry for Red Hat OpenShift to install a cluster that can host
a production-grade, highly-available registry such as Red Hat Quay, which can serve OpenShift
Container Platform content to other clusters.
4.2.3. Mirroring on a local host with mirror registry for Red Hat OpenShift
This procedure explains how to install the mirror registry for Red Hat OpenShift on a local host using the
mirror-registry installer tool. By doing so, users can create a local host registry running on port 443 for
the purpose of storing a mirror of OpenShift Container Platform images.
NOTE
Installing the mirror registry for Red Hat OpenShift using the mirror-registry CLI tool
makes several changes to your machine. After installation, a $HOME/quay-install
directory is created, which has installation files, local storage, and the configuration
bundle. Trusted SSH keys are generated in case the deployment target is the local host,
and systemd files on the host machine are set up to ensure that container runtimes are
persistent. Additionally, an initial user named init is created with an automatically
generated password. All access credentials are printed at the end of the install routine.
Procedure
1. Download the mirror-registry.tar.gz package for the latest version of the mirror registry for
Red Hat OpenShift found on the OpenShift console Downloads page.
2. Install the mirror registry for Red Hat OpenShift on your local host with your current user
account by using the mirror-registry tool. For a full list of available flags, see "mirror registry for
Red Hat OpenShift flags".
$ ./mirror-registry install \
--quayHostname <host_example_com> \
--quayRoot <example_directory_name>
3. Use the user name and password generated during installation to log into the registry by
running the following command:
$ podman login -u init \

-p <password> \
<host_example_com>:8443> \
--tls-verify=false 1
1 You can avoid running --tls-verify=false by configuring your system to trust the generated
rootCA certificates. See "Using SSL to protect connections to Red Hat Quay" and
"Configuring the system to trust the certificate authority" for more information.
NOTE
105
NOTE
You can also log in by accessing the UI at https://<host.example.com>:8443

after installation.
4. You can mirror OpenShift Container Platform images after logging in. Depending on your
needs, see either the "Mirroring the OpenShift Container Platform image repository" or the
"Mirroring Operator catalogs for use with disconnected clusters" sections of this document.
NOTE
If there are issues with images stored by the mirror registry for Red Hat OpenShift
due to storage layer problems, you can remirror the OpenShift Container
Platform images, or reinstall mirror registry on more stable storage.
4.2.4. Updating mirror registry for Red Hat OpenShift from a local host
This procedure explains how to update the mirror registry for Red Hat OpenShift from a local host using
the upgrade command. Updating to the latest version ensures new features, bug fixes, and security
vulnerability fixes.
IMPORTANT
When updating, there is intermittent downtime of your mirror registry, as it is restarted

during the update process.
Prerequisites
You have installed the mirror registry for Red Hat OpenShift on a local host.
Procedure
If you are upgrading the mirror registry for Red Hat OpenShift from 1.2.z → 1.3.0, and your
installation directory is the default at /etc/quay-install, you can enter the following command:
$ sudo ./mirror-registry upgrade -v
NOTE
mirror registry for Red Hat OpenShift migrates Podman volumes for Quay
storage, Postgres data, and /etc/quay-install data to the new $HOME/quay-
install location. This allows you to use mirror registry for Red Hat OpenShift
without the --quayRoot flag during future upgrades.
Users who upgrade mirror registry for Red Hat OpenShift with the ./mirror-
registry upgrade -v flag must include the same credentials used when
creating their mirror registry. For example, if you installed the mirror registry
for Red Hat OpenShift with --quayHostname <host_example_com> and --
quayRoot <example_directory_name>, you must include that string to
properly upgrade the mirror registry.
If you are upgrading the mirror registry for Red Hat OpenShift from 1.2.z → 1.3.0 and you used a
106
If you are upgrading the mirror registry for Red Hat OpenShift from 1.2.z → 1.3.0 and you used a
specified directory in your 1.2.z deployment, you must pass in the new --pgStorage and --
quayStorage flags. For example:
$ sudo ./mirror-registry upgrade --quayHostname <host_example_com> --quayRoot

<example_directory_name> --pgStorage <example_directory_name>/pg-data --quayStorage
<example_directory_name>/quay-storage -v
4.2.5. Mirroring on a remote host with mirror registry for Red Hat OpenShift
This procedure explains how to install the mirror registry for Red Hat OpenShift on a remote host using
the mirror-registry tool. By doing so, users can create a registry to hold a mirror of OpenShift Container
Platform images.
NOTE
Installing the mirror registry for Red Hat OpenShift using the mirror-registry CLI tool
makes several changes to your machine. After installation, a $HOME/quay-install
directory is created, which has installation files, local storage, and the configuration
bundle. Trusted SSH keys are generated in case the deployment target is the local host,
and systemd files on the host machine are set up to ensure that container runtimes are
persistent. Additionally, an initial user named init is created with an automatically
generated password. All access credentials are printed at the end of the install routine.
Procedure
1. Download the mirror-registry.tar.gz package for the latest version of the mirror registry for
Red Hat OpenShift found on the OpenShift console Downloads page.
2. Install the mirror registry for Red Hat OpenShift on your local host with your current user
account by using the mirror-registry tool. For a full list of available flags, see "mirror registry for
Red Hat OpenShift flags".
$ ./mirror-registry install -v \
--targetHostname <host_example_com> \
--targetUsername <example_user> \
-k ~/.ssh/my_ssh_key \
3. Use the user name and password generated during installation to log into the mirror registry by
$ podman login -u init \

-p <password> \
<host_example_com>:8443> \
--tls-verify=false 1
1 You can avoid running --tls-verify=false by configuring your system to trust the generated
rootCA certificates. See "Using SSL to protect connections to Red Hat Quay" and
"Configuring the system to trust the certificate authority" for more information.
NOTE
107
NOTE
You can also log in by accessing the UI at https://<host.example.com>:8443

after installation.
4. You can mirror OpenShift Container Platform images after logging in. Depending on your
needs, see either the "Mirroring the OpenShift Container Platform image repository" or the
"Mirroring Operator catalogs for use with disconnected clusters" sections of this document.
NOTE
If there are issues with images stored by the mirror registry for Red Hat OpenShift
due to storage layer problems, you can remirror the OpenShift Container
Platform images, or reinstall mirror registry on more stable storage.
4.2.6. Updating mirror registry for Red Hat OpenShift from a remote host
This procedure explains how to update the mirror registry for Red Hat OpenShift from a remote host
using the upgrade command. Updating to the latest version ensures bug fixes and security vulnerability
fixes.
IMPORTANT
When updating, there is intermittent downtime of your mirror registry, as it is restarted

during the update process.
Prerequisites
You have installed the mirror registry for Red Hat OpenShift on a remote host.
Procedure
To upgrade the mirror registry for Red Hat OpenShift from a remote host, enter the following
command:
$ ./mirror-registry upgrade -v --targetHostname <remote_host_url> --targetUsername

<user_name> -k ~/.ssh/my_ssh_key
NOTE
Users who upgrade the mirror registry for Red Hat OpenShift with the ./mirror-
registry upgrade -v flag must include the same credentials used when creating
their mirror registry. For example, if you installed the mirror registry for Red Hat
OpenShift with --quayHostname <host_example_com> and --quayRoot
<example_directory_name>, you must include that string to properly upgrade
the mirror registry.
4.2.7. Replacing mirror registry for Red Hat OpenShift SSL/TLS certificates
In some cases, you might want to update your SSL/TLS certificates for the mirror registry for Red Hat
OpenShift. This is useful in the following scenarios:
If you are replacing the current mirror registry for Red Hat OpenShift certificate.
108
If you are using the same certificate as the previous mirror registry for Red Hat OpenShift
installation.
If you are periodically updating the mirror registry for Red Hat OpenShift certificate.
Use the following procedure to replace mirror registry for Red Hat OpenShift SSL/TLS certificates.
Prerequisites
You have downloaded the ./mirror-registry binary from the OpenShift console Downloads
page.
Procedure
1. Enter the following command to install the mirror registry for Red Hat OpenShift :
$ ./mirror-registry install \
This installs the mirror registry for Red Hat OpenShift to the $HOME/quay-install directory.
2. Prepare a new certificate authority (CA) bundle and generate new ssl.key and ssl.crt key files.
For more information, see Using SSL/TLS to protect connections to Red Hat Quay .
3. Assign /$HOME/quay-install an environment variable, for example, QUAY, by entering the

following command:
$ export QUAY=/$HOME/quay-install
4. Copy the new ssl.crt file to the /$HOME/quay-install directory by entering the following
command:
$ cp ~/ssl.crt $QUAY/quay-config
5. Copy the new ssl.key file to the /$HOME/quay-install directory by entering the following
command:
$ cp ~/ssl.key $QUAY/quay-config
6. Restart the quay-app application pod by entering the following command:
$ systemctl restart quay-app
4.2.8. Uninstalling the mirror registry for Red Hat OpenShift

You can uninstall the mirror registry for Red Hat OpenShift from your local host by running the
following command:
$ ./mirror-registry uninstall -v \
NOTE
109
NOTE
Deleting the mirror registry for Red Hat OpenShift will prompt the user
before deletion. You can use --autoApprove to skip this prompt.
Users who install the mirror registry for Red Hat OpenShift with the --
quayRoot flag must include the --quayRoot flag when uninstalling. For
example, if you installed the mirror registry for Red Hat OpenShift with --
quayRoot example_directory_name, you must include that string to
properly uninstall the mirror registry.
4.2.9. Mirror registry for Red Hat OpenShift flags

The following flags are available for the mirror registry for Red Hat OpenShift :
Flags Description
--autoApprove A boolean value that disables interactive prompts. If set to true, the quayRoot
directory is automatically deleted when uninstalling the mirror registry. Defaults to
false if left unspecified.
--initPassword The password of the init user created during Quay installation. Must be at least
eight characters and contain no whitespace.
--initUser string Shows the username of the initial user. Defaults to init if left unspecified.
--no-color, -c Allows users to disable color sequences and propagate that to Ansible when
running install, uninstall, and upgrade commands.
--pgStorage The folder where Postgres persistent storage data is saved. Defaults to the pg-
storage Podman volume. Root privileges are required to uninstall.
--quayHostname The fully-qualified domain name of the mirror registry that clients will use to
contact the registry. Equivalent to SERVER_HOSTNAME in the Quay
config.yaml. Must resolve by DNS. Defaults to<targetHostname>:8443 if
left unspecified. [1]
--quayStorage The folder where Quay persistent storage data is saved. Defaults to the quay-
storage Podman volume. Root privileges are required to uninstall.
--quayRoot, -r The directory where container image layer and configuration data is saved,
including rootCA.key , rootCA.pem, and rootCA.srl certificates. Defaults to
$HOME/quay-install if left unspecified.
--ssh-key, -k The path of your SSH identity key. Defaults to ~/.ssh/quay_installer if left
unspecified.
--sslCert The path to the SSL/TLS public key / certificate. Defaults to {quayRoot}/quay-
config and is auto-generated if left unspecified.
110
Flags Description
--sslCheckSkip Skips the check for the certificate hostname against the SERVER_HOSTNAME
in the config.yaml file. [2]
--sslKey The path to the SSL/TLS private key used for HTTPS communication. Defaults to
{quayRoot}/quay-config and is auto-generated if left unspecified.
--targetHostname, -H The hostname of the target you want to install Quay to. Defaults to $HOST , for
example, a local host, if left unspecified.
--targetUsername, -u The user on the target host which will be used for SSH. Defaults to $USER , for
example, the current user if left unspecified.
--verbose, -v Shows debug logs and Ansible playbook outputs.
--version Shows the version for the mirror registry for Red Hat OpenShift.
1. --quayHostname must be modified if the public DNS name of your system is different from the
local hostname. Additionally, the --quayHostname flag does not support installation with an IP
address. Installation with a hostname is required.
2. --sslCheckSkip is used in cases when the mirror registry is set behind a proxy and the exposed
hostname is different from the internal Quay hostname. It can also be used when users do not
want the certificates to be validated against the provided Quay hostname during installation.
4.2.10. Mirror registry for Red Hat OpenShift release notes

The mirror registry for Red Hat OpenShift is a small and streamlined container registry that you can use
as a target for mirroring the required container images of OpenShift Container Platform for
disconnected installations.
These release notes track the development of the mirror registry for Red Hat OpenShift in OpenShift
Container Platform.
For an overview of the mirror registry for Red Hat OpenShift , see Creating a mirror registry with mirror
registry for Red Hat OpenShift.
4.2.10.1. Mirror registry for Red Hat OpenShift 1.3.11
Issued: 2024-04-23
Mirror registry for Red Hat OpenShift is now available with Red Hat Quay 3.8.15.
The following advisory is available for the mirror registry for Red Hat OpenShift :
RHBA-2024:1758 - mirror registry for Red Hat OpenShift 1.3.11
111
Issued: 2023-12-07
Issued: 2023-09-19
Issued: 2023-08-16
Issued: 2023-07-19
Issued: 2023-05-30
Issued: 2023-05-18
112
Issued: 2023-04-25
Issued: 2023-04-05
Issued: 2023-03-21
Issued: 2023-03-7
Issued: 2023-02-20
4.2.10.12.1. New features
Mirror registry for Red Hat OpenShift is now supported on Red Hat Enterprise Linux (RHEL) 9
installations.
113
IPv6 support is now available on mirror registry for Red Hat OpenShift local host installations.
IPv6 is currently unsupported on mirror registry for Red Hat OpenShift remote host installations.
A new feature flag, --quayStorage, has been added. By specifying this flag, you can manually
set the location for the Quay persistent storage.
A new feature flag, --pgStorage, has been added. By specifying this flag, you can manually set
the location for the Postgres persistent storage.
Previously, users were required to have root privileges (sudo) to install mirror registry for Red
Hat OpenShift. With this update, sudo is no longer required to install mirror registry for Red Hat
OpenShift.
When mirror registry for Red Hat OpenShift was installed with sudo, an /etc/quay-install
directory that contained installation files, local storage, and the configuration bundle was
created. With the removal of the sudo requirement, installation files and the configuration
bundle are now installed to $HOME/quay-install. Local storage, for example Postgres and
Quay, are now stored in named volumes automatically created by Podman.
To override the default directories that these files are stored in, you can use the command line
arguments for mirror registry for Red Hat OpenShift . For more information about mirror registry
for Red Hat OpenShift command line arguments, see " Mirror registry for Red Hat OpenShift
flags".
4.2.10.12.2. Bug fixes
Previously, the following error could be returned when attempting to uninstall mirror registry for
Red Hat OpenShift: ["Error: no container with name or ID \"quay-postgres\" found: no such
container"], "stdout": "", "stdout_lines": []*. With this update, the order that mirror registry
for Red Hat OpenShift services are stopped and uninstalled have been changed so that the error
no longer occurs when uninstalling mirror registry for Red Hat OpenShift . For more information,
see PROJQUAY-4629.
114
4.2.10.15.1. Bug fixes
Previously, getFQDN() relied on the fully-qualified domain name (FQDN) library to determine
its FQDN, and the FQDN library tried to read the /etc/hosts folder directly. Consequently, on
some Red Hat Enterprise Linux CoreOS (RHCOS) installations with uncommon DNS
configurations, the FQDN library would fail to install and abort the installation. With this update,
mirror registry for Red Hat OpenShift uses hostname to determine the FQDN. As a result, the
FQDN library does not fail to install. (PROJQUAY-4139)
A new feature flag, --no-color (-c) has been added. This feature flag allows users to disable color
sequences and propagate that to Ansible when running install, uninstall, and upgrade commands.
115
4.2.10.22.1. Bug fixes
Previously, all components and workers running inside of the Quay pod Operator had log levels
set to DEBUG. As a result, large traffic logs were created that consumed unnecessary space.
With this update, log levels are set to WARN by default, which reduces traffic information while
emphasizing problem scenarios. (PROJQUAY-3504)
A new command, mirror-registry upgrade has been added. This command upgrades all
container images without interfering with configurations or data.
NOTE
If quayRoot was previously set to something other than default, it must be

passed into the upgrade command.
4.2.10.23.2. Bug fixes
Previously, the absence of quayHostname or targetHostname did not default to the local
hostname. With this update, quayHostname and targetHostname now default to the local
hostname if they are missing. (PROJQUAY-3079)
Previously, the command ./mirror-registry --version returned an unknown flag error. Now,
running ./mirror-registry --version returns the current version of the mirror registry for Red Hat
OpenShift. (PROJQUAY-3086)
Previously, users could not set a password during installation, for example, when running
./mirror-registry install --initUser <user_name> --initPassword <password> --verbose. With
this update, users can set a password during installation. (PROJQUAY-3149)
Previously, the mirror registry for Red Hat OpenShift did not recreate pods if they were
destroyed. Now, pods are recreated if they are destroyed. (PROJQUAY-3261)
4.2.11. Troubleshooting mirror registry for Red Hat OpenShift
116
To assist in troubleshooting mirror registry for Red Hat OpenShift , you can gather logs of systemd
services installed by the mirror registry. The following services are installed:
quay-app.service
quay-postgres.service
quay-redis.service
quay-pod.service
Prerequisites
You have installed mirror registry for Red Hat OpenShift .
Procedure
If you installed mirror registry for Red Hat OpenShift with root privileges, you can get the status
information of its systemd services by entering the following command:
$ sudo systemctl status <service>
If you installed mirror registry for Red Hat OpenShift as a standard user, you can get the status
information of its systemd services by entering the following command:
$ systemctl --user status <service>
4.2.12. Additional resources

Red Hat Quay garbage collection
Using SSL to protect connections to Red Hat Quay
Configuring the system to trust the certificate authority
Mirroring the OpenShift Container Platform image repository
Mirroring Operator catalogs for use with disconnected clusters
4.3. MIRRORING IMAGES FOR A DISCONNECTED INSTALLATION

You can ensure your clusters only use container images that satisfy your organizational controls on
external content. Before you install a cluster on infrastructure that you provision in a restricted network,
you must mirror the required container images into that environment. To mirror container images, you
must have a registry for mirroring.
IMPORTANT
You must have access to the internet to obtain the necessary container images. In this
procedure, you place your mirror registry on a mirror host that has access to both your
network and the internet. If you do not have access to a mirror host, use the Mirroring
Operator catalogs for use with disconnected clusters procedure to copy images to a
device you can move across network boundaries with.
117
You must have a container image registry that supports Docker v2-2 in the location that will
host the OpenShift Container Platform cluster, such as one of the following registries:
Red Hat Quay
JFrog Artifactory
Sonatype Nexus Repository
Harbor
If you have an entitlement to Red Hat Quay, see the documentation on deploying Red Hat Quay
for proof-of-concept purposes or by using the Red Hat Quay Operator . If you need additional
assistance selecting and installing a registry, contact your sales representative or Red Hat
support.
If you do not already have an existing solution for a container image registry, subscribers of
OpenShift Container Platform are provided a mirror registry for Red Hat OpenShift . The mirror
registry for Red Hat OpenShift is included with your subscription and is a small-scale container
registry that can be used to mirror the required container images of OpenShift Container
Platform in disconnected installations.
4.3.2. About the mirror registry

You can mirror the images that are required for OpenShift Container Platform installation and
subsequent product updates to a container mirror registry such as Red Hat Quay, JFrog Artifactory,
Sonatype Nexus Repository, or Harbor. If you do not have access to a large-scale container registry, you
can use the mirror registry for Red Hat OpenShift , a small-scale container registry included with
OpenShift Container Platform subscriptions.
You can use any container registry that supports Docker v2-2, such as Red Hat Quay, the mirror registry
for Red Hat OpenShift, Artifactory, Sonatype Nexus Repository, or Harbor. Regardless of your chosen
registry, the procedure to mirror content from Red Hat hosted sites on the internet to an isolated image
registry is the same. After you mirror the content, you configure each cluster to retrieve this content
from your mirror registry.
IMPORTANT
The OpenShift image registry cannot be used as the target registry because it does not
support pushing without a tag, which is required during the mirroring process.
If choosing a container registry that is not the mirror registry for Red Hat OpenShift , it must be reachable
by every machine in the clusters that you provision. If the registry is unreachable, installation, updating,
or normal operations such as workload relocation might fail. For that reason, you must run mirror
registries in a highly available way, and the mirror registries must at least match the production
availability of your OpenShift Container Platform clusters.
When you populate your mirror registry with OpenShift Container Platform images, you can follow two
scenarios. If you have a host that can access both the internet and your mirror registry, but not your
cluster nodes, you can directly mirror the content from that machine. This process is referred to as
connected mirroring. If you have no such host, you must mirror the images to a file system and then bring
that host or removable media into your restricted environment. This process is referred to as
disconnected mirroring.
118
For mirrored registries, to view the source of pulled images, you must review the Trying to access log
entry in the CRI-O logs. Other methods to view the image pull source, such as using the crictl images
command on a node, show the non-mirrored image name, even though the image is pulled from the
mirrored location.
NOTE
Red Hat does not test third party registries with OpenShift Container Platform.
Additional information
For information about viewing the CRI-O logs to view the image source, see Viewing the image pull
source.
4.3.3. Preparing your mirror host

Before you perform the mirror procedure, you must prepare the host to retrieve content and push it to
the remote location.
4.3.3.1. Installing the OpenShift CLI by downloading the binary
You can install the OpenShift CLI (oc) to interact with OpenShift Container Platform from a command-
line interface. You can install oc on Linux, Windows, or macOS.
IMPORTANT
If you installed an earlier version of oc, you cannot use it to complete all of the commands
in OpenShift Container Platform 4.15. Download and install the new version of oc.
Installing the OpenShift CLI on Linux

You can install the OpenShift CLI (oc) binary on Linux by using the following procedure.
Procedure
1. Navigate to the OpenShift Container Platform downloads page on the Red Hat Customer
Portal.
2. Select the architecture from the Product Variant drop-down list.
3. Select the appropriate version from the Version drop-down list.
4. Click Download Now next to the OpenShift v4.15 Linux Client entry and save the file.
5. Unpack the archive:
$ tar xvf <file>
6. Place the oc binary in a directory that is on your PATH.

To check your PATH, execute the following command:
$ echo $PATH
Verification
119
After you install the OpenShift CLI, it is available using the oc command:
$ oc <command>
Installing the OpenShift CLI on Windows

You can install the OpenShift CLI (oc) binary on Windows by using the following procedure.
Procedure
Portal.
3. Click Download Now next to the OpenShift v4.15 Windows Client entry and save the file.
4. Unzip the archive with a ZIP program.
5. Move the oc binary to a directory that is on your PATH.

To check your PATH, open the command prompt and execute the following command:
C:\> path
Verification
C:\> oc <command>
Installing the OpenShift CLI on macOS

You can install the OpenShift CLI (oc) binary on macOS by using the following procedure.
Procedure
Portal.
3. Click Download Now next to the OpenShift v4.15 macOS Client entry and save the file.
NOTE
For macOS arm64, choose the OpenShift v4.15 macOS arm64 Client entry.
4. Unpack and unzip the archive.
5. Move the oc binary to a directory on your PATH.

To check your PATH, open a terminal and execute the following command:
$ echo $PATH
Verification
120
Verification
$ oc <command>
4.3.4. Configuring credentials that allow images to be mirrored

Create a container image registry credentials file that allows mirroring images from Red Hat to your
mirror.

WARNING
Do not use this image registry credentials file as the pull secret when you install a
cluster. If you provide this file when you install cluster, all of the machines in the
cluster will have write access to your mirror registry.

WARNING
This process requires that you have write access to a container image registry on
the mirror registry and adds the credentials to a registry pull secret.
Prerequisites
You configured a mirror registry to use in your disconnected environment.
You identified an image repository location on your mirror registry to mirror images into.
You provisioned a mirror registry account that allows images to be uploaded to that image
repository.
Procedure
Complete the following steps on the installation host:
1. Download your registry.redhat.io pull secret from Red Hat OpenShift Cluster Manager .
2. Make a copy of your pull secret in JSON format:
$ cat ./pull-secret | jq . > <path>/<pull_secret_file_in_json> 1
1 Specify the path to the folder to store the pull secret in and a name for the JSON file that
you create.
The contents of the file resemble the following example:
121
"auths": {
"cloud.openshift.com": {
"auth": "b3BlbnNo...",
"email": "you@example.com"
},
"quay.io": {
},
"registry.connect.redhat.com": {
"auth": "NTE3Njg5Nj...",
},
"registry.redhat.io": {
}
}
}
3. Generate the base64-encoded user name and password or token for your mirror registry:
$ echo -n '<user_name>:<password>' | base64 -w0 1

BGVtbYk3ZHAtqXs=
1 For <user_name> and <password>, specify the user name and password that you
configured for your registry.
4. Edit the JSON file and add a section that describes your registry to it:
"auths": {
"<mirror_registry>": { 1
"auth": "<credentials>", 2
}
},
1 For <mirror_registry>, specify the registry domain name, and optionally the port, that
your mirror registry uses to serve content. For example, registry.example.com or
registry.example.com:8443
2 For <credentials>, specify the base64-encoded user name and password for the mirror
registry.
The file resembles the following example:
{
"auths": {
"registry.example.com": {
"auth": "BGVtbYk3ZHAtqXs=",
},
122
},
"quay.io": {
},
},
}
}
}
4.3.5. Mirroring the OpenShift Container Platform image repository

Mirror the OpenShift Container Platform image repository to your registry to use during cluster
installation or upgrade.
Prerequisites
Your mirror host has access to the internet.
You configured a mirror registry to use in your restricted network and can access the certificate
and credentials that you configured.
You downloaded the pull secret from Red Hat OpenShift Cluster Manager and modified it to
include authentication to your mirror repository.
If you use self-signed certificates, you have specified a Subject Alternative Name in the
certificates.
Procedure
Complete the following steps on the mirror host:
1. Review the OpenShift Container Platform downloads page to determine the version of
OpenShift Container Platform that you want to install and determine the corresponding tag on
the Repository Tags page.
2. Set the required environment variables:
a. Export the release version:
$ OCP_RELEASE=<release_version>
For <release_version>, specify the tag that corresponds to the version of OpenShift
Container Platform to install, such as 4.5.4.
b. Export the local registry name and host port:
$ LOCAL_REGISTRY='<local_registry_host_name>:<local_registry_host_port>'
123
For <local_registry_host_name>, specify the registry domain name for your mirror
repository, and for <local_registry_host_port>, specify the port that it serves content on.
c. Export the local repository name:
$ LOCAL_REPOSITORY='<local_repository_name>'
For <local_repository_name>, specify the name of the repository to create in your

registry, such as ocp4/openshift4.
d. Export the name of the repository to mirror:
$ PRODUCT_REPO='openshift-release-dev'
For a production release, you must specify openshift-release-dev.
e. Export the path to your registry pull secret:
$ LOCAL_SECRET_JSON='<path_to_pull_secret>'
For <path_to_pull_secret>, specify the absolute path to and file name of the pull secret
for your mirror registry that you created.
f. Export the release mirror:
$ RELEASE_NAME="ocp-release"
For a production release, you must specify ocp-release.
g. Export the type of architecture for your cluster:
$ ARCHITECTURE=<cluster_architecture> 1
1 Specify the architecture of the cluster, such as x86_64, aarch64, s390x, or ppc64le.
h. Export the path to the directory to host the mirrored images:
$ REMOVABLE_MEDIA_PATH=<path> 1
1 Specify the full path, including the initial forward slash (/) character.
3. Mirror the version images to the mirror registry:
If your mirror host does not have internet access, take the following actions:
i. Connect the removable media to a system that is connected to the internet.
ii. Review the images and configuration manifests to mirror:
$ oc adm release mirror -a ${LOCAL_SECRET_JSON} \

--from=quay.io/${PRODUCT_REPO}/${RELEASE_NAME}:${OCP_RELEASE}-
${ARCHITECTURE} \
--to=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY} \
124
--to-release-
image=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}-
${ARCHITECTURE} --dry-run
iii. Record the entire imageContentSources section from the output of the previous
command. The information about your mirrors is unique to your mirrored repository, and
you must add the imageContentSources section to the install-config.yaml file during
installation.
iv. Mirror the images to a directory on the removable media:
$ oc adm release mirror -a ${LOCAL_SECRET_JSON} --to-

dir=${REMOVABLE_MEDIA_PATH}/mirror
quay.io/${PRODUCT_REPO}/${RELEASE_NAME}:${OCP_RELEASE}-
${ARCHITECTURE}
v. Take the media to the restricted network environment and upload the images to the
local container registry.
$ oc image mirror -a ${LOCAL_SECRET_JSON} --from-

dir=${REMOVABLE_MEDIA_PATH}/mirror
"file://openshift/release:${OCP_RELEASE}*"
${LOCAL_REGISTRY}/${LOCAL_REPOSITORY} 1
1 For REMOVABLE_MEDIA_PATH, you must use the same path that you specified
when you mirrored the images.
IMPORTANT
Running oc image mirror might result in the following error: error:

unable to retrieve source image. This error occurs when image indexes
include references to images that no longer exist on the image registry.
Image indexes might retain older references to allow users running those
images an upgrade path to newer points on the upgrade graph. As a
temporary workaround, you can use the --skip-missing option to bypass
the error and continue downloading the image index. For more
information, see Service Mesh Operator mirroring failed .
If the local container registry is connected to the mirror host, take the following actions:
i. Directly push the release images to the local registry by using following command:
$ oc adm release mirror -a ${LOCAL_SECRET_JSON} \

--from=quay.io/${PRODUCT_REPO}/${RELEASE_NAME}:${OCP_RELEASE}-
${ARCHITECTURE} \
--to=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY} \
--to-release-
image=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}-
${ARCHITECTURE}
This command pulls the release information as a digest, and its output includes the
imageContentSources data that you require when you install your cluster.
ii. Record the entire imageContentSources section from the output of the previous
125
ii. Record the entire imageContentSources section from the output of the previous
command. The information about your mirrors is unique to your mirrored repository, and
you must add the imageContentSources section to the install-config.yaml file during
installation.
NOTE
The image name gets patched to Quay.io during the mirroring process,
and the podman images will show Quay.io in the registry on the
bootstrap virtual machine.
4. To create the installation program that is based on the content that you mirrored, extract it and
pin it to the release:
If your mirror host does not have internet access, run the following command:
$ oc adm release extract -a ${LOCAL_SECRET_JSON} --icsp-file=<file> \ --

command=openshift-install
"${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}"
If the local container registry is connected to the mirror host, run the following command:
$ oc adm release extract -a ${LOCAL_SECRET_JSON} --command=openshift-install

"${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}-
${ARCHITECTURE}"
IMPORTANT
To ensure that you use the correct images for the version of OpenShift
Container Platform that you selected, you must extract the installation
program from the mirrored content.
You must perform this step on a machine with an active internet connection.
5. For clusters using installer-provisioned infrastructure, run the following command:
$ openshift-install
4.3.6. The Cluster Samples Operator in a disconnected environment

In a disconnected environment, you must take additional steps after you install a cluster to configure the
Cluster Samples Operator. Review the following information in preparation.
4.3.6.1. Cluster Samples Operator assistance for mirroring
During installation, OpenShift Container Platform creates a config map named imagestreamtag-to-
image in the openshift-cluster-samples-operator namespace. The imagestreamtag-to-image config
map contains an entry, the populating image, for each image stream tag.
The format of the key for each entry in the data field in the config map is
<image_stream_name>_<image_stream_tag_name>.
During a disconnected installation of OpenShift Container Platform, the status of the Cluster Samples
126
During a disconnected installation of OpenShift Container Platform, the status of the Cluster Samples
Operator is set to Removed. If you choose to change it to Managed, it installs samples.
NOTE
The use of samples in a network-restricted or discontinued environment may require

access to services external to your network. Some example services include: Github,
Maven Central, npm, RubyGems, PyPi and others. There might be additional steps to
take that allow the cluster samples operators’s objects to reach the services they require.
You can use this config map as a reference for which images need to be mirrored for your image
streams to import.
While the Cluster Samples Operator is set to Removed, you can create your mirrored registry, or
determine which existing mirrored registry you want to use.
Mirror the samples you want to the mirrored registry using the new config map as your guide.
Add any of the image streams you did not mirror to the skippedImagestreams list of the
Cluster Samples Operator configuration object.
Set samplesRegistry of the Cluster Samples Operator configuration object to the mirrored
registry.
Then set the Cluster Samples Operator to Managed to install the image streams you have
mirrored.
4.3.7. Mirroring Operator catalogs for use with disconnected clusters

You can mirror the Operator contents of a Red Hat-provided catalog, or a custom catalog, into a
container image registry using the oc adm catalog mirror command. The target registry must support
Docker v2-2. For a cluster on a restricted network, this registry can be one that the cluster has network
access to, such as a mirror registry created during a restricted network cluster installation.
IMPORTANT
The OpenShift image registry cannot be used as the target registry because it
does not support pushing without a tag, which is required during the mirroring
process.
Running oc adm catalog mirror might result in the following error: error: unable
to retrieve source image. This error occurs when image indexes include
references to images that no longer exist on the image registry. Image indexes
might retain older references to allow users running those images an upgrade
path to newer points on the upgrade graph. As a temporary workaround, you can
use the --skip-missing option to bypass the error and continue downloading the
image index. For more information, see Service Mesh Operator mirroring failed .
The oc adm catalog mirror command also automatically mirrors the index image that is specified
during the mirroring process, whether it be a Red Hat-provided index image or your own custom-built
index image, to the target registry. You can then use the mirrored index image to create a catalog
source that allows Operator Lifecycle Manager (OLM) to load the mirrored catalog onto your OpenShift
Container Platform cluster.
127
Using Operator Lifecycle Manager on restricted networks
4.3.7.1. Prerequisites
Mirroring Operator catalogs for use with disconnected clusters has the following prerequisites:
Workstation with unrestricted network access.
podman version 1.9.3 or later.
If you want to filter, or prune, an existing catalog and selectively mirror only a subset of
Operators, see the following sections:
Installing the opm CLI
Updating or filtering a file-based catalog image
If you want to mirror a Red Hat-provided catalog, run the following command on your
workstation with unrestricted network access to authenticate with registry.redhat.io:
$ podman login registry.redhat.io
Access to a mirror registry that supports Docker v2-2.
On your mirror registry, decide which repository, or namespace, to use for storing mirrored
Operator content. For example, you might create an olm-mirror repository.
If your mirror registry does not have internet access, connect removable media to your
workstation with unrestricted network access.
If you are working with private registries, including registry.redhat.io, set the REG_CREDS
environment variable to the file path of your registry credentials for use in later steps. For
example, for the podman CLI:
$ REG_CREDS=${XDG_RUNTIME_DIR}/containers/auth.json
4.3.7.2. Extracting and mirroring catalog contents
The oc adm catalog mirror command extracts the contents of an index image to generate the
manifests required for mirroring. The default behavior of the command generates manifests, then
automatically mirrors all of the image content from the index image, as well as the index image itself, to
your mirror registry.
Alternatively, if your mirror registry is on a completely disconnected, or airgapped, host, you can first
mirror the content to removable media, move the media to the disconnected environment, then mirror
the content from the media to the registry.
4.3.7.2.1. Mirroring catalog contents to registries on the same network
If your mirror registry is co-located on the same network as your workstation with unrestricted network
access, take the following actions on your workstation.
Procedure
1. If your mirror registry requires authentication, run the following command to log in to the
128
1. If your mirror registry requires authentication, run the following command to log in to the
registry:
$ podman login <mirror_registry>
2. Run the following command to extract and mirror the content to the mirror registry:
$ oc adm catalog mirror \

<index_image> \ 1
<mirror_registry>:<port>[/<repository>] \ 2
[-a ${REG_CREDS}] \ 3
[--insecure] \ 4
[--index-filter-by-os='<platform>/<arch>'] \ 5
[--manifests-only] 6
1 Specify the index image for the catalog that you want to mirror.
2 Specify the fully qualified domain name (FQDN) for the target registry to mirror the
Operator contents to. The mirror registry <repository> can be any existing repository, or
namespace, on the registry, for example olm-mirror as outlined in the prerequisites. If
there is an existing repository found during mirroring, the repository name is added to the
resulting image name. If you do not want the image name to include the repository name,
omit the <repository> value from this line, for example <mirror_registry>:<port>.
3 Optional: If required, specify the location of your registry credentials file. {REG_CREDS} is
required for registry.redhat.io.
4 Optional: If you do not want to configure trust for the target registry, add the --insecure
flag.
5 Optional: Specify which platform and architecture of the index image to select when
multiple variants are available. Images are passed as '<platform>/<arch>[/<variant>]'. This
does not apply to images referenced by the index. Valid values are linux/amd64,
linux/ppc64le, linux/s390x, linux/arm64.
6 Optional: Generate only the manifests required for mirroring without actually mirroring the
image content to a registry. This option can be useful for reviewing what will be mirrored,
and lets you make any changes to the mapping list, if you require only a subset of packages.
You can then use the mapping.txt file with the oc image mirror command to mirror the
modified list of images in a later step. This flag is intended for only advanced selective
mirroring of content from the catalog.
Example output
src image has index label for database path: /database/index.db

using database path mapping: /database/index.db:/tmp/153048078
wrote database to /tmp/153048078 1
...
wrote mirroring manifests to manifests-redhat-operator-index-1614211642 2
1 Directory for the temporary index.db database generated by the command.
2
Record the manifests directory name that is generated. This directory is referenced in
129
Record the manifests directory name that is generated. This directory is referenced in
subsequent procedures.
NOTE
Red Hat Quay does not support nested repositories. As a result, running the oc
adm catalog mirror command will fail with a 401 unauthorized error. As a
workaround, you can use the --max-components=2 option when running the oc
adm catalog mirror command to disable the creation of nested repositories. For
more information on this workaround, see the Unauthorized error thrown while
using catalog mirror command with Quay registry Knowledgebase Solution.
Architecture and operating system support for Operators
4.3.7.2.2. Mirroring catalog contents to airgapped registries
If your mirror registry is on a completely disconnected, or airgapped, host, take the following actions.
Procedure
1. Run the following command on your workstation with unrestricted network access to mirror the
content to local files:

<index_image> \ 1
file:///local/index \ 2
-a ${REG_CREDS} \ 3
--insecure \ 4
--index-filter-by-os='<platform>/<arch>' 5
1 Specify the index image for the catalog that you want to mirror.
2 Specify the content to mirror to local files in your current directory.
3 Optional: If required, specify the location of your registry credentials file.
flag.
multiple variants are available. Images are specified as '<platform>/<arch>[/<variant>]'.
This does not apply to images referenced by the index. Valid values are linux/amd64,
linux/ppc64le, linux/s390x, linux/arm64, and .*
Example output
...
info: Mirroring completed in 5.93s (5.915MB/s)
wrote mirroring manifests to manifests-my-index-1614985528 1
130
To upload local images to a registry, run:
oc adm catalog mirror file://local/index/myrepo/my-index:v1 REGISTRY/REPOSITORY 2
1 Record the manifests directory name that is generated. This directory is referenced in
subsequent procedures.
2 Record the expanded file:// path that is based on your provided index image. This path is
referenced in a subsequent step.
This command creates a v2/ directory in your current directory.
2. Copy the v2/ directory to removable media.
3. Physically remove the media and attach it to a host in the disconnected environment that has
access to the mirror registry.
4. If your mirror registry requires authentication, run the following command on your host in the
disconnected environment to log in to the registry:
$ podman login <mirror_registry>
5. Run the following command from the parent directory containing the v2/ directory to upload the
images from local files to the mirror registry:

file://local/index/<repository>/<index_image>:<tag> \ 1
<mirror_registry>:<port>[/<repository>] \ 2
-a ${REG_CREDS} \ 3
--insecure \ 4
--index-filter-by-os='<platform>/<arch>' 5
1 Specify the file:// path from the previous command output.
2 Specify the fully qualified domain name (FQDN) for the target registry to mirror the
Operator contents to. The mirror registry <repository> can be any existing repository, or
namespace, on the registry, for example olm-mirror as outlined in the prerequisites. If
there is an existing repository found during mirroring, the repository name is added to the
resulting image name. If you do not want the image name to include the repository name,
omit the <repository> value from this line, for example <mirror_registry>:<port>.
3 Optional: If required, specify the location of your registry credentials file.
flag.
multiple variants are available. Images are specified as '<platform>/<arch>[/<variant>]'.
This does not apply to images referenced by the index. Valid values are linux/amd64,
linux/ppc64le, linux/s390x, linux/arm64, and .*
NOTE
131
NOTE
Red Hat Quay does not support nested repositories. As a result, running the oc
adm catalog mirror command will fail with a 401 unauthorized error. As a
workaround, you can use the --max-components=2 option when running the oc
adm catalog mirror command to disable the creation of nested repositories. For
more information on this workaround, see the Unauthorized error thrown while
using catalog mirror command with Quay registry Knowledgebase Solution.
6. Run the oc adm catalog mirror command again. Use the newly mirrored index image as the
source and the same mirror registry target used in the previous step:

<mirror_registry>:<port>/<index_image> \
<mirror_registry>:<port>[/<repository>] \
--manifests-only \ 1
[-a ${REG_CREDS}] \
[--insecure]
1 The --manifests-only flag is required for this step so that the command does not copy all
of the mirrored content again.
IMPORTANT
This step is required because the image mappings in the

imageContentSourcePolicy.yaml file generated during the previous step must
be updated from local paths to valid mirror locations. Failure to do so will cause
errors when you create the ImageContentSourcePolicy object in a later step.
After you mirror the catalog, you can continue with the remainder of your cluster installation. After your
cluster installation has finished successfully, you must specify the manifests directory from this
procedure to create the ImageContentSourcePolicy and CatalogSource objects. These objects are
required to enable installation of Operators from OperatorHub.
Architecture and operating system support for Operators
4.3.7.3. Generated manifests
After mirroring Operator catalog content to your mirror registry, a manifests directory is generated in
your current directory.
If you mirrored content to a registry on the same network, the directory name takes the following
pattern:
manifests-<index_image_name>-<random_number>
If you mirrored content to a registry on a disconnected host in the previous section, the directory name
takes the following pattern:
manifests-index/<repository>/<index_image_name>-<random_number>
132
NOTE
The manifests directory name is referenced in subsequent procedures.
The manifests directory contains the following files, some of which might require further modification:
The catalogSource.yaml file is a basic definition for a CatalogSource object that is pre-
populated with your index image tag and other relevant metadata. This file can be used as is or
modified to add the catalog source to your cluster.
IMPORTANT
If you mirrored the content to local files, you must modify your
catalogSource.yaml file to remove any backslash ( /) characters from the
metadata.name field. Otherwise, when you attempt to create the object, it fails
with an "invalid resource name" error.
The imageContentSourcePolicy.yaml file defines an ImageContentSourcePolicy object that

can configure nodes to translate between the image references stored in Operator manifests
and the mirrored registry.
NOTE
If your cluster uses an ImageContentSourcePolicy object to configure

repository mirroring, you can use only global pull secrets for mirrored registries.
You cannot add a pull secret to a project.
The mapping.txt file contains all of the source images and where to map them in the target
registry. This file is compatible with the oc image mirror command and can be used to further
customize the mirroring configuration.
IMPORTANT
If you used the --manifests-only flag during the mirroring process and want to
further trim the subset of packages to mirror, see the steps in the Mirroring a
package manifest format catalog image procedure of the OpenShift Container
Platform 4.7 documentation about modifying your mapping.txt file and using the
file with the oc image mirror command.
4.3.7.4. Postinstallation requirements
After you mirror the catalog, you can continue with the remainder of your cluster installation. After your
cluster installation has finished successfully, you must specify the manifests directory from this
procedure to create the ImageContentSourcePolicy and CatalogSource objects. These objects are
required to populate and enable installation of Operators from OperatorHub.
Populating OperatorHub from mirrored Operator catalogs
Updating or filtering a file-based catalog image
4.3.8. Next steps
133
Install a cluster on infrastructure that you provision in your restricted network, such as on
VMware vSphere, bare metal, or Amazon Web Services.

See Gathering data about specific features for more information about using must-gather.
4.4. MIRRORING IMAGES FOR A DISCONNECTED INSTALLATION

USING THE OC-MIRROR PLUGIN
Running your cluster in a restricted network without direct internet connectivity is possible by installing
the cluster from a mirrored set of OpenShift Container Platform container images in a private registry.
This registry must be running at all times as long as the cluster is running. See the Prerequisites section
for more information.
You can use the oc-mirror OpenShift CLI (oc) plugin to mirror images to a mirror registry in your fully or
partially disconnected environments. You must run oc-mirror from a system with internet connectivity in
order to download the required images from the official Red Hat registries.
The following steps outline the high-level workflow on how to use the oc-mirror plugin to mirror images
to a mirror registry:
1. Create an image set configuration file.
2. Mirror the image set to the mirror registry by using one of the following methods:
Mirror an image set directly to the mirror registry.
Mirror an image set to disk, transfer the image set to the target environment, then upload
the image set to the target mirror registry.
3. Configure your cluster to use the resources generated by the oc-mirror plugin.
4. Repeat these steps to update your mirror registry as necessary.
4.4.1. About the oc-mirror plugin

You can use the oc-mirror OpenShift CLI (oc) plugin to mirror all required OpenShift Container
Platform content and other images to your mirror registry by using a single tool. It provides the following
features:
Provides a centralized method to mirror OpenShift Container Platform releases, Operators,

helm charts, and other images.
Maintains update paths for OpenShift Container Platform and Operators.
Uses a declarative image set configuration file to include only the OpenShift Container
Platform releases, Operators, and images that your cluster needs.
Performs incremental mirroring, which reduces the size of future image sets.
Prunes images from the target mirror registry that were excluded from the image set
configuration since the previous execution.
Optionally generates supporting artifacts for OpenShift Update Service (OSUS) usage.
When using the oc-mirror plugin, you specify which content to mirror in an image set configuration file. In
134
When using the oc-mirror plugin, you specify which content to mirror in an image set configuration file. In
this YAML file, you can fine-tune the configuration to only include the OpenShift Container Platform
releases and Operators that your cluster needs. This reduces the amount of data that you need to
download and transfer. The oc-mirror plugin can also mirror arbitrary helm charts and additional
container images to assist users in seamlessly synchronizing their workloads onto mirror registries.
The first time you run the oc-mirror plugin, it populates your mirror registry with the required content to
perform your disconnected cluster installation or update. In order for your disconnected cluster to
continue receiving updates, you must keep your mirror registry updated. To update your mirror registry,
you run the oc-mirror plugin using the same configuration as the first time you ran it. The oc-mirror
plugin references the metadata from the storage backend and only downloads what has been released
since the last time you ran the tool. This provides update paths for OpenShift Container Platform and
Operators and performs dependency resolution as required.
IMPORTANT
When using the oc-mirror CLI plugin to populate a mirror registry, any further updates to
the mirror registry must be made using the oc-mirror tool.
4.4.2. oc-mirror compatibility and support

The oc-mirror plugin supports mirroring OpenShift Container Platform payload images and Operator
catalogs for OpenShift Container Platform versions 4.10 and later.
NOTE
On aarch64, ppc64le, and s390x architectures the oc-mirror plugin is only supported for
OpenShift Container Platform versions 4.14 and later.
Use the latest available version of the oc-mirror plugin regardless of which versions of OpenShift
Container Platform you need to mirror.
IMPORTANT
If you used the Technology Preview OCI local catalogs feature for the oc-mirror plugin
for OpenShift Container Platform 4.12, you can no longer use the OCI local catalogs
feature of the oc-mirror plugin to copy a catalog locally and convert it to OCI format as a
first step to mirroring to a fully disconnected cluster.
4.4.3. About the mirror registry

You can mirror the images that are required for OpenShift Container Platform installation and
subsequent product updates to a container mirror registry that supports Docker v2-2, such as Red Hat
Quay. If you do not have access to a large-scale container registry, you can use the mirror registry for
Red Hat OpenShift, which is a small-scale container registry included with OpenShift Container Platform
subscriptions.
Regardless of your chosen registry, the procedure to mirror content from Red Hat hosted sites on the
internet to an isolated image registry is the same. After you mirror the content, you configure each
cluster to retrieve this content from your mirror registry.
IMPORTANT
135
IMPORTANT
The OpenShift image registry cannot be used as the target registry because it does not
support pushing without a tag, which is required during the mirroring process.
If choosing a container registry that is not the mirror registry for Red Hat OpenShift , it must be reachable
by every machine in the clusters that you provision. If the registry is unreachable, installation, updating,
or normal operations such as workload relocation might fail. For that reason, you must run mirror
registries in a highly available way, and the mirror registries must at least match the production
availability of your OpenShift Container Platform clusters.
When you populate your mirror registry with OpenShift Container Platform images, you can follow two
scenarios. If you have a host that can access both the internet and your mirror registry, but not your
cluster nodes, you can directly mirror the content from that machine. This process is referred to as
connected mirroring. If you have no such host, you must mirror the images to a file system and then bring
that host or removable media into your restricted environment. This process is referred to as
disconnected mirroring.
For mirrored registries, to view the source of pulled images, you must review the Trying to access log
entry in the CRI-O logs. Other methods to view the image pull source, such as using the crictl images
command on a node, show the non-mirrored image name, even though the image is pulled from the
mirrored location.
NOTE
Red Hat does not test third party registries with OpenShift Container Platform.
For information about viewing the CRI-O logs to view the image source, see Viewing the image
pull source.
You must have a container image registry that supports Docker v2-2 in the location that will
host the OpenShift Container Platform cluster, such as Red Hat Quay.
NOTE
If you use Red Hat Quay, you must use version 3.6 or later with the oc-mirror
plugin. If you have an entitlement to Red Hat Quay, see the documentation on
deploying Red Hat Quay for proof-of-concept purposes or by using the Red Hat
Quay Operator. If you need additional assistance selecting and installing a
registry, contact your sales representative or Red Hat Support.
If you do not already have an existing solution for a container image registry, subscribers of
OpenShift Container Platform are provided a mirror registry for Red Hat OpenShift . The mirror
registry for Red Hat OpenShift is included with your subscription and is a small-scale container
registry that can be used to mirror the required container images of OpenShift Container
Platform in disconnected installations.
4.4.5. Preparing your mirror hosts
Before you can use the oc-mirror plugin to mirror images, you must install the plugin and create a
136
Before you can use the oc-mirror plugin to mirror images, you must install the plugin and create a
container image registry credentials file to allow the mirroring from Red Hat to your mirror.
4.4.5.1. Installing the oc-mirror OpenShift CLI plugin
To use the oc-mirror OpenShift CLI plugin to mirror registry images, you must install the plugin. If you
are mirroring image sets in a fully disconnected environment, ensure that you install the oc-mirror plugin
on the host with internet access and the host in the disconnected environment with access to the mirror
registry.
Prerequisites
You have installed the OpenShift CLI (oc).
Procedure
1. Download the oc-mirror CLI plugin.
a. Navigate to the Downloads page of the OpenShift Cluster Manager.
b. Under the OpenShift disconnected installation tools section, click Download for
OpenShift Client (oc) mirror plugin and save the file.
2. Extract the archive:
$ tar xvzf oc-mirror.tar.gz
3. If necessary, update the plugin file to be executable:
$ chmod +x oc-mirror
NOTE
Do not rename the oc-mirror file.
4. Install the oc-mirror CLI plugin by placing the file in your PATH, for example, /usr/local/bin:
$ sudo mv oc-mirror /usr/local/bin/.
Verification
Run oc mirror help to verify that the plugin was successfully installed:
$ oc mirror help
Installing and using CLI plugins
4.4.5.2. Configuring credentials that allow images to be mirrored
137
mirror.

WARNING
Do not use this image registry credentials file as the pull secret when you install a
cluster. If you provide this file when you install cluster, all of the machines in the
cluster will have write access to your mirror registry.

WARNING
This process requires that you have write access to a container image registry on
the mirror registry and adds the credentials to a registry pull secret.
Prerequisites
You configured a mirror registry to use in your disconnected environment.
You identified an image repository location on your mirror registry to mirror images into.
You provisioned a mirror registry account that allows images to be uploaded to that image
repository.
Procedure
Complete the following steps on the installation host:
1. Download your registry.redhat.io pull secret from Red Hat OpenShift Cluster Manager .
2. Make a copy of your pull secret in JSON format:
$ cat ./pull-secret | jq . > <path>/<pull_secret_file_in_json> 1
1 Specify the path to the folder to store the pull secret in and a name for the JSON file that
you create.
The contents of the file resemble the following example:
{
"auths": {
},
"quay.io": {
138
},
},
}
}
}
3. Save the file either as ~/.docker/config.json or $XDG_RUNTIME_DIR/containers/auth.json.
4. Generate the base64-encoded user name and password or token for your mirror registry:
$ echo -n '<user_name>:<password>' | base64 -w0 1

BGVtbYk3ZHAtqXs=
1 For <user_name> and <password>, specify the user name and password that you
configured for your registry.
5. Edit the JSON file and add a section that describes your registry to it:
"auths": {
"<mirror_registry>": { 1
"auth": "<credentials>", 2
}
},
1 For <mirror_registry>, specify the registry domain name, and optionally the port, that
your mirror registry uses to serve content. For example, registry.example.com or
registry.example.com:8443
2 For <credentials>, specify the base64-encoded user name and password for the mirror
registry.
The file resembles the following example:
{
"auths": {
"registry.example.com": {
"auth": "BGVtbYk3ZHAtqXs=",
},
},
"quay.io": {
139
},
},
}
}
}
4.4.6. Creating the image set configuration

Before you can use the oc-mirror plugin to mirror image sets, you must create an image set
configuration file. This image set configuration file defines which OpenShift Container Platform
releases, Operators, and other images to mirror, along with other configuration settings for the oc-
mirror plugin.
You must specify a storage backend in the image set configuration file. This storage backend can be a
local directory or a registry that supports Docker v2-2. The oc-mirror plugin stores metadata in this
storage backend during image set creation.
IMPORTANT
Do not delete or modify the metadata that is generated by the oc-mirror plugin. You
must use the same storage backend every time you run the oc-mirror plugin for the same
mirror registry.
Prerequisites
You have created a container image registry credentials file. For instructions, see Configuring
credentials that allow images to be mirrored.
Procedure
1. Use the oc mirror init command to create a template for the image set configuration and save
it to a file called imageset-config.yaml:
$ oc mirror init --registry example.com/mirror/oc-mirror-metadata > imageset-config.yaml 1
1 Replace example.com/mirror/oc-mirror-metadata with the location of your registry for

the storage backend.
2. Edit the file and adjust the settings as necessary:
kind: ImageSetConfiguration
apiVersion: mirror.openshift.io/v1alpha2
archiveSize: 4 1
storageConfig: 2
registry:
imageURL: example.com/mirror/oc-mirror-metadata 3
140
skipTLS: false
mirror:
platform:
channels:
- name: stable-4.15 4
type: ocp
graph: true 5
operators:
- catalog: registry.redhat.io/redhat/redhat-operator-index:v4.15 6
packages:
- name: serverless-operator 7
channels:
- name: stable 8
additionalImages:
- name: registry.redhat.io/ubi9/ubi:latest 9
helm: {}
1 Add archiveSize to set the maximum size, in GiB, of each file within the image set.
2 Set the back-end location to save the image set metadata to. This location can be a
registry or local directory. It is required to specify storageConfig values.
3 Set the registry URL for the storage backend.
4 Set the channel to retrieve the OpenShift Container Platform images from.
5 Add graph: true to build and push the graph-data image to the mirror registry. The graph-
data image is required to create OpenShift Update Service (OSUS). The graph: true field
also generates the UpdateService custom resource manifest. The oc command-line
interface (CLI) can use the UpdateService custom resource manifest to create OSUS. For
more information, see About the OpenShift Update Service .
6 Set the Operator catalog to retrieve the OpenShift Container Platform images from.
7 Specify only certain Operator packages to include in the image set. Remove this field to
retrieve all packages in the catalog.
8 Specify only certain channels of the Operator packages to include in the image set. You
must always include the default channel for the Operator package even if you do not use
the bundles in that channel. You can find the default channel by running the following
command: oc mirror list operators --catalog=<catalog_name> --package=
<package_name>.
9 Specify any additional images to include in image set.
NOTE
The graph: true field also mirrors the ubi-micro image along with other mirrored
images.
See Image set configuration parameters for the full list of parameters and Image set
configuration examples for various mirroring use cases.
3. Save the updated file.
141
This image set configuration file is required by the oc mirror command when mirroring content.
Image set configuration parameters
Image set configuration examples
Using the OpenShift Update Service in a disconnected environment
4.4.7. Mirroring an image set to a mirror registry

You can use the oc-mirror CLI plugin to mirror images to a mirror registry in a partially disconnected
environment or in a fully disconnected environment.
These procedures assume that you already have your mirror registry set up.
4.4.7.1. Mirroring an image set in a partially disconnected environment
In a partially disconnected environment, you can mirror an image set directly to the target mirror registry.
4.4.7.1.1. Mirroring from mirror to mirror
You can use the oc-mirror plugin to mirror an image set directly to a target mirror registry that is
accessible during image set creation.
You are required to specify a storage backend in the image set configuration file. This storage backend
can be a local directory or a Docker v2 registry. The oc-mirror plugin stores metadata in this storage
backend during image set creation.
IMPORTANT
mirror registry.
Prerequisites
You have access to the internet to get the necessary container images.
You have installed the oc-mirror CLI plugin.
You have created the image set configuration file.
Procedure
Run the oc mirror command to mirror the images from the specified image set configuration to
a specified registry:
$ oc mirror --config=./<imageset-config.yaml> \ 1
docker://registry.example:5000 2
142
1 Specify the image set configuration file that you created. For example, imageset-
config.yaml.
2 Specify the registry to mirror the image set file to. The registry must start with docker://. If
you specify a top-level namespace for the mirror registry, you must also use this same
namespace on subsequent executions.
Verification
1. Navigate into the oc-mirror-workspace/ directory that was generated.
2. Navigate into the results directory, for example, results-1639608409/.
3. Verify that YAML files are present for the ImageContentSourcePolicy and CatalogSource
resources.
NOTE
The repositoryDigestMirrors section of the ImageContentSourcePolicy YAML file is

used for the install-config.yaml file during installation.
Next steps
Configure your cluster to use the resources generated by oc-mirror.
Troubleshooting
Unable to retrieve source image .
4.4.7.2. Mirroring an image set in a fully disconnected environment
To mirror an image set in a fully disconnected environment, you must first mirror the image set to disk ,
then mirror the image set file on disk to a mirror .
4.4.7.2.1. Mirroring from mirror to disk
You can use the oc-mirror plugin to generate an image set and save the contents to disk. The
generated image set can then be transferred to the disconnected environment and mirrored to the
target registry.
IMPORTANT
Depending on the configuration specified in the image set configuration file, using oc-
mirror to mirror images might download several hundreds of gigabytes of data to disk.
The initial image set download when you populate the mirror registry is often the largest.
Because you only download the images that changed since the last time you ran the
command, when you run the oc-mirror plugin again, the generated image set is often
smaller.
You are required to specify a storage backend in the image set configuration file. This storage backend
can be a local directory or a docker v2 registry. The oc-mirror plugin stores metadata in this storage
backend during image set creation.
IMPORTANT
143
IMPORTANT
mirror registry.
Prerequisites
You have access to the internet to obtain the necessary container images.
Procedure
Run the oc mirror command to mirror the images from the specified image set configuration to
disk:
$ oc mirror --config=./imageset-config.yaml \ 1
file://<path_to_output_directory> 2
1 Pass in the image set configuration file that was created. This procedure assumes that it is
named imageset-config.yaml.
2 Specify the target directory where you want to output the image set file. The target
directory path must start with file://.
Verification
1. Navigate to your output directory:
$ cd <path_to_output_directory>
2. Verify that an image set .tar file was created:
$ ls
Example output
mirror_seq1_000000.tar
Next steps
Transfer the image set .tar file to the disconnected environment.
Troubleshooting
144
4.4.7.2.2. Mirroring from disk to mirror
You can use the oc-mirror plugin to mirror the contents of a generated image set to the target mirror
registry.
Prerequisites
You have installed the OpenShift CLI (oc) in the disconnected environment.
You have installed the oc-mirror CLI plugin in the disconnected environment.
You have generated the image set file by using the oc mirror command.
You have transferred the image set file to the disconnected environment.
Procedure
Run the oc mirror command to process the image set file on disk and mirror the contents to a
target mirror registry:
$ oc mirror --from=./mirror_seq1_000000.tar \ 1
1 Pass in the image set .tar file to mirror, named mirror_seq1_000000.tar in this example. If
an archiveSize value was specified in the image set configuration file, the image set might
be broken up into multiple .tar files. In this situation, you can pass in a directory that
contains the image set .tar files.
2 Specify the registry to mirror the image set file to. The registry must start with docker://. If
you specify a top-level namespace for the mirror registry, you must also use this same
This command updates the mirror registry with the image set and generates the
ImageContentSourcePolicy and CatalogSource resources.
Verification
1. Navigate into the oc-mirror-workspace/ directory that was generated.
2. Navigate into the results directory, for example, results-1639608409/.
3. Verify that YAML files are present for the ImageContentSourcePolicy and CatalogSource
resources.
Next steps
Troubleshooting
4.4.8. Configuring your cluster to use the resources generated by oc-mirror
After you have mirrored your image set to the mirror registry, you must apply the generated
145
After you have mirrored your image set to the mirror registry, you must apply the generated
ImageContentSourcePolicy, CatalogSource, and release image signature resources into the cluster.
The ImageContentSourcePolicy resource associates the mirror registry with the source registry and
redirects image pull requests from the online registries to the mirror registry. The CatalogSource
resource is used by Operator Lifecycle Manager (OLM) to retrieve information about the available
Operators in the mirror registry. The release image signatures are used to verify the mirrored release
images.
Prerequisites
You have mirrored the image set to the registry mirror in the disconnected environment.
You have access to the cluster as a user with the cluster-admin role.
Procedure
1. Log in to the OpenShift CLI as a user with the cluster-admin role.
2. Apply the YAML files from the results directory to the cluster by running the following
command:
$ oc apply -f ./oc-mirror-workspace/results-1639608409/
3. If you mirrored release images, apply the release image signatures to the cluster by running the
following command:
$ oc apply -f ./oc-mirror-workspace/results-1639608409/release-signatures/
NOTE
If you are mirroring Operators instead of clusters, you do not need to run $ oc
apply -f ./oc-mirror-workspace/results-1639608409/release-signatures/.
Running that command will return an error, as there are no release image
signatures to apply.
Verification
1. Verify that the ImageContentSourcePolicy resources were successfully installed by running

the following command:
$ oc get imagecontentsourcepolicy
2. Verify that the CatalogSource resources were successfully installed by running the following
command:
$ oc get catalogsource -n openshift-marketplace
4.4.9. Keeping your mirror registry content updated

After your target mirror registry is populated with the initial image set, be sure to update it regularly so
that it has the latest content. You can optionally set up a cron job, if possible, so that the mirror registry
is updated on a regular basis.
146
Ensure that you update your image set configuration to add or remove OpenShift Container Platform
and Operator releases as necessary. Any images that are removed are pruned from the mirror registry.
4.4.9.1. About updating your mirror registry content
When you run the oc-mirror plugin again, it generates an image set that only contains new and updated
images since the previous execution. Because it only pulls in the differences since the previous image
set was created, the generated image set is often smaller and faster to process than the initial image
set.
IMPORTANT
Generated image sets are sequential and must be pushed to the target mirror registry in
order. You can derive the sequence number from the file name of the generated image
set archive file.
Adding new and updated images

Depending on the settings in your image set configuration, future executions of oc-mirror can mirror
additional new and updated images. Review the settings in your image set configuration to ensure that
you are retrieving new versions as necessary. For example, you can set the minimum and maximum
versions of Operators to mirror if you want to restrict to specific versions. Alternatively, you can set the
minimum version as a starting point to mirror, but keep the version range open so you keep receiving
new Operator versions on future executions of oc-mirror. Omitting any minimum or maximum version
gives you the full version history of an Operator in a channel. Omitting explicitly named channels gives
you all releases in all channels of the specified Operator. Omitting any named Operator gives you the
entire catalog of all Operators and all their versions ever released.
All these constraints and conditions are evaluated against the publicly released content by Red Hat on
every invocation of oc-mirror. This way, it automatically picks up new releases and entirely new
Operators. Constraints can be specified by only listing a desired set of Operators, which will not
automatically add other newly released Operators into the mirror set. You can also specify a particular
release channel, which limits mirroring to just this channel and not any new channels that have been
added. This is important for Operator products, such as Red Hat Quay, that use different release
channels for their minor releases. Lastly, you can specify a maximum version of a particular Operator,
which causes the tool to only mirror the specified version range so that you do not automatically get any
newer releases past the maximum version mirrored. In all these cases, you must update the image set
configuration file to broaden the scope of the mirroring of Operators to get other Operators, new
channels, and newer versions of Operators to be available in your target registry.
It is recommended to align constraints like channel specification or version ranges with the release
strategy that a particular Operator has chosen. For example, when the Operator uses a stable channel,
you should restrict mirroring to that channel and potentially a minimum version to find the right balance
between download volume and getting stable updates regularly. If the Operator chooses a release
version channel scheme, for example stable-3.7, you should mirror all releases in that channel. This
allows you to keep receiving patch versions of the Operator, for example 3.7.1. You can also regularly
adjust the image set configuration to add channels for new product releases, for example stable-3.8.
Pruning images
Images are pruned automatically from the target mirror registry if they are no longer included in the
latest image set that was generated and mirrored. This allows you to easily manage and clean up
unneeded content and reclaim storage resources.
If there are OpenShift Container Platform releases or Operator versions that you no longer need, you
can modify your image set configuration to exclude them, and they will be pruned from the mirror
registry upon mirroring. This can be done by adjusting a minimum or maximum version range setting per
147
Operator in the image set configuration file or by deleting the Operator from the list of Operators to
mirror from the catalog. You can also remove entire Operator catalogs or entire OpenShift Container
Platform releases from the configuration file.
IMPORTANT
If there are no new or updated images to mirror, the excluded images are not pruned
from the target mirror registry. Additionally, if an Operator publisher removes an
Operator version from a channel, the removed versions are pruned from the target mirror
registry.
To disable automatic pruning of images from the target mirror registry, pass the --skip-pruning flag to
the oc mirror command.
4.4.9.2. Updating your mirror registry content
After you publish the initial image set to the mirror registry, you can use the oc-mirror plugin to keep
your disconnected clusters updated.
Depending on your image set configuration, oc-mirror automatically detects newer releases of
OpenShift Container Platform and your selected Operators that have been released after you
completed the initial mirror. It is recommended to run oc-mirror at regular intervals, for example in a
nightly cron job, to receive product and security updates on a timely basis.
Prerequisites
You have used the oc-mirror plugin to mirror the initial image set to your mirror registry.
You have access to the storage backend that was used for the initial execution of the oc-mirror
plugin.
NOTE
You must use the same storage backend as the initial execution of oc-mirror for
the same mirror registry. Do not delete or modify the metadata image that is
generated by the oc-mirror plugin.
Procedure
1. If necessary, update your image set configuration file to pick up new OpenShift Container
Platform and Operator versions. See Image set configuration examples for example mirroring
use cases.
2. Follow the same steps that you used to mirror your initial image set to the mirror registry. For
instructions, see Mirroring an image set in a partially disconnected environment or Mirroring an
image set in a fully disconnected environment.
IMPORTANT
148
IMPORTANT
You must provide the same storage backend so that only a differential image
set is created and mirrored.
If you specified a top-level namespace for the mirror registry during the
initial image set creation, then you must use this same namespace every time
you run the oc-mirror plugin for the same mirror registry.
3. Configure your cluster to use the resources generated by oc-mirror.
Image set configuration examples
Mirroring an image set in a partially disconnected environment
Mirroring an image set in a fully disconnected environment
Configuring your cluster to use the resources generated by oc-mirror
4.4.10. Performing a dry run

You can use oc-mirror to perform a dry run, without actually mirroring any images. This allows you to
review the list of images that would be mirrored, as well as any images that would be pruned from the
mirror registry. A dry run also allows you to catch any errors with your image set configuration early or
use the generated list of images with other tools to carry out the mirroring operation.
Prerequisites
Procedure
1. Run the oc mirror command with the --dry-run flag to perform a dry run:
docker://registry.example:5000 \ 2
--dry-run 3
1 Pass in the image set configuration file that was created. This procedure assumes that it is
named imageset-config.yaml.
2 Specify the mirror registry. Nothing is mirrored to this registry as long as you use the --dry-
run flag.
3 Use the --dry-run flag to generate the dry run artifacts and not an actual image set file.
149
Example output
Checking push permissions for registry.example:5000

Creating directory: oc-mirror-workspace/src/publish
Creating directory: oc-mirror-workspace/src/v2
Creating directory: oc-mirror-workspace/src/charts
Creating directory: oc-mirror-workspace/src/release-signatures
No metadata detected, creating new workspace
wrote mirroring manifests to oc-mirror-workspace/operators.1658342351/manifests-redhat-
operator-index
...
info: Planning completed in 31.48s

info: Dry run complete
Writing image mapping to oc-mirror-workspace/mapping.txt
2. Navigate into the workspace directory that was generated:
$ cd oc-mirror-workspace/
3. Review the mapping.txt file that was generated.

This file contains a list of all images that would be mirrored.
4. Review the pruning-plan.json file that was generated.

This file contains a list of all images that would be pruned from the mirror registry when the
image set is published.
NOTE
The pruning-plan.json file is only generated if your oc-mirror command points

to your mirror registry and there are images to be pruned.
4.4.11. Including local OCI Operator catalogs

While mirroring OpenShift Container Platform releases, Operator catalogs, and additional images from a
registry to a partially disconnected cluster, you can include Operator catalog images from a local file-
based catalog on disk. The local catalog must be in the Open Container Initiative (OCI) format.
The local catalog and its contents are mirrored to your target mirror registry based on the filtering
information in the image set configuration file.
IMPORTANT
When mirroring local OCI catalogs, any OpenShift Container Platform releases or
additional images that you want to mirror along with the local OCI-formatted catalog
must be pulled from a registry.
You cannot mirror OCI catalogs along with an oc-mirror image set file on disk.
One example use case for using the OCI feature is if you have a CI/CD system building an OCI catalog to
a location on disk, and you want to mirror that OCI catalog along with an OpenShift Container Platform
release to your mirror registry.
150
NOTE
If you used the Technology Preview OCI local catalogs feature for the oc-mirror plugin
for OpenShift Container Platform 4.12, you can no longer use the OCI local catalogs
feature of the oc-mirror plugin to copy a catalog locally and convert it to OCI format as a
first step to mirroring to a fully disconnected cluster.
Prerequisites
Procedure
1. Create the image set configuration file and adjust the settings as necessary.
The following example image set configuration mirrors an OCI catalog on disk along with an
OpenShift Container Platform release and a UBI image from registry.redhat.io.
storageConfig:
local:
path: /home/user/metadata 1
mirror:
platform:
channels:
- name: stable-4.15 2
type: ocp
graph: false
operators:
- catalog: oci:///home/user/oc-mirror/my-oci-catalog 3
targetCatalog: my-namespace/redhat-operator-index 4
packages:
- name: aws-load-balancer-operator
- catalog: registry.redhat.io/redhat/redhat-operator-index:v4.15 5
packages:
- name: rhacs-operator
additionalImages:
- name: registry.redhat.io/ubi9/ubi:latest 6
1 Set the back-end location to save the image set metadata to. This location can be a
registry or local directory. It is required to specify storageConfig values.
2 Optionally, include an OpenShift Container Platform release to mirror from

registry.redhat.io.
3 Specify the absolute path to the location of the OCI catalog on disk. The path must start
with oci:// when using the OCI feature.
4 Optionally, specify an alternative namespace and name to mirror the catalog as.
5 Optionally, specify additional Operator catalogs to pull from a registry.
151
6 Optionally, specify additional images to pull from a registry.
2. Run the oc mirror command to mirror the OCI catalog to a target mirror registry:
1 Pass in the image set configuration file. This procedure assumes that it is named
imageset-config.yaml.
2 Specify the registry to mirror the content to. The registry must start with docker://. If you
specify a top-level namespace for the mirror registry, you must also use this same
Optionally, you can specify other flags to adjust the behavior of the OCI feature:
--oci-insecure-signature-policy
Do not push signatures to the target mirror registry.
--oci-registries-config
Specify the path to a TOML-formatted registries.conf file. You can use this to mirror from a
different registry, such as a pre-production location for testing, without having to change the
image set configuration file. This flag only affects local OCI catalogs, not any other mirrored
content.
Example registries.conf file
[[registry]]
location = "registry.redhat.io:5000"
insecure = false
blocked = false
mirror-by-digest-only = true
prefix = ""
[[registry.mirror]]
location = "preprod-registry.example.com"
insecure = false
Next steps
Configuring your cluster to use the resources generated by oc-mirror
4.4.12. Image set configuration parameters

The oc-mirror plugin requires an image set configuration file that defines what images to mirror. The
following table lists the available parameters for the ImageSetConfiguration resource.
Table 4.1. ImageSetConfiguration parameters
152
Parameter Description Values
apiVersion The API version for the String. For

ImageSetConfiguration content. example:
mirror.openshif
t.io/v1alpha2 .
archiveSize The maximum size, in GiB, of each archive Integer. For

file within the image set. example: 4
mirror The configuration of the image set. Object
mirror.additionalImages The additional images configuration of Array of objects.

the image set. For example:
additionalIma
ges:
- name:
registry.redha
t.io/ubi8/ubi:lat
est
mirror.additionalImages.name The tag or digest of the image to mirror. String. For

example:
registry.redhat.i
o/ubi8/ubi:latest
mirror.blockedImages The full tag, digest, or pattern of images Array of strings.

to block from mirroring. For example:
docker.io/librar
y/alpine
mirror.helm The helm configuration of the image set. Object

Note that the oc-mirror plugin supports
only helm charts that do not require user
input when rendered.
mirror.helm.local The local helm charts to mirror. Array of objects.

For example:
local:
- name:
podinfo
path:
/test/podinfo-
5.0.0.tar.gz
mirror.helm.local.name The name of the local helm chart to String. For

mirror. example: podinfo.
153
mirror.helm.local.path The path of the local helm chart to mirror. String. For
example:
/test/podinfo-
5.0.0.tar.gz.
mirror.helm.repositories The remote helm repositories to mirror Array of objects.

from. For example:
repositories:
- name:
podinfo
url:
https://exampl
e.github.io/po
dinfo
charts:
- name:
podinfo
version:
5.0.0
mirror.helm.repositories.name The name of the helm repository to mirror String. For

from. example: podinfo.
mirror.helm.repositories.url The URL of the helm repository to mirror String. For

from. example:
https://example.
github.io/podinf
o.
mirror.helm.repositories.charts The remote helm charts to mirror. Array of objects.
mirror.helm.repositories.charts.na The name of the helm chart to mirror. String. For

me example: podinfo.
mirror.helm.repositories.charts.ver The version of the named helm chart to String. For

sion mirror. example: 5.0.0.
154
mirror.operators The Operators configuration of the image Array of objects.

set. For example:
operators:
- catalog:
registry.redha
t.io/redhat/red
hat-operator-
index:v4.15
packages:
- name:
elasticsearch-
operator
minVersion:
'2.4.0'
mirror.operators.catalog The Operator catalog to include in the String. For

image set. example:
registry.redhat.i
o/redhat/redhat-
operator-
index:v4.15.
mirror.operators.full When true, downloads the full catalog, Boolean. The

Operator package, or Operator channel. default value is
false.
mirror.operators.packages The Operator packages configuration. Array of objects.

For example:
operators:
- catalog:
registry.redha
t.io/redhat/red
hat-operator-
index:v4.15
packages:
- name:
elasticsearch-
operator
minVersion:
'5.2.3-31'
mirror.operators.packages.name The Operator package name to include in String. For

the image set example:
elasticsearch-
operator .
155
mirror.operators.packages.channel The Operator package channel Object

s configuration.
mirror.operators.packages.channel The Operator channel name, unique String. For

s.name within a package, to include in the image example: fast or
set. stable-v4.15.
mirror.operators.packages.channel The highest version of the Operator String. For

s.maxVersion mirror across all channels in which it example: 5.2.3-31
exists. See the following note for further
information.
mirror.operators.packages.channel The name of the minimum bundle to String. For

s.minBundle include, plus all bundles in the update example:
graph to the channel head. Set this field bundleName
only if the named bundle has no semantic
version metadata.
mirror.operators.packages.channel The lowest version of the Operator to String. For

s.minVersion mirror across all channels in which it example: 5.2.3-31
information.
mirror.operators.packages.maxVers The highest version of the Operator to String. For

ion mirror across all channels in which it example: 5.2.3-31.
information.
mirror.operators.packages.minVers The lowest version of the Operator to String. For

ion mirror across all channels in which it example: 5.2.3-31.
information.
mirror.operators.skipDependencies If true, dependencies of bundles are not Boolean. The

included. default value is
false.
mirror.operators.targetCatalog An alternative name and optional String. For

namespace hierarchy to mirror the example: my-
referenced catalog as. namespace/my-
operator-
catalog
156
mirror.operators.targetName An alternative name to mirror the String. For

referenced catalog as. example: my-
operator-
The targetName parameter is catalog
deprecated. Use the targetCatalog
parameter instead.
mirror.operators.targetTag An alternative tag to append to the String. For

targetName or targetCatalog. example: v1
mirror.platform The platform configuration of the image Object

set.
mirror.platform.architectures The architecture of the platform release Array of strings.

payload to mirror. For example:
architectures:
- amd64
- arm64
- multi
- ppc64le
- s390x
The default value

is amd64. The
value multi
ensures that the
mirroring is
supported for all
available
architectures,
eliminating the
need to specify
individual
architectures.
mirror.platform.channels The platform channel configuration of the Array of objects.

image set. For example:
channels:
- name:
stable-4.10
- name:
stable-4.15
mirror.platform.channels.full When true, sets the minVersion to the Boolean. The

first release in the channel and the default value is
maxVersion to the last release in the false.
channel.
157
mirror.platform.channels.name The name of the release channel. String. For

example: stable-
4.15
mirror.platform.channels.minVersio The minimum version of the referenced String. For

n platform to be mirrored. example: 4.12.6
mirror.platform.channels.maxVersi The highest version of the referenced String. For

on platform to be mirrored. example: 4.15.1
mirror.platform.channels.shortestP Toggles shortest path mirroring or full Boolean. The

ath range mirroring. default value is
false.
mirror.platform.channels.type The type of the platform to be mirrored. String. For

example: ocp or
okd . The default
is ocp .
mirror.platform.graph Indicates whether the OSUS graph is Boolean. The

added to the image set and subsequently default value is
published to the mirror. false.
storageConfig The back-end configuration of the image Object

set.
storageConfig.local The local back-end configuration of the Object

image set.
storageConfig.local.path The path of the directory to contain the String. For

image set metadata. example:
./path/to/dir/.
storageConfig.registry The registry back-end configuration of Object

the image set.
storageConfig.registry.imageURL The back-end registry URI. Can optionally String. For

include a namespace reference in the example:
URI. quay.io/myuser/
imageset:metad
ata.
storageConfig.registry.skipTLS Optionally skip TLS verification of the Boolean. The

referenced back-end registry. default value is
false.
NOTE
158
NOTE
Using the minVersion and maxVersion properties to filter for a specific Operator
version range can result in a multiple channel heads error. The error message states that
there are multiple channel heads. This is because when the filter is applied, the update
graph of the Operator is truncated.
Operator Lifecycle Manager requires that every Operator channel contains versions that
form an update graph with exactly one end point, that is, the latest version of the
Operator. When the filter range is applied, that graph can turn into two or more separate
graphs or a graph that has more than one end point.
To avoid this error, do not filter out the latest version of an Operator. If you still run into
the error, depending on the Operator, either the maxVersion property must be increased
or the minVersion property must be decreased. Because every Operator graph can be
different, you might need to adjust these values until the error resolves.
4.4.13. Image set configuration examples

The following ImageSetConfiguration file examples show the configuration for various mirroring use
cases.
Use case: Including the shortest OpenShift Container Platform update path
The following ImageSetConfiguration file uses a local storage backend and includes all OpenShift
Container Platform versions along the shortest update path from the minimum version of 4.11.37 to the
maximum version of 4.12.15.
Example ImageSetConfiguration file
storageConfig:
local:
path: /home/user/metadata
mirror:
platform:
channels:
- name: stable-4.12
minVersion: 4.11.37
maxVersion: 4.12.15
shortestPath: true
Use case: Including all versions of OpenShift Container Platform from a minimum to the latest
version for multi-architecture releases
The following ImageSetConfiguration file uses a registry storage backend and includes all OpenShift
Container Platform versions starting at a minimum version of 4.13.4 to the latest version in the channel.
On every invocation of oc-mirror with this image set configuration, the latest release of the stable-4.13
channel is evaluated, so running oc-mirror at regular intervals ensures that you automatically receive the
latest releases of OpenShift Container Platform images.
By setting the value of platform.architectures to multi, you can ensure that the mirroring is supported
for multi-architecture releases.
159
storageConfig:
registry:
imageURL: example.com/mirror/oc-mirror-metadata
skipTLS: false
mirror:
platform:
architectures:
- "multi"
channels:
- name: stable-4.13
minVersion: 4.13.4
maxVersion: 4.13.6
Use case: Including Operator versions from a minimum to the latest

The following ImageSetConfiguration file uses a local storage backend and includes only the Red Hat
Advanced Cluster Security for Kubernetes Operator, versions starting at 4.0.1 and later in the stable
channel.
NOTE
When you specify a minimum or maximum version range, you might not receive all
Operator versions in that range.
By default, oc-mirror excludes any versions that are skipped or replaced by a newer
version in the Operator Lifecycle Manager (OLM) specification. Operator versions that
are skipped might be affected by a CVE or contain bugs. Use a newer version instead. For
more information on skipped and replaced versions, see Creating an update graph with
OLM.
To receive all Operator versions in a specified range, you can set the
mirror.operators.full field to true.
storageConfig:
local:
path: /home/user/metadata
mirror:
operators:
- catalog: registry.redhat.io/redhat/redhat-operator-index:v4.15
packages:
- name: rhacs-operator
channels:
- name: stable
minVersion: 4.0.1
NOTE
To specify a maximum version instead of the latest, set the

mirror.operators.packages.channels.maxVersion field.
160
Use case: Including the Nutanix CSI Operator

The following ImageSetConfiguration file uses a local storage backend and includes the Nutanix CSI
Operator, the OpenShift Update Service (OSUS) graph image, and an additional Red Hat Universal
Base Image (UBI).
storageConfig:
registry:
imageURL: mylocalregistry/ocp-mirror/openshift4
skipTLS: false
mirror:
platform:
channels:
- name: stable-4.11
type: ocp
graph: true
operators:
- catalog: registry.redhat.io/redhat/certified-operator-index:v4.15
packages:
- name: nutanixcsioperator
channels:
- name: stable
additionalImages:
- name: registry.redhat.io/ubi9/ubi:latest
Use case: Including the default Operator channel

The following ImageSetConfiguration file includes the stable-5.7 and stable channels for the
OpenShift Elasticsearch Operator. Even if only the packages from the stable-5.7 channel are needed,
the stable channel must also be included in the ImageSetConfiguration file, because it is the default
channel for the Operator. You must always include the default channel for the Operator package even if
you do not use the bundles in that channel.
TIP
You can find the default channel by running the following command: oc mirror list operators --
catalog=<catalog_name> --package=<package_name>.
storageConfig:
registry:
skipTLS: false
mirror:
operators:
packages:
- name: elasticsearch-operator
161
channels:
- name: stable-5.7
- name: stable
Use case: Including an entire catalog (all versions)

The following ImageSetConfiguration file sets the mirror.operators.full field to true to include all
versions for an entire Operator catalog.
storageConfig:
registry:
skipTLS: false
mirror:
operators:
full: true
Use case: Including an entire catalog (channel heads only)

The following ImageSetConfiguration file includes the channel heads for an entire Operator catalog.
By default, for each Operator in the catalog, oc-mirror includes the latest Operator version (channel
head) from the default channel. If you want to mirror all Operator versions, and not just the channel
heads, you must set the mirror.operators.full field to true.
This example also uses the targetCatalog field to specify an alternative namespace and name to mirror
the catalog as.
storageConfig:
registry:
skipTLS: false
mirror:
operators:
targetCatalog: my-namespace/my-operator-catalog
Use case: Including arbitrary images and helm charts

The following ImageSetConfiguration file uses a registry storage backend and includes helm charts and
an additional Red Hat Universal Base Image (UBI).
archiveSize: 4
storageConfig:
162
registry:
skipTLS: false
mirror:
platform:
architectures:
- "s390x"
channels:
- name: stable-4.15
operators:
helm:
repositories:
- name: redhat-helm-charts
url: https://raw.githubusercontent.com/redhat-developer/redhat-helm-charts/master
charts:
- name: ibm-mongodb-enterprise-helm
version: 0.2.0
additionalImages:
- name: registry.redhat.io/ubi9/ubi:latest
4.4.14. Command reference for oc-mirror

The following tables describe the oc mirror subcommands and flags:
Table 4.2. oc mirror subcommands
Subcommand Description
completion Generate the autocompletion script for the specified shell.
describe Output the contents of an image set.
help Show help about any subcommand.
init Output an initial image set configuration template.
list List available platform and Operator content and their version.
version Output the oc-mirror version.
Table 4.3. oc mirror flags
Flag Description
-c, --config <string> Specify the path to an image set configuration file.
--continue-on-error If any non image-pull related error occurs, continue and attempt to
mirror as much as possible.
163
Flag Description
--dest-skip-tls Disable TLS validation for the target registry.
--dest-use-http Use plain HTTP for the target registry.
--dry-run Print actions without mirroring images. Generates mapping.txt and

pruning-plan.json files.
--from <string> Specify the path to an image set archive that was generated by an
execution of oc-mirror to load into a target registry.
-h, --help Show the help.
--ignore-history Ignore past mirrors when downloading images and packing layers.
Disables incremental mirroring and might download more data.
--manifests-only Generate manifests for ImageContentSourcePolicy objects to

configure a cluster to use the mirror registry, but do not actually mirror
any images. To use this flag, you must pass in an image set archive with
the --from flag.
--max-nested-paths <int> Specify the maximum number of nested paths for destination registries
that limit nested paths. The default is 0.
--max-per-registry <int> Specify the number of concurrent requests allowed per registry. The
default is 6.
--oci-insecure-signature- Do not push signatures when mirroring local OCI catalogs (with --
policy include-local-oci-catalogs).
--oci-registries-config Provide a registries configuration file to specify an alternative registry

location to copy from when mirroring local OCI catalogs (with --
include-local-oci-catalogs).
--skip-cleanup Skip removal of artifact directories.
--skip-image-pin Do not replace image tags with digest pins in Operator catalogs.
--skip-metadata-check Skip metadata when publishing an image set. This is only recommended
when the image set was created with --ignore-history.
--skip-missing If an image is not found, skip it instead of reporting an error and aborting
execution. Does not apply to custom images explicitly specified in the
image set configuration.
--skip-pruning Disable automatic pruning of images from the target mirror registry.
--skip-verification Skip digest verification.
164
Flag Description
--source-skip-tls Disable TLS validation for the source registry.
--source-use-http Use plain HTTP for the source registry.
-v, --verbose <int> Specify the number for the log level verbosity. Valid values are 0 - 9. The
default is 0.

About cluster updates in a disconnected environment
165
CHAPTER 5. INSTALLING ON ALIBABA
5.1. PREPARING TO INSTALL ON ALIBABA CLOUD
IMPORTANT
Alibaba Cloud on OpenShift Container Platform is a Technology Preview feature only.

Technology Preview features are not supported with Red Hat production service level
agreements (SLAs) and might not be functionally complete. Red Hat does not
recommend using them in production. These features provide early access to upcoming
product features, enabling customers to test functionality and provide feedback during
the development process.
For more information about the support scope of Red Hat Technology Preview features,
see Technology Preview Features Support Scope .
You reviewed details about the OpenShift Container Platform installation and update
processes.
You read the documentation on selecting a cluster installation method and preparing it for
users.
5.1.2. Requirements for installing OpenShift Container Platform on Alibaba Cloud

Before installing OpenShift Container Platform on Alibaba Cloud, you must configure and register your
domain, create a Resource Access Management (RAM) user for the installation, and review the
supported Alibaba Cloud data center regions and zones for the installation.
5.1.3. Registering and Configuring Alibaba Cloud Domain

To install OpenShift Container Platform, the Alibaba Cloud account you use must have a dedicated
public hosted zone in your account. This zone must be authoritative for the domain. This service
provides cluster DNS resolution and name lookup for external connections to the cluster.
Procedure
1. Identify your domain, or subdomain, and registrar. You can transfer an existing domain and
registrar or obtain a new one through Alibaba Cloud or another source.
NOTE
If you purchase a new domain through Alibaba Cloud, it takes time for the
relevant DNS changes to propagate. For more information about purchasing
domains through Alibaba Cloud, see Alibaba Cloud domains.
2. If you are using an existing domain and registrar, migrate its DNS to Alibaba Cloud. See Domain
name transfer in the Alibaba Cloud documentation.
3. Configure DNS for your domain. This includes:
Registering a generic domain name .
166
Completing real-name verification for your domain name .
Applying for an Internet Content Provider (ICP) filing .
Enabling domain name resolution.

Use an appropriate root domain, such as openshiftcorp.com, or subdomain, such as
clusters.openshiftcorp.com.
4. If you are using a subdomain, follow the procedures of your company to add its delegation
records to the parent domain.
5.1.4. Supported Alibaba regions

You can deploy an OpenShift Container Platform cluster to the regions listed in the Alibaba Regions and
zones documentation.
5.1.5. Next steps

Create the required Alibaba Cloud resources .
5.2. CREATING THE REQUIRED ALIBABA CLOUD RESOURCES

Before you install OpenShift Container Platform, you must use the Alibaba Cloud console to create a
Resource Access Management (RAM) user that has sufficient permissions to install OpenShift
Container Platform into your Alibaba Cloud. This user must also have permissions to create new RAM
users. You can also configure and use the ccoctl tool to create new credentials for the OpenShift
Container Platform components with the permissions that they require.
IMPORTANT

5.2.1. Creating the required RAM user

You must have a Alibaba Cloud Resource Access Management (RAM) user for the installation that has
sufficient privileges. You can use the Alibaba Cloud Resource Access Management console to create a
new user or modify an existing user. Later, you create credentials in OpenShift Container Platform
based on this user’s permissions.
When you configure the RAM user, be sure to consider the following requirements:
The user must have an Alibaba Cloud AccessKey ID and AccessKey secret pair.
For a new user, you can select Open API Access for the Access Mode when creating the
user. This mode generates the required AccessKey pair.
For an existing user, you can add an AccessKey pair or you can obtain the AccessKey pair for
167
For an existing user, you can add an AccessKey pair or you can obtain the AccessKey pair for
that user.
NOTE
When created, the AccessKey secret is displayed only once. You must
immediately save the AccessKey pair because the AccessKey pair is required
for API calls.
Add the AccessKey ID and secret to the ~/.alibabacloud/credentials file on your local
computer. Alibaba Cloud automatically creates this file when you log in to the console. The
Cloud Credential Operator (CCO) utility, ccoutil, uses these credentials when processing
Credential Request objects.
For example:
[default] # Default client

type = access_key # Certification type: access_key
access_key_id = LTAI5t8cefXKmt # Key 1
access_key_secret = wYx56mszAN4Uunfh # Secret
1 Add your AccessKeyID and AccessKeySecret here.
The RAM user must have the AdministratorAccess policy to ensure that the account has
sufficient permission to create the OpenShift Container Platform cluster. This policy grants
permissions to manage all Alibaba Cloud resources.
When you attach the AdministratorAccess policy to a RAM user, you grant that user full access
to all Alibaba Cloud services and resources. If you do not want to create a user with full access,
create a custom policy with the following actions that you can add to your RAM user for
installation. These actions are sufficient to install OpenShift Container Platform.
TIP
You can copy and paste the following JSON code into the Alibaba Cloud console to create a
custom poicy. For information on creating custom policies, see Create a custom policy in the
Alibaba Cloud documentation.
Example 5.1. Example custom policy JSON file
{
"Version": "1",
"Statement": [
{
"Action": [
"tag:ListTagResources",
"tag:UntagResources"
],
"Resource": "*",
"Effect": "Allow"
},
{
"Action": [
"vpc:DescribeVpcs",
"vpc:DeleteVpc",
168
"vpc:DescribeVSwitches",
"vpc:DeleteVSwitch",
"vpc:DescribeEipAddresses",
"vpc:DescribeNatGateways",
"vpc:ReleaseEipAddress",
"vpc:DeleteNatGateway",
"vpc:DescribeSnatTableEntries",
"vpc:CreateSnatEntry",
"vpc:AssociateEipAddress",
"vpc:ListTagResources",
"vpc:TagResources",
"vpc:DescribeVSwitchAttributes",
"vpc:CreateVSwitch",
"vpc:CreateNatGateway",
"vpc:DescribeRouteTableList",
"vpc:CreateVpc",
"vpc:AllocateEipAddress",
"vpc:ListEnhanhcedNatGatewayAvailableZones"
],
"Resource": "*",
"Effect": "Allow"
},
{
"Action": [
"ecs:ModifyInstanceAttribute",
"ecs:DescribeSecurityGroups",
"ecs:DeleteSecurityGroup",
"ecs:DescribeSecurityGroupReferences",
"ecs:DescribeSecurityGroupAttribute",
"ecs:RevokeSecurityGroup",
"ecs:DescribeInstances",
"ecs:DeleteInstances",
"ecs:DescribeNetworkInterfaces",
"ecs:DescribeInstanceRamRole",
"ecs:DescribeUserData",
"ecs:DescribeDisks",
"ecs:ListTagResources",
"ecs:AuthorizeSecurityGroup",
"ecs:RunInstances",
"ecs:TagResources",
"ecs:ModifySecurityGroupPolicy",
"ecs:CreateSecurityGroup",
"ecs:DescribeAvailableResource",
"ecs:DescribeRegions",
"ecs:AttachInstanceRamRole"
],
"Resource": "*",
"Effect": "Allow"
},
{
"Action": [
"pvtz:DescribeRegions",
"pvtz:DescribeZones",
"pvtz:DeleteZone",
"pvtz:DeleteZoneRecord",
"pvtz:BindZoneVpc",
169
"pvtz:DescribeZoneRecords",
"pvtz:AddZoneRecord",
"pvtz:SetZoneRecordStatus",
"pvtz:DescribeZoneInfo",
"pvtz:DescribeSyncEcsHostTask",
"pvtz:AddZone"
],
"Resource": "*",
"Effect": "Allow"
},
{
"Action": [
"slb:DescribeLoadBalancers",
"slb:SetLoadBalancerDeleteProtection",
"slb:DeleteLoadBalancer",
"slb:SetLoadBalancerModificationProtection",
"slb:DescribeLoadBalancerAttribute",
"slb:AddBackendServers",
"slb:DescribeLoadBalancerTCPListenerAttribute",
"slb:SetLoadBalancerTCPListenerAttribute",
"slb:StartLoadBalancerListener",
"slb:CreateLoadBalancerTCPListener",
"slb:ListTagResources",
"slb:TagResources",
"slb:CreateLoadBalancer"
],
"Resource": "*",
"Effect": "Allow"
},
{
"Action": [
"ram:ListResourceGroups",
"ram:DeleteResourceGroup",
"ram:ListPolicyAttachments",
"ram:DetachPolicy",
"ram:GetResourceGroup",
"ram:CreateResourceGroup",
"ram:DeleteRole",
"ram:GetPolicy",
"ram:DeletePolicy",
"ram:ListPoliciesForRole",
"ram:CreateRole",
"ram:AttachPolicyToRole",
"ram:GetRole",
"ram:CreatePolicy",
"ram:CreateUser",
"ram:DetachPolicyFromRole",
"ram:CreatePolicyVersion",
"ram:DetachPolicyFromUser",
"ram:ListPoliciesForUser",
"ram:AttachPolicyToUser",
"ram:CreateUser",
"ram:GetUser",
"ram:DeleteUser",
"ram:CreateAccessKey",
"ram:ListAccessKeys",
170
"ram:DeleteAccessKey",
"ram:ListUsers",
"ram:ListPolicyVersions"
],
"Resource": "*",
"Effect": "Allow"
},
{
"Action": [
"oss:DeleteBucket",
"oss:DeleteBucketTagging",
"oss:GetBucketTagging",
"oss:GetBucketCors",
"oss:GetBucketPolicy",
"oss:GetBucketLifecycle",
"oss:GetBucketReferer",
"oss:GetBucketTransferAcceleration",
"oss:GetBucketLog",
"oss:GetBucketWebSite",
"oss:GetBucketInfo",
"oss:PutBucketTagging",
"oss:PutBucket",
"oss:OpenOssService",
"oss:ListBuckets",
"oss:GetService",
"oss:PutBucketACL",
"oss:GetBucketLogging",
"oss:ListObjects",
"oss:GetObject",
"oss:PutObject",
"oss:DeleteObject"
],
"Resource": "*",
"Effect": "Allow"
},
{
"Action": [
"alidns:DescribeDomainRecords",
"alidns:DeleteDomainRecord",
"alidns:DescribeDomains",
"alidns:DescribeDomainRecordInfo",
"alidns:AddDomainRecord",
"alidns:SetDomainRecordStatus"
],
"Resource": "*",
"Effect": "Allow"
},
{
"Action": "bssapi:CreateInstance",
"Resource": "*",
"Effect": "Allow"
},
{
"Action": "ram:PassRole",
"Resource": "*",
"Effect": "Allow",
171
"Condition": {
"StringEquals": {
"acs:Service": "ecs.aliyuncs.com"
}
}
}
]
}
For more information about creating a RAM user and granting permissions, see Create a RAM user and
Grant permissions to a RAM user in the Alibaba Cloud documentation.
5.2.2. Configuring the Cloud Credential Operator utility

To assign RAM users and policies that provide long-term RAM AccessKeys (AKs) for each in-cluster
component, extract and prepare the Cloud Credential Operator (CCO) utility (ccoctl) binary.
NOTE
The ccoctl utility is a Linux binary that must run in a Linux environment.
Prerequisites
You have access to an OpenShift Container Platform account with cluster administrator access.
Procedure
1. Obtain the OpenShift Container Platform release image by running the following command:
$ RELEASE_IMAGE=$(./openshift-install version | awk '/release image/ {print $3}')
2. Obtain the CCO container image from the OpenShift Container Platform release image by
$ CCO_IMAGE=$(oc adm release info --image-for='cloud-credential-operator'

$RELEASE_IMAGE -a ~/.pull-secret)
NOTE
Ensure that the architecture of the $RELEASE_IMAGE matches the

architecture of the environment in which you will use the ccoctl tool.
3. Extract the ccoctl binary from the CCO container image within the OpenShift Container
Platform release image by running the following command:
$ oc image extract $CCO_IMAGE --file="/usr/bin/ccoctl" -a ~/.pull-secret
4. Change the permissions to make ccoctl executable by running the following command:
172
$ chmod 775 ccoctl
Verification
To verify that ccoctl is ready to use, display the help file by running the following command:
$ ccoctl --help
Output of ccoctl --help
OpenShift credentials provisioning tool
Usage:
ccoctl [command]
Available Commands:
alibabacloud Manage credentials objects for alibaba cloud
aws Manage credentials objects for AWS cloud
azure Manage credentials objects for Azure
gcp Manage credentials objects for Google cloud
help Help about any command
ibmcloud Manage credentials objects for IBM Cloud
nutanix Manage credentials objects for Nutanix
Flags:
-h, --help help for ccoctl
Use "ccoctl [command] --help" for more information about a command.
Preparing to update a cluster with manually maintained credentials
5.2.3. Next steps

Install a cluster on Alibaba Cloud infrastructure that is provisioned by the OpenShift Container
Platform installation program, by using one of the following methods:
Installing a cluster quickly on Alibaba Cloud: You can install a cluster quickly by using the
default configuration options.
Installing a customized cluster on Alibaba Cloud: The installation program allows for some
customization to be applied at the installation stage. Many other customization options are
available post-installation.
5.3. INSTALLING A CLUSTER QUICKLY ON ALIBABA CLOUD

In OpenShift Container Platform version 4.15, you can install a cluster on Alibaba Cloud that uses the
IMPORTANT
173
IMPORTANT

processes.
users.
You registered your domain.
If you use a firewall, you configured it to allow the sites that your cluster requires access to.
You have created the required Alibaba Cloud resources .
If the cloud Resource Access Management (RAM) APIs are not accessible in your environment,
or if you do not want to store an administrator-level credential secret in the kube-system
namespace, you can manually create and maintain Resource Access Management (RAM)
credentials.
5.3.2. Internet access for OpenShift Container Platform

In OpenShift Container Platform 4.15, you require access to the internet to install your cluster.
You must have internet access to:
Access OpenShift Cluster Manager to download the installation program and perform
subscription management. If the cluster has internet access and you do not disable Telemetry,
that service automatically entitles your cluster.
Access Quay.io to obtain the packages that are required to install your cluster.
Obtain the packages that are required to perform cluster updates.
IMPORTANT
If your cluster cannot have direct internet access, you can perform a restricted network
installation on some types of infrastructure that you provision. During that process, you
download the required content and use it to populate a mirror registry with the
installation packages. With some installation types, the environment that you install your
cluster in will not require internet access. Before you update the cluster, you update the
content of the mirror registry.
5.3.3. Generating a key pair for cluster node SSH access
174
During an OpenShift Container Platform installation, you can provide an SSH public key to the
installation program. The key is passed to the Red Hat Enterprise Linux CoreOS (RHCOS) nodes
through their Ignition config files and is used to authenticate SSH access to the nodes. The key is added
to the ~/.ssh/authorized_keys list for the core user on each node, which enables password-less
authentication.
After the key is passed to the nodes, you can use the key pair to SSH in to the RHCOS nodes as the user
core. To access the nodes through SSH, the private key identity must be managed by SSH for your local
user.
If you want to SSH in to your cluster nodes to perform installation debugging or disaster recovery, you
must provide the SSH public key during the installation process. The ./openshift-install gather
command also requires the SSH public key to be in place on the cluster nodes.
IMPORTANT
Do not skip this procedure in production environments, where disaster recovery and
debugging is required.
NOTE
You must use a local key, not one that you configured with platform-specific approaches
such as AWS key pairs.
Procedure
1. If you do not have an existing SSH key pair on your local machine to use for authentication onto
your cluster nodes, create one. For example, on a computer that uses a Linux operating system,
run the following command:
$ ssh-keygen -t ed25519 -N '' -f <path>/<file_name> 1
1 Specify the path and file name, such as ~/.ssh/id_ed25519, of the new SSH key. If you have
an existing key pair, ensure your public key is in the your ~/.ssh directory.
NOTE
If you plan to install an OpenShift Container Platform cluster that uses the RHEL
cryptographic libraries that have been submitted to NIST for FIPS 140-2/140-3
Validation on only the x86_64, ppc64le, and s390x architectures, do not create a
key that uses the ed25519 algorithm. Instead, create a key that uses the rsa or
ecdsa algorithm.
2. View the public SSH key:
$ cat <path>/<file_name>.pub
For example, run the following to view the ~/.ssh/id_ed25519.pub public key:
$ cat ~/.ssh/id_ed25519.pub
3. Add the SSH private key identity to the SSH agent for your local user, if it has not already been
175
added. SSH agent management of the key is required for password-less SSH authentication
onto your cluster nodes, or if you want to use the ./openshift-install gather command.
NOTE
On some distributions, default SSH private key identities such as ~/.ssh/id_rsa

and ~/.ssh/id_dsa are managed automatically.
a. If the ssh-agent process is not already running for your local user, start it as a background
task:
$ eval "$(ssh-agent -s)"
Example output
Agent pid 31874
NOTE
If your cluster is in FIPS mode, only use FIPS-compliant algorithms to

generate the SSH key. The key must be either RSA or ECDSA.
4. Add your SSH private key to the ssh-agent:
$ ssh-add <path>/<file_name> 1
1 Specify the path and file name for your SSH private key, such as ~/.ssh/id_ed25519
Example output
Identity added: /home/<you>/<path>/<file_name> (<computer_name>)
Next steps
When you install OpenShift Container Platform, provide the SSH public key to the installation
program.
5.3.4. Obtaining the installation program

Before you install OpenShift Container Platform, download the installation file on the host you are using
for installation.
Prerequisites
You have a computer that runs Linux or macOS, with 500 MB of local disk space.
Procedure
1. Access the Infrastructure Provider page on the OpenShift Cluster Manager site. If you have a
176
Red Hat account, log in with your credentials. If you do not, create an account.
2. Select your infrastructure provider.
3. Navigate to the page for your installation type, download the installation program that
corresponds with your host operating system and architecture, and place the file in the
directory where you will store the installation configuration files.
IMPORTANT
The installation program creates several files on the computer that you use to
install your cluster. You must keep the installation program and the files that the
installation program creates after you finish installing the cluster. Both files are
required to delete the cluster.
IMPORTANT
Deleting the files created by the installation program does not remove your
cluster, even if the cluster failed during installation. To remove your cluster,
complete the OpenShift Container Platform uninstallation procedures for your
specific cloud provider.
4. Extract the installation program. For example, on a computer that uses a Linux operating
system, run the following command:
$ tar -xvf openshift-install-linux.tar.gz
5. Download your installation pull secret from Red Hat OpenShift Cluster Manager . This pull secret
allows you to authenticate with the services that are provided by the included authorities,
including Quay.io, which serves the container images for OpenShift Container Platform
components.
5.3.5. Creating the installation configuration file

You can customize the OpenShift Container Platform cluster you install on Alibaba Cloud.
Prerequisites
You have the OpenShift Container Platform installation program and the pull secret for your
cluster.
Procedure
1. Create the install-config.yaml file.
a. Change to the directory that contains the installation program and run the following
command:
$ ./openshift-install create install-config --dir <installation_directory> 1
1 For <installation_directory>, specify the directory name to store the files that the
installation program creates.
177
When specifying the directory:
Verify that the directory has the execute permission. This permission is required to run
Terraform binaries under the installation directory.
Use an empty directory. Some installation assets, such as bootstrap X.509 certificates,
have short expiration intervals, therefore you must not reuse an installation directory. If
you want to reuse individual files from another cluster installation, you can copy them
into your directory. However, the file names for the installation assets might change
between releases. Use caution when copying installation files from an earlier OpenShift
Container Platform version.
b. At the prompts, provide the configuration details for your cloud:
i. Optional: Select an SSH key to use to access your cluster machines.
NOTE
For production OpenShift Container Platform clusters on which you want

to perform installation debugging or disaster recovery, specify an SSH
key that your ssh-agent process uses.
ii. Select alibabacloud as the platform to target.
iii. Select the region to deploy the cluster to.
iv. Select the base domain to deploy the cluster to. The base domain corresponds to the
public DNS zone that you created for your cluster.
v. Provide a descriptive name for your cluster.
2. Installing the cluster into Alibaba Cloud requires that the Cloud Credential Operator (CCO)
operate in manual mode. Modify the install-config.yaml file to set the credentialsMode
parameter to Manual:
Example install-config.yaml configuration file with credentialsMode set to Manual
apiVersion: v1
baseDomain: cluster1.example.com
credentialsMode: Manual 1
compute:
- architecture: amd64
hyperthreading: Enabled
...
1 Add this line to set the credentialsMode to Manual.
3. Back up the install-config.yaml file so that you can use it to install multiple clusters.
IMPORTANT
The install-config.yaml file is consumed during the installation process. If you

want to reuse the file, you must back it up now.
178
5.3.6. Generating the required installation manifests

You must generate the Kubernetes manifest and Ignition config files that the cluster needs to configure
the machines.
Procedure
1. Generate the manifests by running the following command from the directory that contains the
installation program:
$ openshift-install create manifests --dir <installation_directory>
where:
<installation_directory>
Specifies the directory in which the installation program creates files.
5.3.7. Creating credentials for OpenShift Container Platform components with the
ccoctl tool
You can use the OpenShift Container Platform Cloud Credential Operator (CCO) utility to automate
the creation of Alibaba Cloud RAM users and policies for each in-cluster component.
NOTE
By default, ccoctl creates objects in the directory in which the commands are run. To
create the objects in a different directory, use the --output-dir flag. This procedure uses
<path_to_ccoctl_output_dir> to refer to this directory.
Prerequisites
You must have:
Extracted and prepared the ccoctl binary.
Created a RAM user with sufficient permission to create the OpenShift Container Platform
cluster.
Added the AccessKeyID (access_key_id) and AccessKeySecret (access_key_secret) of that

RAM user into the ~/.alibabacloud/credentials file on your local computer.
Procedure
1. Set a $RELEASE_IMAGE variable with the release image from your installation file by running
2. Extract the list of CredentialsRequest objects from the OpenShift Container Platform release
image by running the following command:
$ oc adm release extract \

--from=$RELEASE_IMAGE \
--credentials-requests \
179
--included \ 1
--install-config=<path_to_directory_with_installation_configuration>/install-config.yaml \ 2
--to=<path_to_directory_for_credentials_requests> 3
1 The --included parameter includes only the manifests that your specific cluster
configuration requires.
2 Specify the location of the install-config.yaml file.
3 Specify the path to the directory where you want to store the CredentialsRequest
objects. If the specified directory does not exist, this command creates it.
NOTE
This command might take a few moments to run.
3. Use the ccoctl tool to process all CredentialsRequest objects by running the following
command:
a. Run the following command to use the tool:
$ ccoctl alibabacloud create-ram-users \

--name <name> \ 1
--region=<alibaba_region> \ 2
--credentials-requests-dir=<path_to_credentials_requests_directory> \ 3
--output-dir=<path_to_ccoctl_output_dir> 4
1 Specify the name used to tag any cloud resources that are created for tracking.
2 Specify the Alibaba Cloud region in which cloud resources will be created.
3 Specify the directory containing the files for the component CredentialsRequest
objects.
4 Specify the directory where the generated component credentials secrets will be
placed.
NOTE
If your cluster uses Technology Preview features that are enabled by the
TechPreviewNoUpgrade feature set, you must include the --enable-tech-
preview parameter.
Example output
2022/02/11 16:18:26 Created RAM User: user1-alicloud-openshift-machine-api-

alibabacloud-credentials
2022/02/11 16:18:27 Ready for creating new ram policy user1-alicloud-openshift-
machine-api-alibabacloud-credentials-policy-policy
2022/02/11 16:18:27 RAM policy user1-alicloud-openshift-machine-api-alibabacloud-
credentials-policy-policy has created
2022/02/11 16:18:28 Policy user1-alicloud-openshift-machine-api-alibabacloud-
180
credentials-policy-policy has attached on user user1-alicloud-openshift-machine-api-

2022/02/11 16:18:29 Created access keys for RAM User: user1-alicloud-openshift-
machine-api-alibabacloud-credentials
2022/02/11 16:18:29 Saved credentials configuration to: user1-
alicloud/manifests/openshift-machine-api-alibabacloud-credentials-credentials.yaml
...
NOTE
A RAM user can have up to two AccessKeys at the same time. If you run
ccoctl alibabacloud create-ram-users more than twice, the previously
generated manifests secret becomes stale and you must reapply the newly
generated secrets.
b. Verify that the OpenShift Container Platform secrets are created:
$ ls <path_to_ccoctl_output_dir>/manifests
Example output
openshift-cluster-csi-drivers-alibaba-disk-credentials-credentials.yaml
openshift-image-registry-installer-cloud-credentials-credentials.yaml
openshift-ingress-operator-cloud-credentials-credentials.yaml
openshift-machine-api-alibabacloud-credentials-credentials.yaml
You can verify that the RAM users and policies are created by querying Alibaba Cloud. For
more information, refer to Alibaba Cloud documentation on listing RAM users and policies.
4. Copy the generated credential files to the target manifests directory:
$ cp ./<path_to_ccoctl_output_dir>/manifests/*credentials.yaml
./<path_to_installation>dir>/manifests/
where:
<path_to_ccoctl_output_dir>
Specifies the directory created by the ccoctl alibabacloud create-ram-users command.
<path_to_installation_dir>
5.3.8. Deploying the cluster

You can install OpenShift Container Platform on a compatible cloud platform.
IMPORTANT
You can run the create cluster command of the installation program only once, during
initial installation.
Prerequisites
181
You have configured an account with the cloud platform that hosts your cluster.
cluster.
You have verified that the cloud provider account on your host has the correct permissions to
deploy the cluster. An account with incorrect permissions causes the installation process to fail
with an error message that displays the missing permissions.
Procedure
Change to the directory that contains the installation program and initialize the cluster
deployment:
$ ./openshift-install create cluster --dir <installation_directory> \ 1

--log-level=info 2
1 For <installation_directory>, specify the location of your customized ./install-

config.yaml file.
2 To view different installation details, specify warn, debug, or error instead of info.
Verification
When the cluster deployment completes successfully:
The terminal displays directions for accessing your cluster, including a link to the web console
and credentials for the kubeadmin user.
Credential information also outputs to <installation_directory>/.openshift_install.log.
IMPORTANT
Do not delete the installation program or the files that the installation program creates.
Both are required to delete the cluster.
Example output
...
INFO Install complete!
INFO To access the cluster as the system:admin user when using 'oc', run 'export
KUBECONFIG=/home/myuser/install_dir/auth/kubeconfig'
INFO Access the OpenShift web-console here: https://console-openshift-
console.apps.mycluster.example.com
INFO Login to the console with user: "kubeadmin", and password: "password"
INFO Time elapsed: 36m22s
IMPORTANT
182
IMPORTANT
It is recommended that you use Ignition config files within 12 hours after they are
generated because the 24-hour certificate rotates from 16 to 22 hours after the
cluster is installed. By using the Ignition config files within 12 hours, you can avoid
5.3.9. Installing the OpenShift CLI by downloading the binary

IMPORTANT

Procedure
Portal.
$ tar xvf <file>

$ echo $PATH
Verification
$ oc <command>
183

Procedure
Portal.

C:\> path
Verification
C:\> oc <command>

Procedure
Portal.
NOTE

$ echo $PATH
Verification
184
$ oc <command>
5.3.10. Logging in to the cluster by using the CLI

You can log in to your cluster as a default system user by exporting the cluster kubeconfig file. The
kubeconfig file contains information about the cluster that is used by the CLI to connect a client to the
correct cluster and API server. The file is specific to a cluster and is created during OpenShift Container
Platform installation.
Prerequisites
You deployed an OpenShift Container Platform cluster.
You installed the oc CLI.
Procedure
1. Export the kubeadmin credentials:
$ export KUBECONFIG=<installation_directory>/auth/kubeconfig 1
1 For <installation_directory>, specify the path to the directory that you stored the
installation files in.
2. Verify you can run oc commands successfully using the exported configuration:
$ oc whoami
Example output
system:admin
5.3.11. Logging in to the cluster by using the web console

The kubeadmin user exists by default after an OpenShift Container Platform installation. You can log in
to your cluster as the kubeadmin user by using the OpenShift Container Platform web console.
Prerequisites
You have access to the installation host.
You completed a cluster installation and all cluster Operators are available.
Procedure
1. Obtain the password for the kubeadmin user from the kubeadmin-password file on the
installation host:
$ cat <installation_directory>/auth/kubeadmin-password
NOTE
185
NOTE
Alternatively, you can obtain the kubeadmin password from the

<installation_directory>/.openshift_install.log log file on the installation host.
2. List the OpenShift Container Platform web console route:
$ oc get routes -n openshift-console | grep 'console-openshift'
NOTE
Alternatively, you can obtain the OpenShift Container Platform route from the
Example output
console console-openshift-console.apps.<cluster_name>.<base_domain> console

https reencrypt/Redirect None
3. Navigate to the route detailed in the output of the preceding command in a web browser and
log in as the kubeadmin user.
5.3.12. Telemetry access for OpenShift Container Platform

In OpenShift Container Platform 4.15, the Telemetry service, which runs by default to provide metrics
about cluster health and the success of updates, requires internet access. If your cluster is connected to
the internet, Telemetry runs automatically, and your cluster is registered to OpenShift Cluster Manager.
After you confirm that your OpenShift Cluster Manager inventory is correct, either maintained
automatically by Telemetry or manually by using OpenShift Cluster Manager, use subscription watch to
track your OpenShift Container Platform subscriptions at the account or multi-cluster level.
See Accessing the web console for more details about accessing and understanding the
OpenShift Container Platform web console.
See About remote health monitoring for more information about the Telemetry service
5.3.13. Next steps

Validating an installation.
Customize your cluster.
If necessary, you can opt out of remote health reporting .
5.4. INSTALLING A CLUSTER ON ALIBABA CLOUD WITH

CUSTOMIZATIONS
In OpenShift Container Platform version 4.15, you can install a customized cluster on infrastructure that
186
the installation program provisions on Alibaba Cloud. To customize the installation, you modify
parameters in the install-config.yaml file before you install the cluster.
NOTE
The scope of the OpenShift Container Platform installation configurations is intentionally

narrow. It is designed for simplicity and ensured success. You can complete many more
OpenShift Container Platform configuration tasks after an installation completes.
IMPORTANT

processes.
users.
credentials.

IMPORTANT
187
IMPORTANT

authentication.
user.
IMPORTANT
NOTE
Procedure
NOTE
ecdsa algorithm.
188
NOTE

task:
Example output
Agent pid 31874
NOTE

Example output
Next steps
program.

for installation.
189
Prerequisites
Procedure
IMPORTANT
IMPORTANT
components.
5.4.4.1. Creating the installation configuration file
Prerequisites
cluster.
Procedure
command:
190
NOTE

apiVersion: v1
compute:
...
3. Modify the install-config.yaml file. You can find more information about the available
191
parameters in the "Installation configuration parameters" section.
IMPORTANT

Installation configuration parameters for Alibaba Cloud
5.4.4.2. Generating the required installation manifests
the machines.
Procedure
where:
5.4.4.3. Creating credentials for OpenShift Container Platform components with the ccoctl
tool
NOTE
Prerequisites
You must have:
cluster.

192
Procedure

--included \ 1
NOTE
command:

--name <name> \ 1
objects.
placed.
NOTE
193
NOTE
preview parameter.
Example output

...
NOTE
generated secrets.
Example output
where:
194
5.4.4.4. Sample customized install-config.yaml file for Alibaba Cloud
You can customize the installation configuration file (install-config.yaml) to specify more details about
your cluster’s platform or modify the values of the required parameters.
apiVersion: v1
baseDomain: alicloud-dev.devcluster.openshift.com
credentialsMode: Manual
compute:
name: worker
platform: {}
replicas: 3
controlPlane:
architecture: amd64
name: master
platform: {}
replicas: 3
metadata:
creationTimestamp: null
name: test-cluster 1
networking:
clusterNetwork:
- cidr: 10.128.0.0/14
hostPrefix: 23
machineNetwork:
- cidr: 10.0.0.0/16
networkType: OVNKubernetes 2
serviceNetwork:
- 172.30.0.0/16
platform:
alibabacloud:
defaultMachinePlatform: 3
instanceType: ecs.g6.xlarge
systemDiskCategory: cloud_efficiency
systemDiskSize: 200
region: ap-southeast-1 4
resourceGroupID: rg-acfnw6j3hyai 5
vpcID: vpc-0xifdjerdibmaqvtjob2b 6
vswitchIDs: 7
- vsw-0xi8ycgwc8wv5rhviwdq5
- vsw-0xiy6v3z2tedv009b4pz2
publish: External
pullSecret: '{"auths": {"cloud.openshift.com": {"auth": ... }' 8
sshKey: |
ssh-rsa AAAA... 9
1 Required. The installation program prompts you for a cluster name.
195
2 The cluster network plugin to install. The default value OVNKubernetes is the only supported
value.
3 Optional. Specify parameters for machine pools that do not define their own platform
configuration.
4 Required. The installation program prompts you for the region to deploy the cluster to.
5 Optional. Specify an existing resource group where the cluster should be installed.
8 Required. The installation program prompts you for the pull secret.
9 Optional. The installation program prompts you for the SSH key value that you use to access the
machines in your cluster.
6 7 Optional. These are example vswitchID values.
5.4.4.5. Configuring the cluster-wide proxy during installation
Production environments can deny direct access to the internet and instead have an HTTP or HTTPS
proxy available. You can configure a new OpenShift Container Platform cluster to use a proxy by
configuring the proxy settings in the install-config.yaml file.
Prerequisites
You have an existing install-config.yaml file.
You reviewed the sites that your cluster requires access to and determined whether any of
them need to bypass the proxy. By default, all cluster egress traffic is proxied, including calls to
hosting cloud provider APIs. You added sites to the Proxy object’s spec.noProxy field to
bypass the proxy if necessary.
NOTE
The Proxy object status.noProxy field is populated with the values of the
networking.machineNetwork[].cidr, networking.clusterNetwork[].cidr, and
networking.serviceNetwork[] fields from your installation configuration.
For installations on Amazon Web Services (AWS), Google Cloud Platform (GCP),
Microsoft Azure, and Red Hat OpenStack Platform (RHOSP), the Proxy object
status.noProxy field is also populated with the instance metadata endpoint
(169.254.169.254).
Procedure
1. Edit your install-config.yaml file and add the proxy settings. For example:
apiVersion: v1
baseDomain: my.domain.com
proxy:
httpProxy: http://<username>:<pswd>@<ip>:<port> 1
httpsProxy: https://<username>:<pswd>@<ip>:<port> 2
noProxy: example.com 3
additionalTrustBundle: | 4
196
-----BEGIN CERTIFICATE-----
<MY_TRUSTED_CA_CERT>
-----END CERTIFICATE-----
additionalTrustBundlePolicy: <policy_to_add_additionalTrustBundle> 5
1 A proxy URL to use for creating HTTP connections outside the cluster. The URL scheme
must be http.
2 A proxy URL to use for creating HTTPS connections outside the cluster.
3 A comma-separated list of destination domain names, IP addresses, or other network

CIDRs to exclude from proxying. Preface a domain with . to match subdomains only. For
example, .y.com matches x.y.com, but not y.com. Use * to bypass the proxy for all
destinations.
4 If provided, the installation program generates a config map that is named user-ca-bundle
in the openshift-config namespace that contains one or more additional CA certificates
that are required for proxying HTTPS connections. The Cluster Network Operator then
creates a trusted-ca-bundle config map that merges these contents with the Red Hat
Enterprise Linux CoreOS (RHCOS) trust bundle, and this config map is referenced in the
trustedCA field of the Proxy object. The additionalTrustBundle field is required unless
the proxy’s identity certificate is signed by an authority from the RHCOS trust bundle.
5 Optional: The policy to determine the configuration of the Proxy object to reference the
user-ca-bundle config map in the trustedCA field. The allowed values are Proxyonly and
Always. Use Proxyonly to reference the user-ca-bundle config map only when
http/https proxy is configured. Use Always to always reference the user-ca-bundle
config map. The default value is Proxyonly.
NOTE
The installation program does not support the proxy readinessEndpoints field.
NOTE
If the installer times out, restart and then complete the deployment by using the
wait-for command of the installer. For example:
$ ./openshift-install wait-for install-complete --log-level debug
2. Save the file and reference it when installing OpenShift Container Platform.
The installation program creates a cluster-wide proxy that is named cluster that uses the proxy settings
in the provided install-config.yaml file. If no proxy settings are provided, a cluster Proxy object is still
created, but it will have a nil spec.
NOTE
Only the Proxy object named cluster is supported, and no additional proxies can be
created.
197
IMPORTANT
Prerequisites
cluster.
Procedure
deployment:

--log-level=info 2

config.yaml file.
Verification
IMPORTANT
Example output
...
198
IMPORTANT

IMPORTANT

Procedure
Portal.
$ tar xvf <file>

$ echo $PATH
199
Verification
$ oc <command>

Procedure
Portal.

C:\> path
Verification
C:\> oc <command>

Procedure
Portal.
NOTE

$ echo $PATH
200
Verification
$ oc <command>

Prerequisites
Procedure
$ oc whoami
Example output
system:admin

Prerequisites
Procedure
installation host:
201
NOTE

NOTE
Example output


See About remote health monitoring for more information about the Telemetry service.
OpenShift Container Platform web console
5.4.10. Next steps

202
5.5. INSTALLING A CLUSTER ON ALIBABA CLOUD WITH NETWORK

CUSTOMIZATIONS
In OpenShift Container Platform 4.15, you can install a cluster on Alibaba Cloud with customized network
configuration options. By customizing your network configuration, your cluster can coexist with existing
IP address allocations in your environment and integrate with existing MTU and VXLAN configurations.
You must set most of the network configuration parameters during installation, and you can modify only
kubeProxy configuration parameters in a running cluster.
IMPORTANT

processes.
users.
credentials.

IMPORTANT
203
IMPORTANT

authentication.
user.
IMPORTANT
NOTE
Procedure
NOTE
ecdsa algorithm.
204
NOTE

task:
Example output
Agent pid 31874
NOTE

Example output
Next steps
program.

for installation.
205
Prerequisites
Procedure
IMPORTANT
IMPORTANT
components.
5.5.5. Network configuration phases

There are two phases prior to OpenShift Container Platform installation where you can customize the
network configuration.
Phase 1
You can customize the following network-related fields in the install-config.yaml file before you
create the manifest files:
networking.networkType
networking.clusterNetwork
networking.serviceNetwork
206
networking.machineNetwork
For more information on these fields, refer to Installation configuration parameters.
NOTE
Set the networking.machineNetwork to match the CIDR that the preferred

NIC resides in.
IMPORTANT
The CIDR range 172.17.0.0/16 is reserved by libVirt. You cannot use this
range or any range that overlaps with this range for any networks in your
cluster.
Phase 2
After creating the manifest files by running openshift-install create manifests, you can define a
customized Cluster Network Operator manifest with only the fields you want to modify. You can use
the manifest to specify advanced network configuration.
You cannot override the values specified in phase 1 in the install-config.yaml file during phase 2.
However, you can further customize the network plugin during phase 2.
You can customize the OpenShift Container Platform cluster you install on
Prerequisites
cluster.
Procedure
command:
207
NOTE

ii. Enter a descriptive name for your cluster.
IMPORTANT

the machines.
Procedure
where:
NOTE
208
Prerequisites
You must have:
Procedure

--included \ 1
NOTE
apiVersion: v1
compute:
name: worker
platform: {}
replicas: 3
controlPlane:
architecture: amd64
name: master
209
platform: {}
replicas: 3
metadata:
networking:
clusterNetwork:
- cidr: 10.128.0.0/14
hostPrefix: 23
machineNetwork:
- cidr: 10.0.0.0/16
serviceNetwork:
- 172.30.0.0/16
platform:
alibabacloud:
systemDiskSize: 200
vswitchIDs: 7
publish: External
sshKey: |
ssh-rsa AAAA... 9
value.
configuration.
210
Prerequisites
NOTE
(169.254.169.254).
Procedure
apiVersion: v1
proxy:
must be http.

destinations.
211
NOTE
NOTE
NOTE
created.
5.5.6. Cluster Network Operator configuration

The configuration for the cluster network is specified as part of the Cluster Network Operator (CNO)
configuration and stored in a custom resource (CR) object that is named cluster. The CR specifies the
fields for the Network API in the operator.openshift.io API group.
The CNO configuration inherits the following fields during cluster installation from the Network API in
the Network.config.openshift.io API group:
clusterNetwork
IP address pools from which pod IP addresses are allocated.
serviceNetwork
IP address pool for services.
defaultNetwork.type
Cluster network plugin. OVNKubernetes is the only supported plugin during installation.
You can specify the cluster network plugin configuration for your cluster by setting the fields for the
defaultNetwork object in the CNO object named cluster.
5.5.6.1. Cluster Network Operator configuration object
212
The fields for the Cluster Network Operator (CNO) are described in the following table:
Table 5.1. Cluster Network Operator configuration object
Field Type Description
metadata.name string The name of the CNO object. This name is always cluster.
spec.clusterNet array A list specifying the blocks of IP addresses from which pod IP
work addresses are allocated and the subnet prefix length assigned to
each individual node in the cluster. For example:
spec:
clusterNetwork:
- cidr: 10.128.0.0/19
hostPrefix: 23
- cidr: 10.128.32.0/19
hostPrefix: 23
spec.serviceNet array A block of IP addresses for services. The OpenShift SDN and
work OVN-Kubernetes network plugins support only a single IP
address block for the service network. For example:
spec:
serviceNetwork:
- 172.30.0.0/14
You can customize this field only in the install-config.yaml file

before you create the manifests. The value is read-only in the
manifest file.
spec.defaultNet object Configures the network plugin for the cluster network.
work
spec.kubeProxy object The fields for this object specify the kube-proxy configuration. If
Config you are using the OVN-Kubernetes cluster network plugin, the
kube-proxy configuration has no effect.
defaultNetwork object configuration

The values for the defaultNetwork object are defined in the following table:
Table 5.2. defaultNetwork object
213
type string OVNKubernetes. The Red Hat OpenShift

Networking network plugin is selected during
installation. This value cannot be changed after
cluster installation.
NOTE
OpenShift Container Platform uses

the OVN-Kubernetes network plugin
by default. OpenShift SDN is no
longer available as an installation
choice for new clusters.
ovnKubernetesConfig object This object is only valid for the OVN-Kubernetes

network plugin.
Configuration for the OVN-Kubernetes network plugin

The following table describes the configuration fields for the OVN-Kubernetes network plugin:
Table 5.3. ovnKubernetesConfig object
mtu integer The maximum transmission unit (MTU) for the Geneve (Generic
Network Virtualization Encapsulation) overlay network. This is
detected automatically based on the MTU of the primary
network interface. You do not normally need to override the
detected MTU.
If the auto-detected value is not what you expect it to be,

confirm that the MTU on the primary network interface on your
nodes is correct. You cannot use this option to change the MTU
value of the primary network interface on the nodes.
If your cluster requires different MTU values for different nodes,

you must set this value to 100 less than the lowest MTU value in
your cluster. For example, if some nodes in your cluster have an
MTU of 9001, and some have an MTU of1500, you must set this
value to 1400.
genevePort integer The port to use for all Geneve packets. The default value is
6081. This value cannot be changed after cluster installation.
ipsecConfig object Specify a configuration object for customizing the IPsec

configuration.
policyAuditConf object Specify a configuration object for customizing network policy

ig audit logging. If unset, the defaults audit log settings are used.
214
gatewayConfig object Optional: Specify a configuration object for customizing how

egress traffic is sent to the node gateway.
NOTE
While migrating egress traffic, you can expect

some disruption to workloads and service traffic
until the Cluster Network Operator (CNO)
successfully rolls out the changes.
215
v4InternalSubne If your existing The default value is 100.64.0.0/16.

t network
infrastructure
overlaps with the
100.64.0.0/16
IPv4 subnet, you
can specify a
different IP
address range for
internal use by
OVN-Kubernetes.
You must ensure
that the IP address
range does not
overlap with any
other subnet used
by your OpenShift
Container
Platform
installation. The IP
address range
must be larger
than the maximum
number of nodes
that can be added
to the cluster. For
example, if the
clusterNetwork.
cidr value is
10.128.0.0/14
and the
clusterNetwork.
hostPrefix value
is /23, then the
maximum number
of nodes is 2^(23-
14)=512 .
This field cannot

be changed after
installation.
216
v6InternalSubne If your existing The default value is fd98::/48.

t network
infrastructure
overlaps with the
fd98::/48 IPv6
subnet, you can
specify a different
IP address range
for internal use by
OVN-Kubernetes.
You must ensure
that the IP address
range does not
overlap with any
other subnet used
by your OpenShift
Container
Platform
address range
must be larger
than the maximum
number of nodes
that can be added
to the cluster.
This field cannot

be changed after
installation.
Table 5.4. policyAuditConfig object
rateLimit integer The maximum number of messages to generate every second

per node. The default value is 20 messages per second.
maxFileSize integer The maximum size for the audit log in bytes. The default value is
50000000 or 50 MB.
maxLogFiles integer The maximum number of log files that are retained.
217
destination string One of the following additional audit log targets:
libc
The libc syslog() function of the journald process on the
host.
udp:<host>:<port>
A syslog server. Replace <host>:<port> with the host and
port of the syslog server.
unix:<file>
A Unix Domain Socket file specified by <file> .
null
Do not send the audit logs to any additional target.
syslogFacility string The syslog facility, such as kern, as defined by RFC5424. The
default value is local0.
Table 5.5. gatewayConfig object
routingViaHost boolean Set this field to true to send egress traffic from pods to the
host networking stack. For highly-specialized installations and
applications that rely on manually configured routes in the
kernel routing table, you might want to route egress traffic to
the host networking stack. By default, egress traffic is processed
in OVN to exit the cluster and is not affected by specialized
routes in the kernel routing table. The default value is false.
This field has an interaction with the Open vSwitch hardware

offloading feature. If you set this field to true, you do not
receive the performance benefits of the offloading because
egress traffic is processed by the host networking stack.
ipForwarding object You can control IP forwarding for all traffic on OVN-Kubernetes
managed interfaces by using the ipForwarding specification in
the Network resource. Specify Restricted to only allow IP
forwarding for Kubernetes related traffic. Specify Global to
allow forwarding of all IP traffic. For new installations, the default
is Restricted . For updates to OpenShift Container Platform
4.14 or later, the default is Global.
Table 5.6. ipsecConfig object
218
mode string Specifies the behavior of the IPsec implementation. Must be

one of the following values:
Disabled: IPsec is not enabled on cluster nodes.
External: IPsec is enabled for network traffic with

external hosts.
Full : IPsec is enabled for pod traffic and network

traffic with external hosts.
Example OVN-Kubernetes configuration with IPSec enabled
defaultNetwork:
type: OVNKubernetes
ovnKubernetesConfig:
mtu: 1400
genevePort: 6081
ipsecConfig:
mode: Full
IMPORTANT
Using OVNKubernetes can lead to a stack exhaustion problem on IBM Power®.
kubeProxyConfig object configuration (OpenShiftSDN container network interface only)

The values for the kubeProxyConfig object are defined in the following table:
Table 5.7. kubeProxyConfig object
iptablesSyncPeriod string The refresh period for iptables rules. The default
value is 30s. Valid suffixes include s, m, and h and
are described in the Go time package
documentation.
NOTE
Because of performance
improvements introduced in
OpenShift Container Platform 4.3
and greater, adjusting the
iptablesSyncPeriod parameter is
no longer necessary.
219
proxyArguments.iptables- array The minimum duration before refreshing iptables

min-sync-period rules. This field ensures that the refresh does not
happen too frequently. Valid suffixes include s, m,
and h and are described in the Go time package.
The default value is:
kubeProxyConfig:
proxyArguments:
iptables-min-sync-period:
- 0s
5.5.7. Specifying advanced network configuration

You can use advanced network configuration for your network plugin to integrate your cluster into your
existing network environment. You can specify advanced network configuration only before you install
the cluster.
IMPORTANT
Customizing your network configuration by modifying the OpenShift Container Platform

manifest files created by the installation program is not supported. Applying a manifest
file that you create, as in the following procedure, is supported.
Prerequisites
You have created the install-config.yaml file and completed any modifications to it.
Procedure
1. Change to the directory that contains the installation program and create the manifests:
$ ./openshift-install create manifests --dir <installation_directory> 1
1 <installation_directory> specifies the name of the directory that contains the install-
config.yaml file for your cluster.
2. Create a stub manifest file for the advanced network configuration that is named cluster-
network-03-config.yml in the <installation_directory>/manifests/ directory:
apiVersion: operator.openshift.io/v1
kind: Network
metadata:
name: cluster
spec:
3. Specify the advanced network configuration for your cluster in the cluster-network-03-
config.yml file, such as in the following example:
Enable IPsec for the OVN-Kubernetes network provider
220
kind: Network
metadata:
name: cluster
spec:
defaultNetwork:
ipsecConfig:
mode: Full
4. Optional: Back up the manifests/cluster-network-03-config.yml file. The installation program

consumes the manifests/ directory when you create the Ignition config files.
5.5.8. Configuring hybrid networking with OVN-Kubernetes

You can configure your cluster to use hybrid networking with the OVN-Kubernetes network plugin. This
allows a hybrid cluster that supports different node networking configurations.
NOTE
This configuration is necessary to run both Linux and Windows nodes in the same cluster.
Prerequisites
You defined OVNKubernetes for the networking.networkType parameter in the install-

config.yaml file. See the installation documentation for configuring OpenShift Container
Platform network customizations on your chosen cloud provider for more information.
Procedure
$ ./openshift-install create manifests --dir <installation_directory>
where:
Specifies the name of the directory that contains the install-config.yaml file for your
cluster.
$ cat <<EOF > <installation_directory>/manifests/cluster-network-03-config.yml

kind: Network
metadata:
name: cluster
spec:
EOF
where:
221
Specifies the directory name that contains the manifests/ directory for your cluster.
3. Open the cluster-network-03-config.yml file in an editor and configure OVN-Kubernetes with

hybrid networking, such as in the following example:
Specify a hybrid networking configuration
kind: Network
metadata:
name: cluster
spec:
defaultNetwork:
hybridOverlayConfig:
hybridClusterNetwork: 1
- cidr: 10.132.0.0/14
hostPrefix: 23
hybridOverlayVXLANPort: 9898 2
1 Specify the CIDR configuration used for nodes on the additional overlay network. The
hybridClusterNetwork CIDR cannot overlap with the clusterNetwork CIDR.
2 Specify a custom VXLAN port for the additional overlay network. This is required for
running Windows nodes in a cluster installed on vSphere, and must not be configured for
any other cloud provider. The custom port can be any open port excluding the default 4789
port. For more information on this requirement, see the Microsoft documentation on Pod-
to-pod connectivity between hosts is broken.
NOTE
Windows Server Long-Term Servicing Channel (LTSC): Windows Server 2019 is

not supported on clusters with a custom hybridOverlayVXLANPort value
because this Windows server version does not support selecting a custom VXLAN
port.
4. Save the cluster-network-03-config.yml file and quit the text editor.

deletes the manifests/ directory when creating the cluster.

IMPORTANT
Prerequisites
222
cluster.
Procedure
deployment:

--log-level=info 2

config.yaml file.
Verification
IMPORTANT
Example output
...
IMPORTANT
223
IMPORTANT

IMPORTANT

Procedure
Portal.
$ tar xvf <file>

$ echo $PATH
Verification
$ oc <command>
224

Procedure
Portal.

C:\> path
Verification
C:\> oc <command>

Procedure
Portal.
NOTE

$ echo $PATH
Verification
225
$ oc <command>

Prerequisites
Procedure
$ oc whoami
Example output
system:admin

Prerequisites
Procedure
installation host:
NOTE
226
NOTE

NOTE
Example output


5.5.14. Next steps

Validate an installation.
5.6. INSTALLING A CLUSTER ON ALIBABA CLOUD INTO AN EXISTING

VPC
227
In OpenShift Container Platform version 4.15, you can install a cluster into an existing Alibaba Virtual
Private Cloud (VPC) on Alibaba Cloud Services. The installation program provisions the required
infrastructure, which can then be customized. To customize the VPC installation, modify the parameters
in the 'install-config.yaml' file before you install the cluster.
NOTE

IMPORTANT

processes.
users.
credentials.
5.6.2. Using a custom VPC

In OpenShift Container Platform 4.15, you can deploy a cluster into existing subnets in an existing Virtual
Private Cloud (VPC) in the Alibaba Cloud Platform. By deploying OpenShift Container Platform into an
existing Alibaba VPC, you can avoid limit constraints in new accounts and more easily adhere to your
organization’s operational constraints. If you cannot obtain the infrastructure creation permissions that
are required to create the VPC yourself, use this installation option. You must configure networking
using vSwitches.
5.6.2.1. Requirements for using your VPC
The union of the VPC CIDR block and the machine network CIDR must be non-empty. The vSwitches
must be within the machine network.
The installation program does not create the following components:
228
VPC
vSwitches
Route table
NAT gateway
NOTE
The installation program requires that you use the cloud-provided DNS server. Using a
custom DNS server is not supported and causes the installation to fail.
5.6.2.2. VPC validation
To ensure that the vSwitches you provide are suitable, the installation program confirms the following
data:
All the vSwitches that you specify must exist.
You have provided one or more vSwitches for control plane machines and compute machines.
The vSwitches' CIDRs belong to the machine CIDR that you specified.
5.6.2.3. Division of permissions
Some individuals can create different resources in your cloud than others. For example, you might be
able to create application-specific items, like instances, buckets, and load balancers, but not
networking-related components, such as VPCs or vSwitches.
5.6.2.4. Isolation between clusters
If you deploy OpenShift Container Platform into an existing network, the isolation of cluster services is
reduced in the following ways:
You can install multiple OpenShift Container Platform clusters in the same VPC.
ICMP ingress is allowed to the entire network.
TCP 22 ingress (SSH) is allowed to the entire network.
Control plane TCP 6443 ingress (Kubernetes API) is allowed to the entire network.
Control plane TCP 22623 ingress (MCS) is allowed to the entire network.

229
IMPORTANT

authentication.
user.
IMPORTANT
NOTE
Procedure
NOTE
230
NOTE
ecdsa algorithm.
NOTE

task:
Example output
Agent pid 31874
NOTE

Example output
Next steps
231
program.

for installation.
Prerequisites
Procedure
IMPORTANT
IMPORTANT
components.
Prerequisites
232
cluster.
Procedure
command:
NOTE

apiVersion: v1
233
compute:
...
IMPORTANT

apiVersion: v1
compute:
name: worker
platform: {}
replicas: 3
controlPlane:
architecture: amd64
name: master
platform: {}
replicas: 3
metadata:
networking:
clusterNetwork:
- cidr: 10.128.0.0/14
hostPrefix: 23
machineNetwork:
- cidr: 10.0.0.0/16
serviceNetwork:
- 172.30.0.0/16
platform:
234
alibabacloud:
systemDiskSize: 200
vswitchIDs: 7
publish: External
sshKey: |
ssh-rsa AAAA... 9
value.
configuration.
the machines.
Procedure
where:
5.6.5.4. Configuring the Cloud Credential Operator utility
235
To create and manage cloud credentials from outside of the cluster when the Cloud Credential
Operator (CCO) is operating in manual mode, extract and prepare the CCO utility (ccoctl) binary.
NOTE
Prerequisites
Procedure

NOTE

$ chmod 775 ccoctl
Verification
$ ccoctl --help
Usage:
ccoctl [command]
236
Available Commands:
Flags:
5.6.5.5. Creating credentials for OpenShift Container Platform components with the ccoctl
tool
NOTE
Prerequisites
You must have:
cluster.

Procedure

--included \ 1
237
NOTE
command:

--name <name> \ 1
objects.
placed.
NOTE
preview parameter.
Example output

238

...
NOTE
generated secrets.
Example output
where:

IMPORTANT
Prerequisites
cluster.
239
Procedure
deployment:

--log-level=info 2

config.yaml file.
Verification
IMPORTANT
Example output
...
IMPORTANT
240
IMPORTANT

IMPORTANT

Procedure
Portal.
$ tar xvf <file>

$ echo $PATH
Verification
$ oc <command>
241

Procedure
Portal.

C:\> path
Verification
C:\> oc <command>

Procedure
Portal.
NOTE

$ echo $PATH
Verification
242
$ oc <command>

Prerequisites
Procedure
$ oc whoami
Example output
system:admin

Prerequisites
Procedure
installation host:
NOTE
243
NOTE

NOTE
Example output


5.6.11. Next steps

5.7. INSTALLATION CONFIGURATION PARAMETERS FOR ALIBABA

CLOUD
Before you deploy an OpenShift Container Platform cluster on Alibaba Cloud, you provide parameters
to customize your cluster and the platform that hosts it. When you create the install-config.yaml file,
244
you provide values for the required parameters through the command line. You can then modify the
install-config.yaml file to customize your cluster further.
5.7.1. Available installation configuration parameters for Alibaba Cloud

The following tables specify the required, optional, and Alibaba Cloud-specific installation configuration
parameters that you can set as part of the installation process.
NOTE
After installation, you cannot modify these parameters in the install-config.yaml file.
5.7.1.1. Required configuration parameters
Required installation configuration parameters are described in the following table:
Table 5.8. Required parameters
The API version for the String

apiVersion: install-config.yaml
content. The current version is
v1. The installation program
may also support older API
versions.
The base domain of your A fully-qualified domain or subdomain name, such as

baseDomain: cloud provider. The base example.com .
domain is used to create
routes to your OpenShift
Container Platform cluster
components. The full DNS
name for your cluster is a
combination of the
baseDomain and
metadata.name parameter
values that uses the
<metadata.name>.
<baseDomain> format.
Kubernetes resource Object

metadata: ObjectMeta, from which only
the name parameter is
consumed.
The name of the cluster. DNS String of lowercase letters, hyphens (- ), and periods
metadata: records for the cluster are all (.), such as dev.
name: subdomains of
{{.metadata.name}}.
{{.baseDomain}}.
245
The configuration for the Object

platform: specific platform upon which
to perform the installation:
alibabacloud, aws,
baremetal, azure , gcp ,
ibmcloud, nutanix,
openstack, powervs ,
vsphere, or {} . For additional
information about platform.
<platform> parameters,
consult the table for your
specific platform that follows.
Get a pull secret from Red

pullSecret: Hat OpenShift Cluster {
Manager to authenticate "auths":{
downloading container "cloud.openshift.com":{
images for OpenShift "auth":"b3Blb=",
Container Platform "email":"you@example.com"
components from services },
such as Quay.io. "quay.io":{
"auth":"b3Blb=",
"email":"you@example.com"
}
}
}
5.7.1.2. Network configuration parameters
You can customize your installation configuration based on the requirements of your existing network
infrastructure. For example, you can expand the IP address block for the cluster network or provide
different IP address blocks than the defaults.
Only IPv4 addresses are supported.
NOTE
Globalnet is not supported with Red Hat OpenShift Data Foundation disaster recovery
solutions. For regional disaster recovery scenarios, ensure that you use a nonoverlapping
range of private IP addresses for the cluster and service networks in each cluster.
Table 5.9. Network parameters
246
The configuration for the cluster Object

networking: network.
NOTE
You cannot modify

parameters specified
by the networking
object after
installation.
The Red Hat OpenShift Networking OVNKubernetes. OVNKubernetes

networking: network plugin to install. is a CNI plugin for Linux networks and
networkType: hybrid networks that contain both
Linux and Windows servers. The
default value is OVNKubernetes.
The IP address blocks for pods. An array of objects. For example:

networking:
clusterNetwork: The default value is 10.128.0.0/14 networking:
with a host prefix of /23. clusterNetwork:
- cidr: 10.128.0.0/14
If you specify multiple IP address
hostPrefix: 23
blocks, the blocks must not overlap.
Required if you use An IP address block in Classless Inter-

networking: networking.clusterNetwork. An IP Domain Routing (CIDR) notation. The
clusterNetwork: address block. prefix length for an IPv4 block is
cidr: between 0 and 32.
An IPv4 network.
The subnet prefix length to assign to A subnet prefix.

networking: each individual node. For example, if
clusterNetwork: hostPrefix is set to 23 then each The default value is 23.
hostPrefix: node is assigned a /23 subnet out of
the given cidr. A hostPrefix value of
23 provides 510 (2^(32 - 23) - 2) pod
IP addresses.
The IP address block for services. The An array with an IP address block in
networking: default value is 172.30.0.0/16. CIDR format. For example:
serviceNetwork:
The OVN-Kubernetes network plugins
networking:
supports only a single IP address block
serviceNetwork:
for the service network.
- 172.30.0.0/16
247
The IP address blocks for machines. An array of objects. For example:

networking:
machineNetwork: If you specify multiple IP address networking:
blocks, the blocks must not overlap. machineNetwork:
- cidr: 10.0.0.0/16
Required if you use An IP network block in CIDR notation.

networking: networking.machineNetwork . An
machineNetwork: IP address block. The default value is For example, 10.0.0.0/16.
cidr: 10.0.0.0/16 for all platforms other
than libvirt and IBM Power® Virtual
NOTE
Server. For libvirt, the default value is
192.168.126.0/24 . For IBM Power® Set the
Virtual Server, the default value is networking.machin
192.168.0.0/24. eNetwork to match
the CIDR that the
preferred NIC resides
in.
5.7.1.3. Optional configuration parameters
Optional installation configuration parameters are described in the following table:
Table 5.10. Optional parameters
A PEM-encoded X.509 certificate String

additionalTrustBun bundle that is added to the nodes'
dle: trusted certificate store. This trust
bundle may also be used when a proxy
has been configured.
Controls the installation of optional String array

capabilities: core cluster components. You can
reduce the footprint of your OpenShift
Container Platform cluster by disabling
optional components. For more
information, see the "Cluster
capabilities" page in Installing.
248
Selects an initial set of optional String

capabilities: capabilities to enable. Valid values are
None, v4.11, v4.12 and vCurrent.
baselineCapabilityS The default value is vCurrent.
et:
Extends the set of optional capabilities String array

capabilities: beyond what you specify in
baselineCapabilitySet . You may
additionalEnabledC specify multiple capabilities in this
apabilities: parameter.
Enables workload partitioning, which None or AllNodes . None is the

cpuPartitioningMod isolates OpenShift Container Platform default value.
e: services, cluster management
workloads, and infrastructure pods to
run on a reserved set of CPUs.
Workload partitioning can only be
enabled during installation and cannot
be disabled after installation. While this
field enables workload partitioning, it
does not configure workloads to use
specific CPUs. For more information,
see the Workload partitioning page in
the Scalability and Performance
section.
The configuration for the machines Array of MachinePool objects.

compute: that comprise the compute nodes.
Determines the instruction set String

compute: architecture of the machines in the
architecture: pool. Currently, clusters with varied
architectures are not supported. All
pools must specify the same
architecture. Valid values are amd64
(the default).
249
Whether to enable or disable Enabled or Disabled

compute: simultaneous multithreading, or
hyperthreading: hyperthreading, on compute
machines. By default, simultaneous
multithreading is enabled to increase
the performance of your machines'
cores.
IMPORTANT
If you disable
simultaneous
multithreading, ensure
that your capacity
planning accounts for
the dramatically
decreased machine
performance.
Required if you use compute. The worker

compute: name of the machine pool.
name:
Required if you use compute. Use this alibabacloud, aws, azure , gcp ,
compute: parameter to specify the cloud ibmcloud, nutanix, openstack,
platform: provider to host the worker machines. powervs , vsphere, or {}
This parameter value must match the
controlPlane.platform parameter
value.
The number of compute machines, A positive integer greater than or equal

compute: which are also known as worker to 2. The default value is 3.
replicas: machines, to provision.
Enables the cluster for a feature set. A String. The name of the feature set to
featureSet: feature set is a collection of OpenShift enable, such as
Container Platform features that are TechPreviewNoUpgrade.
not enabled by default. For more
information about enabling a feature
set during installation, see "Enabling
features using feature gates".

controlPlane: that comprise the control plane.
250

controlPlane: architecture of the machines in the
(the default).

controlPlane: simultaneous multithreading, or
hyperthreading: hyperthreading, on control plane
cores.
IMPORTANT
If you disable
simultaneous
that your capacity
the dramatically
decreased machine
performance.
Required if you use controlPlane . master

controlPlane: The name of the machine pool.
name:
Required if you use controlPlane . alibabacloud, aws, azure , gcp ,

controlPlane: Use this parameter to specify the cloud ibmcloud, nutanix, openstack,
platform: provider that hosts the control plane powervs , vsphere, or {}
machines. This parameter value must
match the compute.platform
parameter value.
The number of control plane machines Supported values are 3, or 1 when

controlPlane: to provision. deploying single-node OpenShift.
replicas:
The Cloud Credential Operator (CCO) Mint , Passthrough, Manual or an

credentialsMode: mode. If no mode is specified, the empty string ( ""). [1]
CCO dynamically tries to determine
the capabilities of the provided
credentials, with a preference for mint
mode on the platforms where multiple
modes are supported.
251
Enable or disable FIPS mode. The false or true

fips: default is false (disabled). If FIPS
mode is enabled, the Red Hat
Enterprise Linux CoreOS (RHCOS)
machines that OpenShift Container
Platform runs on bypass the default
Kubernetes cryptography suite and use
the cryptography modules that are
provided with RHCOS instead.
IMPORTANT
To enable FIPS mode

for your cluster, you
must run the
installation program
from a Red Hat
Enterprise Linux
(RHEL) computer
configured to operate
in FIPS mode. For
more information
about configuring
FIPS mode on RHEL,
see Installing the
system in FIPS mode.
When running Red Hat
Enterprise Linux
(RHEL) or Red Hat
Enterprise Linux
CoreOS (RHCOS)
booted in FIPS mode,
OpenShift Container
Platform core
components use the
RHEL cryptographic
libraries that have
been submitted to
NIST for FIPS 140-
2/140-3 Validation on
only the x86_64,
ppc64le, and s390x
architectures.
NOTE
252
NOTE
If you are using Azure
File storage, you
cannot enable FIPS
mode.
Sources and repositories for the Array of objects. Includes a source

imageContentSour release-image content. and, optionally, mirrors, as described
ces: in the following rows of this table.
Required if you use String

imageContentSour imageContentSources . Specify the
ces: repository that users refer to, for
source: example, in image pull specifications.
Specify one or more repositories that Array of strings

imageContentSour may also contain the same images.
ces:
mirrors:
253
How to publish or expose the user- Internal or External. The default

publish: facing endpoints of your cluster, such value is External.
as the Kubernetes API, OpenShift
routes. Setting this field to Internal is not
supported on non-cloud platforms.
IMPORTANT
If the value of the field

is set to Internal , the
cluster will become
non-functional. For
more information,
refer to BZ#1953035.
The SSH key to authenticate access to For example, sshKey: ssh-ed25519

sshKey: your cluster machines. AAAA...
NOTE
For production
OpenShift Container
Platform clusters on
which you want to
perform installation
debugging or disaster
recovery, specify an
SSH key that your
ssh-agent process
uses.
1. Not all CCO modes are supported for all cloud providers. For more information about CCO
modes, see the "Managing cloud provider credentials" entry in the Authentication and
authorization content.
5.7.1.4. Additional Alibaba Cloud configuration parameters
Additional Alibaba Cloud configuration parameters are described in the following table. The
alibabacloud parameters are the configuration used when installing on Alibaba Cloud. The
defaultMachinePlatform parameters are the default configuration used when installing on Alibaba
Cloud for machine pools that do not define their own platform configuration.
These parameters apply to both compute machines and control plane machines where specified.
NOTE
254
NOTE
If defined, the parameters compute.platform.alibabacloud and

controlPlane.platform.alibabacloud will overwrite
platform.alibabacloud.defaultMachinePlatform settings for compute machines and
control plane machines respectively.
Table 5.11. Optional Alibaba Cloud parameters
The imageID used to create String.

compute: the ECS instance. ImageID
platform: must belong to the same
region as the cluster.
alibabacloud:
imageID:
InstanceType defines the ECS String.

compute: instance type. Example:
platform: ecs.g6.large
alibabacloud:
instanceType:
Defines the category of the String.

compute: system disk. Examples:
platform: cloud_efficiency,cloud_e
ssd
alibabacloud:
systemDiskCa
tegory:
Defines the size of the system Integer.

compute: disk in gibibytes (GiB).
platform:
alibabacloud:
systemDisksiz
e:
The list of availability zones String list.

compute: that can be used. Examples:
platform: cn-hangzhou-h, cn-
hangzhou-j
alibabacloud:
zones:
255
The imageID used to create String.

controlPlane: the ECS instance. ImageID
platform: must belong to the same
alibabacloud:
imageID:
InstanceType defines the ECS String.

controlPlane: instance type. Example:
platform: ecs.g6.xlarge
alibabacloud:
instanceType:
Defines the category of the String.

controlPlane: system disk. Examples:
platform: cloud_efficiency,cloud_e
ssd
alibabacloud:
systemDiskCa
tegory:
Defines the size of the system Integer.

controlPlane: disk in gibibytes (GiB).
platform:
alibabacloud:
systemDisksiz
e:
The list of availability zones String list.

controlPlane: that can be used. Examples:
platform: cn-hangzhou-h, cn-
hangzhou-j
alibabacloud:
zones:
Required. The Alibaba Cloud String.

platform: region where the cluster will
be created.
alibabacloud:
region:
256
The ID of an already existing String.

platform: resource group where the
cluster will be installed. If
alibabacloud: empty, the installation
program will create a new
resourceGrou resource group for the cluster.
pID:
Additional keys and values to Object.

platform: apply to all Alibaba Cloud
resources created for the
alibabacloud: cluster.
tags:
The ID of an already existing String.

platform: VPC where the cluster should
be installed. If empty, the
alibabacloud: installation program will create
vpcID: a new VPC for the cluster.
The ID list of already existing String list.

platform: VSwitches where cluster
resources will be created. The
alibabacloud: existing VSwitches can only
vswitchIDs: be used when also using
existing VPC. If empty, the
installation program will create
new VSwitches for the cluster.
For both compute machines String.

platform: and control plane machines,
the image ID that should be
alibabacloud: used to create ECS instance.
If set, the image ID should
defaultMachin belong to the same region as
ePlatform: the cluster.
imageID:
257
For both compute machines String.

the ECS instance type used to
alibabacloud: create the ECS instance.
Example: ecs.g6.xlarge
defaultMachin
ePlatform:
instanceType:
For both compute machines String, for example "", cloud_efficiency,

platform: and control plane machines, cloud_essd.
the category of the system
alibabacloud: disk. Examples:
cloud_efficiency,
defaultMachin cloud_essd.
ePlatform:
systemDiskCa
tegory:
For both compute machines Integer.

the size of the system disk in
alibabacloud: gibibytes (GiB). The minimum
is 120.
defaultMachin
ePlatform:
systemDiskSiz
e:
For both compute machines String list.

the list of availability zones
alibabacloud: that can be used. Examples:
cn-hangzhou-h, cn-
defaultMachin hangzhou-j
ePlatform:
zones:
258
The ID of an existing private String.

platform: zone into which to add DNS
records for the cluster’s
alibabacloud: internal API. An existing
private zone can only be used
privateZoneID when also using existing VPC.
: The private zone must be
associated with the VPC
containing the subnets. Leave
the private zone unset to have
the installation program
create the private zone on
your behalf.
5.8. UNINSTALLING A CLUSTER ON ALIBABA CLOUD

You can remove a cluster that you deployed to Alibaba Cloud.
5.8.1. Removing a cluster that uses installer-provisioned infrastructure

You can remove a cluster that uses installer-provisioned infrastructure from your cloud.
NOTE
After uninstallation, check your cloud provider for any resources not removed properly,
especially with User Provisioned Infrastructure (UPI) clusters. There might be resources
that the installer did not create or that the installer is unable to access.
Prerequisites
You have a copy of the installation program that you used to deploy the cluster.
You have the files that the installation program generated when you created your cluster.
Procedure
1. From the directory that contains the installation program on the computer that you used to
install the cluster, run the following command:
$ ./openshift-install destroy cluster \

--dir <installation_directory> --log-level info 1 2
2 To view different details, specify warn, debug, or error instead of info.
NOTE
259
NOTE
You must specify the directory that contains the cluster definition files for your
cluster. The installation program requires the metadata.json file in this directory
to delete the cluster.
2. Optional: Delete the <installation_directory> directory and the OpenShift Container Platform
installation program.
260
CHAPTER 6. INSTALLING ON AWS
6.1. PREPARING TO INSTALL ON AWS
processes.
users.
6.1.2. Requirements for installing OpenShift Container Platform on AWS

Before installing OpenShift Container Platform on Amazon Web Services (AWS), you must create an
AWS account. See Configuring an AWS account for details about configuring an account, account limits,
account permissions, IAM user setup, and supported AWS regions.
If the cloud identity and access management (IAM) APIs are not accessible in your environment, or if
you do not want to store an administrator-level credential secret in the kube-system namespace, see
Manually creating long-term credentials for AWS or configuring an AWS cluster to use short-term
credentials with Amazon Web Services Security Token Service (AWS STS).
6.1.3. Choosing a method to install OpenShift Container Platform on AWS

You can install OpenShift Container Platform on installer-provisioned or user-provisioned
infrastructure. The default installation type uses installer-provisioned infrastructure, where the
installation program provisions the underlying infrastructure for the cluster. You can also install
OpenShift Container Platform on infrastructure that you provision. If you do not use infrastructure that
the installation program provisions, you must manage and maintain the cluster resources yourself.
See Installation process for more information about installer-provisioned and user-provisioned
installation processes.
6.1.3.1. Installing a cluster on a single node
Installing OpenShift Container Platform on a single node alleviates some of the requirements for high
availability and large scale clusters. However, you must address the requirements for installing on a
single node, and the additional requirements for installing single-node OpenShift on a cloud provider .
After addressing the requirements for single node installation, use the Installing a customized cluster on
AWS procedure to install the cluster. The installing single-node OpenShift manually section contains an
exemplary install-config.yaml file when installing an OpenShift Container Platform cluster on a single
node.
6.1.3.2. Installing a cluster on installer-provisioned infrastructure
You can install a cluster on AWS infrastructure that is provisioned by the OpenShift Container Platform
installation program, by using one of the following methods:
Installing a cluster quickly on AWS: You can install OpenShift Container Platform on AWS
infrastructure that is provisioned by the OpenShift Container Platform installation program. You
can install a cluster quickly by using the default configuration options.
261
Installing a customized cluster on AWS: You can install a customized cluster on AWS
infrastructure that the installation program provisions. The installation program allows for some
Installing a cluster on AWS with network customizations: You can customize your OpenShift
Container Platform network configuration during installation, so that your cluster can coexist
with your existing IP address allocations and adhere to your network requirements.
Installing a cluster on AWS in a restricted network: You can install OpenShift Container
Platform on AWS on installer-provisioned infrastructure by using an internal mirror of the
installation release content. You can use this method to install a cluster that does not require an
active internet connection to obtain the software components.
Installing a cluster on an existing Virtual Private Cloud: You can install OpenShift Container
Platform on an existing AWS Virtual Private Cloud (VPC). You can use this installation method if
you have constraints set by the guidelines of your company, such as limits when creating new
accounts or infrastructure.
Installing a private cluster on an existing VPC: You can install a private cluster on an existing
AWS VPC. You can use this method to deploy OpenShift Container Platform on an internal
network that is not visible to the internet.
Installing a cluster on AWS into a government or secret region: OpenShift Container

Platform can be deployed into AWS regions that are specifically designed for US government
agencies at the federal, state, and local level, as well as contractors, educational institutions, and
other US customers that must run sensitive workloads in the cloud.
6.1.3.3. Installing a cluster on user-provisioned infrastructure
You can install a cluster on AWS infrastructure that you provision, by using one of the following methods:
Installing a cluster on AWS infrastructure that you provide: You can install OpenShift
Container Platform on AWS infrastructure that you provide. You can use the provided
CloudFormation templates to create stacks of AWS resources that represent each of the
components required for an OpenShift Container Platform installation.
Installing a cluster on AWS in a restricted network with user-provisioned infrastructure: You

can install OpenShift Container Platform on AWS infrastructure that you provide by using an
internal mirror of the installation release content. You can use this method to install a cluster
that does not require an active internet connection to obtain the software components. You can
also use this installation method to ensure that your clusters only use container images that
satisfy your organizational controls on external content. While you can install OpenShift
Container Platform by using the mirrored content, your cluster still requires internet access to
use the AWS APIs.
6.1.4. Next steps

Configuring an AWS account
6.2. CONFIGURING AN AWS ACCOUNT

Before you can install OpenShift Container Platform, you must configure an Amazon Web Services
(AWS) account.
262
6.2.1. Configuring Route 53

To install OpenShift Container Platform, the Amazon Web Services (AWS) account you use must have a
dedicated public hosted zone in your Route 53 service. This zone must be authoritative for the domain.
The Route 53 service provides cluster DNS resolution and name lookup for external connections to the
cluster.
Procedure
registrar or obtain a new one through AWS or another source.
NOTE
If you purchase a new domain through AWS, it takes time for the relevant DNS
changes to propagate. For more information about purchasing domains through
AWS, see Registering Domain Names Using Amazon Route 53 in the AWS
documentation.
2. If you are using an existing domain and registrar, migrate its DNS to AWS. See Making Amazon
Route 53 the DNS Service for an Existing Domain in the AWS documentation.
3. Create a public hosted zone for your domain or subdomain. See Creating a Public Hosted Zone
in the AWS documentation.
4. Extract the new authoritative name servers from the hosted zone records. See Getting the
Name Servers for a Public Hosted Zone in the AWS documentation.
5. Update the registrar records for the AWS Route 53 name servers that your domain uses. For
example, if you registered your domain to a Route 53 service in a different accounts, see the
following topic in the AWS documentation: Adding or Changing Name Servers or Glue Records.
6. If you are using a subdomain, add its delegation records to the parent domain. This gives
Amazon Route 53 responsibility for the subdomain. Follow the delegation procedure outlined by
the DNS provider of the parent domain. See Creating a subdomain that uses Amazon Route 53
as the DNS service without migrating the parent domain in the AWS documentation for an
example high level procedure.
6.2.1.1. Ingress Operator endpoint configuration for AWS Route 53
If you install in either Amazon Web Services (AWS) GovCloud (US) US-West or US-East region, the
Ingress Operator uses us-gov-west-1 region for Route53 and tagging API clients.
The Ingress Operator uses https://tagging.us-gov-west-1.amazonaws.com as the tagging API

endpoint if a tagging custom endpoint is configured that includes the string 'us-gov-east-1'.
For more information on AWS GovCloud (US) endpoints, see the Service Endpoints in the AWS
documentation about GovCloud (US).
IMPORTANT
Private, disconnected installations are not supported for AWS GovCloud when you install
in the us-gov-east-1 region.
263
Example Route 53 configuration
platform:
aws:
region: us-gov-west-1
serviceEndpoints:
- name: ec2
url: https://ec2.us-gov-west-1.amazonaws.com
- name: elasticloadbalancing
url: https://elasticloadbalancing.us-gov-west-1.amazonaws.com
- name: route53
url: https://route53.us-gov.amazonaws.com 1
- name: tagging
url: https://tagging.us-gov-west-1.amazonaws.com 2
1 Route 53 defaults to https://route53.us-gov.amazonaws.com for both AWS GovCloud (US)

regions.
2 Only the US-West region has endpoints for tagging. Omit this parameter if your cluster is in
another region.
6.2.2. AWS account limits

The OpenShift Container Platform cluster uses a number of Amazon Web Services (AWS) components,
and the default Service Limits affect your ability to install OpenShift Container Platform clusters. If you
use certain cluster configurations, deploy your cluster in certain AWS regions, or run multiple clusters
from your account, you might need to request additional resources for your AWS account.
The following table summarizes the AWS components whose limits can impact your ability to install and
run OpenShift Container Platform clusters.
Compone Number of Default AWS Description

nt clusters limit
available by
default
264

nt clusters limit
available by
default
Instance Varies Varies By default, each cluster creates the following

Limits instances:
One bootstrap machine, which is removed

after installation
Three control plane nodes
Three worker nodes
These instance type counts are within a new

account’s default limit. To deploy more worker
nodes, enable autoscaling, deploy large workloads,
or use a different instance type, review your account
limits to ensure that your cluster can deploy the
machines that you need.
In most regions, the worker machines use an

m6i.large instance and the bootstrap and control
plane machines use m6i.xlarge instances. In some
regions, including all regions that do not support
these instance types, m5.large and m5.xlarge
instances are used instead.
Elastic IPs 0 to 1 5 EIPs per To provision the cluster in a highly available

(EIPs) account configuration, the installation program creates a
public and private subnet for each availability zone
within a region. Each private subnet requires aNAT
Gateway, and each NAT gateway requires a
separate elastic IP. Review the AWS region map to
determine how many availability zones are in each
region. To take advantage of the default high
availability, install the cluster in a region with at least
three availability zones. To install a cluster in a
region with more than five availability zones, you
must increase the EIP limit.
IMPORTANT
To use the us-east-1 region, you

must increase the EIP limit for your
account.
Virtual 5 5 VPCs per Each cluster creates its own VPC.

Private region
Clouds
(VPCs)
265

nt clusters limit
available by
default
Elastic 3 20 per region By default, each cluster creates internal and

Load external network load balancers for the master API
Balancing server and a single Classic Load Balancer for the
(ELB/NLB router. Deploying more Kubernetes Service objects
) with type LoadBalancer will create additional load
balancers.
NAT 5 5 per availability The cluster deploys one NAT gateway in each
Gateways zone availability zone.
Elastic At least 12 350 per region The default installation creates 21 ENIs and an ENI
Network for each availability zone in your region. For
Interfaces example, the us-east-1 region contains six
(ENIs) availability zones, so a cluster that is deployed in
that zone uses 27 ENIs. Review the AWS region map
to determine how many availability zones are in each
region.
Additional ENIs are created for additional machines

and ELB load balancers that are created by cluster
usage and deployed workloads.
VPC 20 20 per account Each cluster creates a single VPC Gateway for S3
Gateway access.
S3 buckets 99 100 buckets per Because the installation process creates a

account temporary bucket and the registry component in
each cluster creates a bucket, you can create only
99 OpenShift Container Platform clusters per AWS
account.
Security 250 2,500 per Each cluster creates 10 distinct security groups.
Groups account
6.2.3. Required AWS permissions for the IAM user
NOTE
Your IAM user must have the permission tag:GetResources in the region us-east-1 to
delete the base cluster resources. As part of the AWS API requirement, the OpenShift
Container Platform installation program performs various actions in this region.
When you attach the AdministratorAccess policy to the IAM user that you create in Amazon Web
Services (AWS), you grant that user all of the required permissions. To deploy all components of an
OpenShift Container Platform cluster, the IAM user requires the following permissions:
266
Example 6.1. Required EC2 permissions for installation
ec2:AuthorizeSecurityGroupEgress
ec2:AuthorizeSecurityGroupIngress
ec2:CopyImage
ec2:CreateNetworkInterface
ec2:AttachNetworkInterface
ec2:CreateSecurityGroup
ec2:CreateTags
ec2:CreateVolume
ec2:DeleteSecurityGroup
ec2:DeleteSnapshot
ec2:DeleteTags
ec2:DeregisterImage
ec2:DescribeAccountAttributes
ec2:DescribeAddresses
ec2:DescribeAvailabilityZones
ec2:DescribeDhcpOptions
ec2:DescribeImages
ec2:DescribeInstanceAttribute
ec2:DescribeInstanceCreditSpecifications
ec2:DescribeInstances
ec2:DescribeInstanceTypes
ec2:DescribeInternetGateways
ec2:DescribeKeyPairs
ec2:DescribeNatGateways
ec2:DescribeNetworkAcls
ec2:DescribeNetworkInterfaces
ec2:DescribePrefixLists
ec2:DescribeRegions
267
ec2:DescribeRouteTables
ec2:DescribeSecurityGroups
ec2:DescribeSecurityGroupRules
ec2:DescribeSubnets
ec2:DescribeTags
ec2:DescribeVolumes
ec2:DescribeVpcAttribute
ec2:DescribeVpcClassicLink
ec2:DescribeVpcClassicLinkDnsSupport
ec2:DescribeVpcEndpoints
ec2:DescribeVpcs
ec2:GetEbsDefaultKmsKeyId
ec2:ModifyInstanceAttribute
ec2:ModifyNetworkInterfaceAttribute
ec2:RevokeSecurityGroupEgress
ec2:RevokeSecurityGroupIngress
ec2:RunInstances
ec2:TerminateInstances
Example 6.2. Required permissions for creating network resources during installation
ec2:AllocateAddress
ec2:AssociateAddress
ec2:AssociateDhcpOptions
ec2:AssociateRouteTable
ec2:AttachInternetGateway
ec2:CreateDhcpOptions
ec2:CreateInternetGateway
ec2:CreateNatGateway
ec2:CreateRoute
ec2:CreateRouteTable
268
ec2:CreateSubnet
ec2:CreateVpc
ec2:CreateVpcEndpoint
ec2:ModifySubnetAttribute
ec2:ModifyVpcAttribute
NOTE
If you use an existing VPC, your account does not require these permissions for
creating network resources.
Example 6.3. Required Elastic Load Balancing permissions (ELB) for installation
elasticloadbalancing:AddTags
elasticloadbalancing:ApplySecurityGroupsToLoadBalancer
elasticloadbalancing:AttachLoadBalancerToSubnets
elasticloadbalancing:ConfigureHealthCheck
elasticloadbalancing:CreateLoadBalancer
elasticloadbalancing:CreateLoadBalancerListeners
elasticloadbalancing:DeleteLoadBalancer
elasticloadbalancing:DeregisterInstancesFromLoadBalancer
elasticloadbalancing:DescribeInstanceHealth
elasticloadbalancing:DescribeLoadBalancerAttributes
elasticloadbalancing:DescribeLoadBalancers
elasticloadbalancing:DescribeTags
elasticloadbalancing:ModifyLoadBalancerAttributes
elasticloadbalancing:RegisterInstancesWithLoadBalancer
elasticloadbalancing:SetLoadBalancerPoliciesOfListener
Example 6.4. Required Elastic Load Balancing permissions (ELBv2) for installation
elasticloadbalancing:CreateListener
269
elasticloadbalancing:CreateTargetGroup
elasticloadbalancing:DeregisterTargets
elasticloadbalancing:DescribeListeners
elasticloadbalancing:DescribeTargetGroupAttributes
elasticloadbalancing:DescribeTargetHealth
elasticloadbalancing:ModifyTargetGroup
elasticloadbalancing:ModifyTargetGroupAttributes
elasticloadbalancing:RegisterTargets
Example 6.5. Required IAM permissions for installation
iam:AddRoleToInstanceProfile
iam:CreateInstanceProfile
iam:CreateRole
iam:DeleteInstanceProfile
iam:DeleteRole
iam:DeleteRolePolicy
iam:GetInstanceProfile
iam:GetRole
iam:GetRolePolicy
iam:GetUser
iam:ListInstanceProfilesForRole
iam:ListRoles
iam:ListUsers
iam:PassRole
iam:PutRolePolicy
iam:RemoveRoleFromInstanceProfile
270
iam:SimulatePrincipalPolicy
iam:TagRole
NOTE
If you have not created a load balancer in your AWS account, the IAM user also
requires the iam:CreateServiceLinkedRole permission.
Example 6.6. Required Route 53 permissions for installation
route53:ChangeResourceRecordSets
route53:ChangeTagsForResource
route53:CreateHostedZone
route53:DeleteHostedZone
route53:GetChange
route53:GetHostedZone
route53:ListHostedZones
route53:ListHostedZonesByName
route53:ListResourceRecordSets
route53:ListTagsForResource
route53:UpdateHostedZoneComment
Example 6.7. Required S3 permissions for installation
s3:CreateBucket
s3:DeleteBucket
s3:GetAccelerateConfiguration
s3:GetBucketAcl
s3:GetBucketCors
s3:GetBucketLocation
s3:GetBucketLogging
s3:GetBucketPolicy
s3:GetBucketObjectLockConfiguration
s3:GetBucketRequestPayment
271
s3:GetBucketTagging
s3:GetBucketVersioning
s3:GetBucketWebsite
s3:GetEncryptionConfiguration
s3:GetLifecycleConfiguration
s3:GetReplicationConfiguration
s3:ListBucket
s3:PutBucketAcl
s3:PutBucketTagging
s3:PutEncryptionConfiguration
Example 6.8. S3 permissions that cluster Operators require
s3:DeleteObject
s3:GetObject
s3:GetObjectAcl
s3:GetObjectTagging
s3:GetObjectVersion
s3:PutObject
s3:PutObjectAcl
s3:PutObjectTagging
Example 6.9. Required permissions to delete base cluster resources
autoscaling:DescribeAutoScalingGroups
ec2:DeletePlacementGroup
ec2:DeleteNetworkInterface
ec2:DeleteVolume
elasticloadbalancing:DeleteTargetGroup
elasticloadbalancing:DescribeTargetGroups
iam:DeleteAccessKey
iam:DeleteUser
272
iam:ListAttachedRolePolicies
iam:ListInstanceProfiles
iam:ListRolePolicies
iam:ListUserPolicies
s3:DeleteObject
s3:ListBucketVersions
tag:GetResources
Example 6.10. Required permissions to delete network resources
ec2:DeleteDhcpOptions
ec2:DeleteInternetGateway
ec2:DeleteNatGateway
ec2:DeleteRoute
ec2:DeleteRouteTable
ec2:DeleteSubnet
ec2:DeleteVpc
ec2:DeleteVpcEndpoints
ec2:DetachInternetGateway
ec2:DisassociateRouteTable
ec2:ReleaseAddress
ec2:ReplaceRouteTableAssociation
NOTE
If you use an existing VPC, your account does not require these permissions to delete
network resources. Instead, your account only requires the tag:UntagResources
permission to delete network resources.
Example 6.11. Required permissions to delete a cluster with shared instance roles
iam:UntagRole
Example 6.12. Additional IAM and S3 permissions that are required to create manifests
273
iam:DeleteAccessKey
iam:DeleteUser
iam:DeleteUserPolicy
iam:GetUserPolicy
iam:ListAccessKeys
iam:PutUserPolicy
iam:TagUser
s3:PutBucketPublicAccessBlock
s3:GetBucketPublicAccessBlock
s3:PutLifecycleConfiguration
s3:ListBucket
s3:ListBucketMultipartUploads
s3:AbortMultipartUpload
NOTE
If you are managing your cloud provider credentials with mint mode, the IAM user also
requires the iam:CreateAccessKey and iam:CreateUser permissions.
Example 6.13. Optional permissions for instance and quota checks for installation
ec2:DescribeInstanceTypeOfferings
servicequotas:ListAWSDefaultServiceQuotas
Example 6.14. Optional permissions for the cluster owner account when installing a cluster on a
shared VPC
sts:AssumeRole
6.2.4. Creating an IAM user

Each Amazon Web Services (AWS) account contains a root user account that is based on the email
address you used to create the account. This is a highly-privileged account, and it is recommended to
use it for only initial account and billing configuration, creating an initial set of users, and securing the
account.
Before you install OpenShift Container Platform, create a secondary IAM administrative user. As you
complete the Creating an IAM User in Your AWS Account procedure in the AWS documentation, set the
following options:
274
Procedure
1. Specify the IAM user name and select Programmatic access.
2. Attach the AdministratorAccess policy to ensure that the account has sufficient permission to
create the cluster. This policy provides the cluster with the ability to grant credentials to each
OpenShift Container Platform component. The cluster grants the components only the
credentials that they require.
NOTE
While it is possible to create a policy that grants the all of the required AWS
permissions and attach it to the user, this is not the preferred option. The cluster
will not have the ability to grant additional credentials to individual components,
so the same credentials are used by all components.
3. Optional: Add metadata to the user by attaching tags.
4. Confirm that the user name that you specified is granted the AdministratorAccess policy.
5. Record the access key ID and secret access key values. You must use these values when you
configure your local machine to run the installation program.
IMPORTANT
You cannot use a temporary session token that you generated while using a
multi-factor authentication device to authenticate to AWS when you deploy a
cluster. The cluster continues to use your current AWS credentials to create AWS
resources for the entire life of the cluster, so you must use key-based, long-term
credentials.
6.2.5. IAM Policies and AWS authentication

By default, the installation program creates instance profiles for the bootstrap, control plane, and
compute instances with the necessary permissions for the cluster to operate.
However, you can create your own IAM roles and specify them as part of the installation process. You
might need to specify your own roles to deploy the cluster or to manage the cluster after installation.
For example:
Your organization’s security policies require that you use a more restrictive set of permissions to
install the cluster.
After the installation, the cluster is configured with an Operator that requires access to
additional services.
If you choose to specify your own IAM roles, you can take the following steps:
Begin with the default policies and adapt as required. For more information, see "Default
permissions for IAM instance profiles".
Use the AWS Identity and Access Management Access Analyzer (IAM Access Analyzer) to
create a policy template that is based on the cluster’s activity. For more information see, "Using
AWS IAM Analyzer to create policy templates".
275
6.2.5.1. Default permissions for IAM instance profiles
By default, the installation program creates IAM instance profiles for the bootstrap, control plane and
worker instances with the necessary permissions for the cluster to operate.
The following lists specify the default permissions for control plane and compute machines:
Example 6.15. Default IAM role permissions for control plane instance profiles
ec2:AttachVolume
ec2:CreateTags
ec2:CreateVolume
ec2:DeleteVolume
ec2:Describe*
ec2:DetachVolume
ec2:ModifyVolume
elasticloadbalancing:CreateLoadBalancerPolicy
elasticloadbalancing:DeleteListener
elasticloadbalancing:DeleteLoadBalancerListeners
276
elasticloadbalancing:Describe*
elasticloadbalancing:DetachLoadBalancerFromSubnets
elasticloadbalancing:ModifyListener
elasticloadbalancing:SetLoadBalancerPoliciesForBackendServer
kms:DescribeKey
Example 6.16. Default IAM role permissions for compute instance profiles
ec2:DescribeRegions
6.2.5.2. Specifying an existing IAM role
Instead of allowing the installation program to create IAM instance profiles with the default permissions,
you can use the install-config.yaml file to specify an existing IAM role for control plane and compute
instances.
Prerequisites
Procedure
1. Update compute.platform.aws.iamRole with an existing role for the control plane machines.
Sample install-config.yaml file with an IAM role for compute instances
compute:
- hyperthreading: Enabled
name: worker
277
platform:
aws:
iamRole: ExampleRole
2. Update controlPlane.platform.aws.iamRole with an existing role for the compute machines.
Sample install-config.yaml file with an IAM role for control plane instances
controlPlane:
name: master
platform:
aws:
iamRole: ExampleRole
3. Save the file and reference it when installing the OpenShift Container Platform cluster.
See Deploying the cluster.
6.2.5.3. Using AWS IAM Analyzer to create policy templates
The minimal set of permissions that the control plane and compute instance profiles require depends on
how the cluster is configured for its daily operation.
One way to determine which permissions the cluster instances require is to use the AWS Identity and
Access Management Access Analyzer (IAM Access Analyzer) to create a policy template:
A policy template contains the permissions the cluster has used over a specified period of time.
You can then use the template to create policies with fine-grained permissions.
Procedure
The overall process could be:
1. Ensure that CloudTrail is enabled. CloudTrail records all of the actions and events in your AWS
account, including the API calls that are required to create a policy template. For more
information, see the AWS documentation for working with CloudTrail.
2. Create an instance profile for control plane instances and an instance profile for compute
instances. Be sure to assign each role a permissive policy, such as PowerUserAccess. For more
information, see the AWS documentation for creating instance profile roles .
3. Install the cluster in a development environment and configure it as required. Be sure to deploy
all of applications the cluster will host in a production environment.
4. Test the cluster thoroughly. Testing the cluster ensures that all of the required API calls are
logged.
5. Use the IAM Access Analyzer to create a policy template for each instance profile. For more
information, see the AWS documentation for generating policies based on the CloudTrail logs .
6. Create and add a fine-grained policy to each instance profile.
278
7. Remove the permissive policy from each instance profile.
8. Deploy a production cluster using the existing instance profiles with the new policies.
NOTE
You can add IAM Conditions to your policy to make it more restrictive and compliant with
your organization security requirements.
6.2.6. Supported AWS Marketplace regions

Installing an OpenShift Container Platform cluster using an AWS Marketplace image is available to
customers who purchase the offer in North America.
While the offer must be purchased in North America, you can deploy the cluster to any of the following
supported paritions:
Public
GovCloud
NOTE
Deploying a OpenShift Container Platform cluster using an AWS Marketplace image is

not supported for the AWS secret regions or China regions.
6.2.7. Supported AWS regions

You can deploy an OpenShift Container Platform cluster to the following regions.
NOTE
6.2.7.1. AWS public regions
The following AWS public regions are supported:
af-south-1 (Cape Town)
ap-east-1 (Hong Kong)
ap-northeast-1 (Tokyo)
ap-northeast-2 (Seoul)
ap-northeast-3 (Osaka)
ap-south-1 (Mumbai)
ap-south-2 (Hyderabad)
ap-southeast-1 (Singapore)
279
ap-southeast-2 (Sydney)
ap-southeast-3 (Jakarta)
ap-southeast-4 (Melbourne)
ca-central-1 (Central)
eu-central-1 (Frankfurt)
eu-central-2 (Zurich)
eu-north-1 (Stockholm)
eu-south-1 (Milan)
eu-south-2 (Spain)
eu-west-1 (Ireland)
eu-west-2 (London)
eu-west-3 (Paris)
il-central-1 (Tel Aviv)
me-central-1 (UAE)
me-south-1 (Bahrain)
sa-east-1 (São Paulo)
us-east-1 (N. Virginia)
us-east-2 (Ohio)
us-west-1 (N. California)
us-west-2 (Oregon)
6.2.7.2. AWS GovCloud regions
The following AWS GovCloud regions are supported:
us-gov-west-1
us-gov-east-1
6.2.7.3. AWS SC2S and C2S secret regions
The following AWS secret regions are supported:
us-isob-east-1 Secret Commercial Cloud Services (SC2S)
us-iso-east-1 Commercial Cloud Services (C2S)
6.2.7.4. AWS China regions
280
The following AWS China regions are supported:
cn-north-1 (Beijing)
cn-northwest-1 (Ningxia)
6.2.8. Next steps

Install an OpenShift Container Platform cluster:
Quickly install a cluster with default options on installer-provisioned infrastructure
Install a cluster with cloud customizations on installer-provisioned infrastructure
Install a cluster with network customizations on installer-provisioned infrastructure
Installing a cluster on user-provisioned infrastructure in AWS by using CloudFormation

templates
6.3. INSTALLING A CLUSTER QUICKLY ON AWS

In OpenShift Container Platform version 4.15, you can install a cluster on Amazon Web Services (AWS)
that uses the default configuration options.
processes.
users.
You configured an AWS account to host the cluster.
IMPORTANT
If you have an AWS profile stored on your computer, it must not use a temporary
session token that you generated while using a multi-factor authentication
device. The cluster continues to use your current AWS credentials to create AWS
credentials. To generate appropriate keys, see Managing Access Keys for IAM
Users in the AWS documentation. You can supply the keys when you run the

281
IMPORTANT

authentication.
user.
IMPORTANT
NOTE
Procedure
NOTE
282
NOTE
ecdsa algorithm.
NOTE

task:
Example output
Agent pid 31874
NOTE

Example output
Next steps
283
program.

for installation.
Prerequisites
Procedure
IMPORTANT
IMPORTANT
components.

IMPORTANT
284
IMPORTANT
Prerequisites
cluster.
Procedure
1. Change to the directory that contains the installation program and initialize the cluster
deployment:

--log-level=info 2
Use an empty directory. Some installation assets, such as bootstrap X.509 certificates, have
short expiration intervals, therefore you must not reuse an installation directory. If you want
to reuse individual files from another cluster installation, you can copy them into your
directory. However, the file names for the installation assets might change between
releases. Use caution when copying installation files from an earlier OpenShift Container
Platform version.
2. Provide values at the prompts:
a. Optional: Select an SSH key to use to access your cluster machines.
NOTE
For production OpenShift Container Platform clusters on which you want to

perform installation debugging or disaster recovery, specify an SSH key that
your ssh-agent process uses.
b. Select aws as the platform to target.
c. If you do not have an Amazon Web Services (AWS) profile stored on your computer, enter
285
c. If you do not have an Amazon Web Services (AWS) profile stored on your computer, enter
the AWS access key ID and secret access key for the user that you configured to run the
NOTE
The AWS access key ID and secret access key are stored in
~/.aws/credentials in the home directory of the current user on the
installation host. You are prompted for the credentials by the installation
program if the credentials for the exported profile are not present in the file.
Any credentials that you provide to the installation program are stored in the
file.
d. Select the AWS region to deploy the cluster to.
e. Select the base domain for the Route 53 service that you configured for your cluster.
f. Enter a descriptive name for your cluster.
g. Paste the pull secret from Red Hat OpenShift Cluster Manager .
3. Optional: Remove or disable the AdministratorAccess policy from the IAM account that you
used to install the cluster.
NOTE
The elevated permissions provided by the AdministratorAccess policy are

required only during installation.
Verification
IMPORTANT
Example output
...
IMPORTANT
286
IMPORTANT
See Configuration and credential file settings in the AWS documentation for more information
about AWS profile and credential configuration.

IMPORTANT

Procedure
Portal.
$ tar xvf <file>

$ echo $PATH
287
Verification
$ oc <command>

Procedure
Portal.

C:\> path
Verification
C:\> oc <command>

Procedure
Portal.
NOTE

$ echo $PATH
288
Verification
$ oc <command>

Prerequisites
Procedure
$ oc whoami
Example output
system:admin

Prerequisites
Procedure
installation host:
289
NOTE

NOTE
Example output


6.3.10. Next steps

If necessary, you can remove cloud provider credentials .
290
6.4. INSTALLING A CLUSTER ON AWS WITH CUSTOMIZATIONS

the installation program provisions on Amazon Web Services (AWS). To customize the installation, you
modify parameters in the install-config.yaml file before you install the cluster.
NOTE

processes.
users.
IMPORTANT
resources for the entire life of the cluster, so you must use long-term credentials.
To generate appropriate keys, see Managing Access Keys for IAM Users in the
AWS documentation. You can supply the keys when you run the installation
program.

IMPORTANT
291
IMPORTANT

authentication.
user.
IMPORTANT
NOTE
Procedure
NOTE
ecdsa algorithm.
292
NOTE

task:
Example output
Agent pid 31874
NOTE

Example output
Next steps
program.
6.4.4. Obtaining an AWS Marketplace image
If you are deploying an OpenShift Container Platform cluster using an AWS Marketplace image, you
293
must first subscribe through AWS. Subscribing to the offer provides you with the AMI ID that the
installation program uses to deploy compute nodes.
Prerequisites
You have an AWS account to purchase the offer. This account does not have to be the same
account that is used to install the cluster.
Procedure
1. Complete the OpenShift Container Platform subscription from the AWS Marketplace.
2. Record the AMI ID for your specific AWS Region. As part of the installation process, you must
update the install-config.yaml file with this value before deploying the cluster.
Sample install-config.yaml file with AWS Marketplace compute nodes
apiVersion: v1
baseDomain: example.com
compute:
name: worker
platform:
aws:
amiID: ami-06c4d345f7c207239 1
type: m5.4xlarge
replicas: 3
metadata:
name: test-cluster
platform:
aws:
region: us-east-2 2
sshKey: ssh-ed25519 AAAA...
pullSecret: '{"auths": ...}'
1 The AMI ID from your AWS Marketplace subscription.
2 Your AMI ID is associated with a specific AWS Region. When creating the installation
configuration file, ensure that you select the same AWS Region that you specified when
configuring your subscription.

for installation.
Prerequisites
Procedure
294
IMPORTANT
IMPORTANT
components.

You can customize the OpenShift Container Platform cluster you install on Amazon Web Services
(AWS).
Prerequisites
cluster.
Procedure
command:
295
NOTE

ii. Select AWS as the platform to target.
iii. If you do not have an Amazon Web Services (AWS) profile stored on your computer,
enter the AWS access key ID and secret access key for the user that you configured to
run the installation program.
iv. Select the AWS region to deploy the cluster to.
v. Select the base domain for the Route 53 service that you configured for your cluster.
vi. Enter a descriptive name for your cluster.
NOTE
If you are installing a three-node cluster, be sure to set the compute.replicas

parameter to 0. This ensures that the cluster’s control planes are schedulable. For
more information, see "Installing a three-node cluster on AWS".
IMPORTANT

Installation configuration parameters for AWS
296
6.4.6.1. Minimum resource requirements for cluster installation
Each cluster machine must meet the following minimum requirements:
Table 6.1. Minimum resource requirements
Machine Operating vCPU [1] Virtual RAM Storage Input/Output

System Per Second
(IOPS)[2]
Bootstrap RHCOS 4 16 GB 100 GB 300
Control plane RHCOS 4 16 GB 100 GB 300
Compute RHCOS, RHEL 2 8 GB 100 GB 300

8.6 and later
[3]
1. One vCPU is equivalent to one physical core when simultaneous multithreading (SMT), or
hyperthreading, is not enabled. When enabled, use the following formula to calculate the
corresponding ratio: (threads per core × cores) × sockets = vCPUs.
2. OpenShift Container Platform and Kubernetes are sensitive to disk performance, and faster
storage is recommended, particularly for etcd on the control plane nodes which require a 10 ms
p99 fsync duration. Note that on many cloud platforms, storage size and IOPS scale together, so
you might need to over-allocate storage volume to obtain sufficient performance.
3. As with all user-provisioned installations, if you choose to use RHEL compute machines in your
cluster, you take responsibility for all operating system life cycle management and maintenance,
including performing system updates, applying patches, and completing all other required tasks.
Use of RHEL 7 compute machines is deprecated and has been removed in OpenShift Container
Platform 4.10 and later.
If an instance type for your platform meets the minimum requirements for cluster machines, it is
supported to use in OpenShift Container Platform.
Optimizing storage
6.4.6.2. Tested instance types for AWS
The following Amazon Web Services (AWS) instance types have been tested with OpenShift Container
Platform.
NOTE
Use the machine types included in the following charts for your AWS instances. If you use
an instance type that is not listed in the chart, ensure that the instance size you use
matches the minimum resource requirements that are listed in the section named
"Minimum resource requirements for cluster installation".
297
Example 6.17. Machine types based on 64-bit x86 architecture
c4.*
c5.*
c5a.*
i3.*
m4.*
m5.*
m5a.*
m6a.*
m6i.*
r4.*
r5.*
r5a.*
r6i.*
t3.*
t3a.*
6.4.6.3. Tested instance types for AWS on 64-bit ARM infrastructures
The following Amazon Web Services (AWS) 64-bit ARM instance types have been tested with
OpenShift Container Platform.
NOTE
Use the machine types included in the following charts for your AWS ARM instances. If
you use an instance type that is not listed in the chart, ensure that the instance size you
use matches the minimum resource requirements that are listed in "Minimum resource
requirements for cluster installation".
Example 6.18. Machine types based on 64-bit ARM architecture
c6g.*
m6g.*
6.4.6.4. Sample customized install-config.yaml file for AWS
298
your OpenShift Container Platform cluster’s platform or modify the values of the required parameters.
IMPORTANT
This sample YAML file is provided for reference only. You must obtain your install-
config.yaml file by using the installation program and modify it.
apiVersion: v1
baseDomain: example.com 1
credentialsMode: Mint 2
controlPlane: 3 4
hyperthreading: Enabled 5
name: master
platform:
aws:
zones:
- us-west-2a
- us-west-2b
rootVolume:
iops: 4000
size: 500
type: io1 6
metadataService:
authentication: Optional 7
type: m6i.xlarge
replicas: 3
compute: 8
- hyperthreading: Enabled 9
name: worker
platform:
aws:
rootVolume:
iops: 2000
size: 500
type: io1 10
metadataService:
type: c5.4xlarge
zones:
- us-west-2c
replicas: 3
metadata:
networking:
clusterNetwork:
- cidr: 10.128.0.0/14
hostPrefix: 23
machineNetwork:
- cidr: 10.0.0.0/16
serviceNetwork:
- 172.30.0.0/16
platform:
299
aws:
region: us-west-2 14
propagateUserTags: true 15
userTags:
adminContact: jdoe
costCenter: 7536
amiID: ami-0c5d3e03c0ab9b19a 16
serviceEndpoints: 17
- name: ec2
url: https://vpce-id.ec2.us-west-2.vpce.amazonaws.com
fips: false 18
sshKey: ssh-ed25519 AAAA... 19
pullSecret: '{"auths": ...}' 20
1 12 14 20 Required. The installation program prompts you for this value.
2 Optional: Add this parameter to force the Cloud Credential Operator (CCO) to use the specified
mode. By default, the CCO uses the root credentials in the kube-system namespace to
dynamically try to determine the capabilities of the credentials. For details about CCO modes, see
the "About the Cloud Credential Operator" section in the Authentication and authorization guide.
3 8 15 If you do not provide these parameters and values, the installation program provides the
default value.
4 The controlPlane section is a single mapping, but the compute section is a sequence of mappings.
To meet the requirements of the different data structures, the first line of the compute section
must begin with a hyphen, -, and the first line of the controlPlane section must not. Only one
control plane pool is used.
5 9 Whether to enable or disable simultaneous multithreading, or hyperthreading. By default,

simultaneous multithreading is enabled to increase the performance of your machines' cores. You
can disable it by setting the parameter value to Disabled. If you disable simultaneous
multithreading in some cluster machines, you must disable it in all cluster machines.
IMPORTANT
If you disable simultaneous multithreading, ensure that your capacity planning

accounts for the dramatically decreased machine performance. Use larger instance
types, such as m4.2xlarge or m5.2xlarge, for your machines if you disable
simultaneous multithreading.
6 10 To configure faster storage for etcd, especially for larger clusters, set the storage type as io1 and
set iops to 2000.
7 11 Whether to require the Amazon EC2 Instance Metadata Service v2 (IMDSv2). To require IMDSv2,
set the parameter value to Required. To allow the use of both IMDSv1 and IMDSv2, set the
parameter value to Optional. If no value is specified, both IMDSv1 and IMDSv2 are allowed.
NOTE
The IMDS configuration for control plane machines that is set during cluster
installation can only be changed by using the AWS CLI. The IMDS configuration for
compute machines can be changed by using compute machine sets.
300
value.
16 The ID of the AMI used to boot machines for the cluster. If set, the AMI must belong to the same
17 The AWS service endpoints. Custom endpoints are required when installing to an unknown AWS
region. The endpoint URL must use the https protocol and the host must trust the certificate.
18 Whether to enable or disable FIPS mode. By default, FIPS mode is not enabled. If FIPS mode is
enabled, the Red Hat Enterprise Linux CoreOS (RHCOS) machines that OpenShift Container
Platform runs on bypass the default Kubernetes cryptography suite and use the cryptography
modules that are provided with RHCOS instead.
IMPORTANT
To enable FIPS mode for your cluster, you must run the installation program from a
Red Hat Enterprise Linux (RHEL) computer configured to operate in FIPS mode.
For more information about configuring FIPS mode on RHEL, see Installing the
system in FIPS mode. When running Red Hat Enterprise Linux (RHEL) or Red Hat
Enterprise Linux CoreOS (RHCOS) booted in FIPS mode, OpenShift Container
Platform core components use the RHEL cryptographic libraries that have been
submitted to NIST for FIPS 140-2/140-3 Validation on only the x86_64, ppc64le,
and s390x architectures.
19 You can optionally provide the sshKey value that you use to access the machines in your cluster.
NOTE
For production OpenShift Container Platform clusters on which you want to perform
installation debugging or disaster recovery, specify an SSH key that your ssh-agent
process uses.
Prerequisites
NOTE
301
NOTE
(169.254.169.254).
Procedure
apiVersion: v1
proxy:
noProxy: ec2.<aws_region>.amazonaws.com,elasticloadbalancing.
<aws_region>.amazonaws.com,s3.<aws_region>.amazonaws.com 3
must be http.

destinations. If you have added the Amazon EC2,Elastic Load Balancing, and S3 VPC
endpoints to your VPC, you must add these endpoints to the noProxy field.
NOTE
302
NOTE
NOTE
NOTE
created.

IMPORTANT

Procedure
Portal.
$ tar xvf <file>

303
$ echo $PATH
Verification
$ oc <command>

Procedure
Portal.

C:\> path
Verification
C:\> oc <command>

Procedure
Portal.
NOTE

304
$ echo $PATH
Verification
$ oc <command>
6.4.8. Alternatives to storing administrator-level secrets in the kube-system project

By default, administrator secrets are stored in the kube-system project. If you configured the
credentialsMode parameter in the install-config.yaml file to Manual, you must use one of the
following alternatives:
To manage long-term cloud credentials manually, follow the procedure in Manually creating
long-term credentials.
To implement short-term credentials that are managed outside the cluster for individual
components, follow the procedures in Configuring an AWS cluster to use short-term
credentials.
6.4.8.1. Manually creating long-term credentials
The Cloud Credential Operator (CCO) can be put into manual mode prior to installation in
environments where the cloud identity and access management (IAM) APIs are not reachable, or the
administrator prefers not to store an administrator-level credential secret in the cluster kube-system
namespace.
Procedure
1. If you did not set the credentialsMode parameter in the install-config.yaml configuration file
to Manual, modify the value as shown:
Sample configuration file snippet
apiVersion: v1
# ...
2. If you have not previously created installation manifest files, do so by running the following
command:
where <installation_directory> is the directory in which the installation program creates files.
4. Extract the list of CredentialsRequest custom resources (CRs) from the OpenShift Container
305

--included \ 1
This command creates a YAML file for each CredentialsRequest object.
Sample CredentialsRequest object
apiVersion: cloudcredential.openshift.io/v1
kind: CredentialsRequest
metadata:
name: <component_credentials_request>
namespace: openshift-cloud-credential-operator
...
spec:
providerSpec:
kind: AWSProviderSpec
statementEntries:
- effect: Allow
action:
- iam:GetUser
- iam:GetUserPolicy
- iam:ListAccessKeys
resource: "*"
...
5. Create YAML files for secrets in the openshift-install manifests directory that you generated
previously. The secrets must be stored using the namespace and secret name defined in the
spec.secretRef for each CredentialsRequest object.
Sample CredentialsRequest object with secrets
metadata:
...
spec:
306
providerSpec:
statementEntries:
- effect: Allow
action:
- s3:CreateBucket
- s3:DeleteBucket
resource: "*"
...
secretRef:
name: <component_secret>
namespace: <component_namespace>
...
Sample Secret object
apiVersion: v1
kind: Secret
metadata:
data:
aws_access_key_id: <base64_encoded_aws_access_key_id>
aws_secret_access_key: <base64_encoded_aws_secret_access_key>
IMPORTANT
Before upgrading a cluster that uses manually maintained credentials, you must ensure
that the CCO is in an upgradeable state.
6.4.8.2. Configuring an AWS cluster to use short-term credentials
To install a cluster that is configured to use the AWS Security Token Service (STS), you must configure
the CCO utility and create the required AWS resources for your cluster.
6.4.8.2.1. Configuring the Cloud Credential Operator utility
NOTE
Prerequisites
You have created an AWS account for the ccoctl utility to use with the following permissions:
Example 6.19. Required AWS permissions
307
Required iam permissions
iam:CreateOpenIDConnectProvider
iam:CreateRole
iam:DeleteOpenIDConnectProvider
iam:DeleteRole
iam:GetOpenIDConnectProvider
iam:GetRole
iam:GetUser
iam:ListOpenIDConnectProviders
iam:ListRoles
iam:PutRolePolicy
iam:TagOpenIDConnectProvider
iam:TagRole
Required s3 permissions
s3:CreateBucket
s3:DeleteBucket
s3:DeleteObject
s3:GetBucketAcl
s3:GetBucketTagging
s3:GetObject
s3:GetObjectAcl
s3:GetObjectTagging
s3:ListBucket
s3:PutBucketAcl
s3:PutBucketPolicy
s3:PutBucketTagging
308
s3:PutObject
s3:PutObjectAcl
s3:PutObjectTagging
Required cloudfront permissions
cloudfront:ListCloudFrontOriginAccessIdentities
cloudfront:ListDistributions
cloudfront:ListTagsForResource
If you plan to store the OIDC configuration in a private S3 bucket that is accessed by the IAM
identity provider through a public CloudFront distribution URL, the AWS account that runs the
ccoctl utility requires the following additional permissions:
Example 6.20. Additional permissions for a private S3 bucket with CloudFront
cloudfront:CreateCloudFrontOriginAccessIdentity
cloudfront:CreateDistribution
cloudfront:DeleteCloudFrontOriginAccessIdentity
cloudfront:DeleteDistribution
cloudfront:GetCloudFrontOriginAccessIdentity
cloudfront:GetCloudFrontOriginAccessIdentityConfig
cloudfront:GetDistribution
cloudfront:TagResource
cloudfront:UpdateDistribution
NOTE
These additional permissions support the use of the --create-private-s3-

bucket option when processing credentials requests with the ccoctl aws
create-all command.
Procedure
309

NOTE

$ chmod 775 ccoctl
Verification
$ ccoctl --help
Usage:
ccoctl [command]
Available Commands:
Flags:
6.4.8.2.2. Creating AWS resources with the Cloud Credential Operator utility
You have the following options when creating AWS resources:
You can use the ccoctl aws create-all command to create the AWS resources automatically.
This is the quickest way to create the resources. See Creating AWS resources with a single
command.
If you need to review the JSON files that the ccoctl tool creates before modifying AWS
310
resources, or if the process the ccoctl tool uses to create AWS resources automatically does
not meet the requirements of your organization, you can create the AWS resources individually.
See Creating AWS resources individually .
6.4.8.2.2.1. Creating AWS resources with a single command
If the process the ccoctl tool uses to create AWS resources automatically meets the requirements of
your organization, you can use the ccoctl aws create-all command to automate the creation of AWS
resources.
Otherwise, you can create the AWS resources individually. For more information, see "Creating AWS
resources individually".
NOTE
Prerequisites
You must have:
Procedure

--included \ 1
NOTE
311
NOTE
command:
$ ccoctl aws create-all \

--name=<name> \ 1
--region=<aws_region> \ 2
--output-dir=<path_to_ccoctl_output_dir> \ 4
--create-private-s3-bucket 5
2 Specify the AWS region in which cloud resources will be created.
3 Specify the directory containing the files for the component CredentialsRequest objects.
4 Optional: Specify the directory in which you want the ccoctl utility to create objects. By
default, the utility creates objects in the directory in which the commands are run.
5 Optional: By default, the ccoctl utility stores the OpenID Connect (OIDC) configuration
files in a public S3 bucket and uses the S3 URL as the public OIDC endpoint. To store the
OIDC configuration in a private S3 bucket that is accessed by the IAM identity provider
through a public CloudFront distribution URL instead, use the --create-private-s3-bucket
parameter.
NOTE
preview parameter.
Verification
To verify that the OpenShift Container Platform secrets are created, list the files in the
<path_to_ccoctl_output_dir>/manifests directory:
Example output
cluster-authentication-02-config.yaml
openshift-cloud-credential-operator-cloud-credential-operator-iam-ro-creds-credentials.yaml
openshift-cloud-network-config-controller-cloud-credentials-credentials.yaml
openshift-cluster-api-capa-manager-bootstrap-credentials-credentials.yaml
openshift-cluster-csi-drivers-ebs-cloud-credentials-credentials.yaml
openshift-machine-api-aws-cloud-credentials-credentials.yaml
312
You can verify that the IAM roles are created by querying AWS. For more information, refer to
AWS documentation on listing IAM roles.
6.4.8.2.2.2. Creating AWS resources individually
You can use the ccoctl tool to create AWS resources individually. This option might be useful for an
organization that shares the responsibility for creating these resources among different users or
departments.
Otherwise, you can use the ccoctl aws create-all command to create the AWS resources automatically.
For more information, see "Creating AWS resources with a single command".
NOTE
Some ccoctl commands make AWS API calls to create or modify AWS resources. You
can use the --dry-run flag to avoid making API calls. Using this flag creates JSON files on
the local file system instead. You can review and modify the JSON files and then apply
them with the AWS CLI tool using the --cli-input-json parameters.
Prerequisites
Extract and prepare the ccoctl binary.
Procedure
1. Generate the public and private RSA key files that are used to set up the OpenID Connect
provider for the cluster by running the following command:
$ ccoctl aws create-key-pair
Example output
2021/04/13 11:01:02 Generating RSA keypair

2021/04/13 11:01:03 Writing private key to /<path_to_ccoctl_output_dir>/serviceaccount-
signer.private
2021/04/13 11:01:03 Writing public key to /<path_to_ccoctl_output_dir>/serviceaccount-
signer.public
2021/04/13 11:01:03 Copying signing key for use by installer
where serviceaccount-signer.private and serviceaccount-signer.public are the generated

key files.
This command also creates a private key that the cluster requires during installation in
/<path_to_ccoctl_output_dir>/tls/bound-service-account-signing-key.key.
2. Create an OpenID Connect identity provider and S3 bucket on AWS by running the following
command:
$ ccoctl aws create-identity-provider \

--name=<name> \ 1
313
--public-key-file=<path_to_ccoctl_output_dir>/serviceaccount-signer.public 3
1 <name> is the name used to tag any cloud resources that are created for tracking.
2 <aws-region> is the AWS region in which cloud resources will be created.
3 <path_to_ccoctl_output_dir> is the path to the public key file that the ccoctl aws
create-key-pair command generated.
Example output
2021/04/13 11:16:09 Bucket <name>-oidc created

2021/04/13 11:16:10 OpenID Connect discovery document in the S3 bucket <name>-oidc at
.well-known/openid-configuration updated
2021/04/13 11:16:10 Reading public key
2021/04/13 11:16:10 JSON web key set (JWKS) in the S3 bucket <name>-oidc at keys.json
updated
2021/04/13 11:16:18 Identity Provider created with ARN: arn:aws:iam::
<aws_account_id>:oidc-provider/<name>-oidc.s3.<aws_region>.amazonaws.com
where openid-configuration is a discovery document and keys.json is a JSON web key set file.
This command also creates a YAML configuration file in

/<path_to_ccoctl_output_dir>/manifests/cluster-authentication-02-config.yaml. This file
sets the issuer URL field for the service account tokens that the cluster generates, so that the
AWS IAM identity provider trusts the tokens.
3. Create IAM roles for each component in the cluster:
a. Set a $RELEASE_IMAGE variable with the release image from your installation file by
b. Extract the list of CredentialsRequest objects from the OpenShift Container Platform
release image:

--included \ 1
--install-config=<path_to_directory_with_installation_configuration>/install-config.yaml \
2
c. Use the ccoctl tool to process all CredentialsRequest objects by running the following
314
command:
$ ccoctl aws create-iam-roles \

--name=<name> \
--region=<aws_region> \
--credentials-requests-dir=<path_to_credentials_requests_directory> \
--identity-provider-arn=arn:aws:iam::<aws_account_id>:oidc-provider/<name>-oidc.s3.
<aws_region>.amazonaws.com
NOTE
For AWS environments that use alternative IAM API endpoints, such as
GovCloud, you must also specify your region with the --region parameter.
preview parameter.
For each CredentialsRequest object, ccoctl creates an IAM role with a trust policy that is
tied to the specified OIDC identity provider, and a permissions policy as defined in each
CredentialsRequest object from the OpenShift Container Platform release image.
Verification
Example output
6.4.8.2.3. Incorporating the Cloud Credential Operator utility manifests
To implement short-term security credentials managed outside the cluster for individual components,
you must move the manifest files that the Cloud Credential Operator utility (ccoctl) created to the
correct directories for the installation program.
Prerequisites
315
You have configured the Cloud Credential Operator utility (ccoctl).
You have created the cloud provider resources that are required for your cluster with the ccoctl
utility.
Procedure
apiVersion: v1
# ...
command:
3. Copy the manifests that the ccoctl utility generated to the manifests directory that the
installation program created by running the following command:
$ cp /<path_to_ccoctl_output_dir>/manifests/* ./manifests/
4. Copy the private key that the ccoctl utility generated in the tls directory to the installation
directory by running the following command:
$ cp -a /<path_to_ccoctl_output_dir>/tls .

IMPORTANT
Prerequisites
cluster.
316
Procedure
deployment:

--log-level=info 2

config.yaml file.
NOTE

Verification
IMPORTANT
Example output
...
IMPORTANT
317
IMPORTANT

Prerequisites
Procedure
$ oc whoami
Example output
system:admin

Prerequisites
318
Procedure
installation host:
NOTE

NOTE
Example output


319
6.4.13. Next steps

6.5. INSTALLING A CLUSTER ON AWS WITH NETWORK

CUSTOMIZATIONS
with customized network configuration options. By customizing your network configuration, your cluster
can coexist with existing IP address allocations in your environment and integrate with existing MTU and
VXLAN configurations.
processes.
users.
IMPORTANT

320
IMPORTANT

authentication.
user.
IMPORTANT
NOTE
Procedure
NOTE
321
NOTE
ecdsa algorithm.
NOTE

task:
Example output
Agent pid 31874
NOTE

Example output
Next steps
322
program.

for installation.
Prerequisites
Procedure
IMPORTANT
IMPORTANT
components.

Phase 1
323
NOTE

NIC resides in.
IMPORTANT
cluster.
Phase 2

(AWS).
Prerequisites
cluster.
Procedure
command:
324
NOTE

IMPORTANT

325

System Per Second
(IOPS)[2]

8.6 and later
[3]
Optimizing storage
Platform.
NOTE
c4.*
c5.*
326
c5a.*
i3.*
m4.*
m5.*
m5a.*
m6a.*
m6i.*
r4.*
r5.*
r5a.*
r6i.*
t3.*
t3a.*
NOTE
c6g.*
m6g.*
IMPORTANT
327
apiVersion: v1
controlPlane: 3 4
name: master
platform:
aws:
zones:
- us-west-2a
- us-west-2b
rootVolume:
iops: 4000
size: 500
type: io1 6
metadataService:
type: m6i.xlarge
replicas: 3
compute: 8
name: worker
platform:
aws:
rootVolume:
iops: 2000
size: 500
type: io1 10
metadataService:
type: c5.4xlarge
zones:
- us-west-2c
replicas: 3
metadata:
networking: 13
clusterNetwork:
- cidr: 10.128.0.0/14
hostPrefix: 23
machineNetwork:
- cidr: 10.0.0.0/16
serviceNetwork:
- 172.30.0.0/16
platform:
aws:
userTags:
adminContact: jdoe
costCenter: 7536
328
- name: ec2
fips: false 19
3 8 13 16 If you do not provide these parameters and values, the installation program provides the
default value.

IMPORTANT

set iops to 2000.
NOTE
value.
329
IMPORTANT
NOTE
process uses.
Prerequisites
NOTE
(169.254.169.254).
Procedure
330
apiVersion: v1
proxy:
must be http.

NOTE
NOTE
331
NOTE
created.

IMPORTANT

Procedure
Portal.
$ tar xvf <file>

$ echo $PATH
Verification
$ oc <command>

Procedure
332
Portal.

C:\> path
Verification
C:\> oc <command>

Procedure
Portal.
NOTE

$ echo $PATH
Verification
$ oc <command>
333
credentials.
namespace.
Procedure
apiVersion: v1
# ...
command:

--included \ 1
334
metadata:
...
spec:
providerSpec:
statementEntries:
- effect: Allow
action:
- iam:GetUser
- iam:GetUserPolicy
resource: "*"
...
metadata:
...
spec:
providerSpec:
statementEntries:
- effect: Allow
action:
- s3:CreateBucket
- s3:DeleteBucket
resource: "*"
...
secretRef:
335
...
apiVersion: v1
kind: Secret
metadata:
data:
IMPORTANT
NOTE
Prerequisites
iam:CreateRole
iam:DeleteRole
336
iam:GetRole
iam:GetUser
iam:ListRoles
iam:PutRolePolicy
iam:TagRole
s3:CreateBucket
s3:DeleteBucket
s3:DeleteObject
s3:GetBucketAcl
s3:GetBucketTagging
s3:GetObject
s3:GetObjectAcl
s3:GetObjectTagging
s3:ListBucket
s3:PutBucketAcl
s3:PutBucketPolicy
s3:PutBucketTagging
s3:PutObject
s3:PutObjectAcl
s3:PutObjectTagging
337
NOTE

create-all command.
Procedure

NOTE

338
$ chmod 775 ccoctl
Verification
$ ccoctl --help
Usage:
ccoctl [command]
Available Commands:
Flags:
command.
resources.
339
NOTE
Prerequisites
You must have:
Procedure

--included \ 1
NOTE
command:

--name=<name> \ 1
340
parameter.
NOTE
preview parameter.
Verification
Example output
departments.
341
NOTE
Prerequisites
Procedure
Example output

signer.private
signer.public

key files.
command:

--name=<name> \ 1
342
Example output

updated

release image:

--included \ 1
2
command:

--name=<name> \
NOTE
343
NOTE
preview parameter.
Verification
Example output
Prerequisites
utility.
Procedure
344
apiVersion: v1
# ...
command:

clusterNetwork
serviceNetwork
defaultNetwork.type
345
spec:
clusterNetwork:
- cidr: 10.128.0.0/19
hostPrefix: 23
- cidr: 10.128.32.0/19
hostPrefix: 23
spec:
serviceNetwork:
- 172.30.0.0/14

manifest file.
work

346

NOTE


network plugin.

detected MTU.


value to 1400.

configuration.

347

NOTE


t network
infrastructure
overlaps with the
100.64.0.0/16
IPv4 subnet, you
can specify a
different IP
address range for
internal use by
OVN-Kubernetes.
You must ensure
that the IP address
range does not
overlap with any
other subnet used
by your OpenShift
Container
Platform
address range
must be larger
than the maximum
number of nodes
that can be added
to the cluster. For
example, if the
clusterNetwork.
cidr value is
10.128.0.0/14
and the
clusterNetwork.
hostPrefix value
is /23, then the
maximum number
of nodes is 2^(23-
14)=512 .
This field cannot

be changed after
installation.
348

t network
infrastructure
overlaps with the
fd98::/48 IPv6
subnet, you can
specify a different
IP address range
for internal use by
OVN-Kubernetes.
You must ensure
that the IP address
range does not
overlap with any
other subnet used
by your OpenShift
Container
Platform
address range
must be larger
than the maximum
number of nodes
that can be added
to the cluster.
This field cannot

be changed after
installation.

50000000 or 50 MB.
349
libc
host.
udp:<host>:<port>
unix:<file>
null

350


external hosts.

defaultNetwork:
type: OVNKubernetes
mtu: 1400
genevePort: 6081
ipsecConfig:
mode: Full
IMPORTANT

documentation.
NOTE
351

kubeProxyConfig:
proxyArguments:
- 0s

the cluster.
IMPORTANT

Prerequisites
Procedure
kind: Network
metadata:
name: cluster
spec:
352
kind: Network
metadata:
name: cluster
spec:
defaultNetwork:
ipsecConfig:
mode: Full

NOTE
For more information on using a Network Load Balancer (NLB) on AWS, see Configuring
Ingress cluster traffic on AWS using a Network Load Balancer.
6.5.11. Configuring an Ingress Controller Network Load Balancer on a new AWS

cluster
You can create an Ingress Controller backed by an AWS Network Load Balancer (NLB) on a new cluster.
Prerequisites
Create the install-config.yaml file and complete any modifications to it.
Procedure
Create an Ingress Controller backed by an AWS NLB on a new cluster.
1 For <installation_directory>, specify the name of the directory that contains the install-
2. Create a file that is named cluster-ingress-default-ingresscontroller.yaml in the

<installation_directory>/manifests/ directory:
$ touch <installation_directory>/manifests/cluster-ingress-default-ingresscontroller.yaml 1
1 For <installation_directory>, specify the directory name that contains the manifests/
directory for your cluster.
After creating the file, several network configuration files are in the manifests/ directory, as
shown:
$ ls <installation_directory>/manifests/cluster-ingress-default-ingresscontroller.yaml
353
Example output
cluster-ingress-default-ingresscontroller.yaml
3. Open the cluster-ingress-default-ingresscontroller.yaml file in an editor and enter a custom

resource (CR) that describes the Operator configuration you want:
kind: IngressController
metadata:
name: default
namespace: openshift-ingress-operator
spec:
endpointPublishingStrategy:
loadBalancer:
scope: External
providerParameters:
type: AWS
aws:
type: NLB
type: LoadBalancerService
4. Save the cluster-ingress-default-ingresscontroller.yaml file and quit the text editor.
5. Optional: Back up the manifests/cluster-ingress-default-ingresscontroller.yaml file. The

installation program deletes the manifests/ directory when creating the cluster.

NOTE
Prerequisites

Procedure
where:
354
cluster.

kind: Network
metadata:
name: cluster
spec:
EOF
where:

kind: Network
metadata:
name: cluster
spec:
defaultNetwork:
- cidr: 10.132.0.0/14
hostPrefix: 23
NOTE

port.
355

NOTE
For more information on using Linux and Windows nodes in the same cluster, see
Understanding Windows container workloads .

IMPORTANT
Prerequisites
cluster.
Procedure
deployment:

--log-level=info 2

config.yaml file.
NOTE

Verification
356
IMPORTANT
Example output
...
IMPORTANT

Prerequisites
Procedure
357
$ oc whoami
Example output
system:admin

Prerequisites
Procedure
installation host:
NOTE

NOTE
Example output
358


6.5.17. Next steps

6.6. INSTALLING A CLUSTER ON AWS IN A RESTRICTED NETWORK

In OpenShift Container Platform version 4.15, you can install a cluster on Amazon Web Services (AWS) in
a restricted network by creating an internal mirror of the installation release content on an existing
Amazon Virtual Private Cloud (VPC).
processes.
users.
You mirrored the images for a disconnected installation to your registry and obtained the
imageContentSources data for your version of OpenShift Container Platform.
IMPORTANT
359
IMPORTANT
Because the installation media is on the mirror host, you can use that computer
to complete all installation steps.
You have an existing VPC in AWS. When installing to a restricted network using installer-
provisioned infrastructure, you cannot use the installer-provisioned VPC. You must use a user-
provisioned VPC that satisfies one of the following requirements:
Contains the mirror registry
Has firewall rules or a peering connection to access the mirror registry hosted elsewhere
IMPORTANT
You downloaded the AWS CLI and installed it on your computer. See Install the AWS CLI Using
the Bundled Installer (Linux, macOS, or Unix) in the AWS documentation.
If you use a firewall and plan to use the Telemetry service, you configured the firewall to allow
the sites that your cluster requires access to.
NOTE
If you are configuring a proxy, be sure to also review this site list.
6.6.2. About installations in restricted networks

In OpenShift Container Platform 4.15, you can perform an installation that does not require an active
connection to the internet to obtain software components. Restricted network installations can be
completed using installer-provisioned infrastructure or user-provisioned infrastructure, depending on
the cloud platform to which you are installing the cluster.
If you choose to perform a restricted network installation on a cloud platform, you still require access to
its cloud APIs. Some cloud functions, like Amazon Web Service’s Route 53 DNS and IAM services, require
internet access. Depending on your network, you might require less internet access for an installation on
bare metal hardware, Nutanix, or on VMware vSphere.
To complete a restricted network installation, you must create a registry that mirrors the contents of the
OpenShift image registry and contains the installation media. You can create this registry on a mirror
host, which can access both the internet and your closed network, or by using other methods that meet
your restrictions.
6.6.2.1. Additional limits
Clusters in restricted networks have the following additional limitations and restrictions:
360
The ClusterVersion status includes an Unable to retrieve available updates error.
By default, you cannot use the contents of the Developer Catalog because you cannot access
the required image stream tags.
6.6.3. About using a custom VPC

In OpenShift Container Platform 4.15, you can deploy a cluster into existing subnets in an existing
Amazon Virtual Private Cloud (VPC) in Amazon Web Services (AWS). By deploying OpenShift
Container Platform into an existing AWS VPC, you might be able to avoid limit constraints in new
accounts or more easily abide by the operational constraints that your company’s guidelines set. If you
cannot obtain the infrastructure creation permissions that are required to create the VPC yourself, use
this installation option.
Because the installation program cannot know what other components are also in your existing subnets,
it cannot choose subnet CIDRs and so forth on your behalf. You must configure networking for the
subnets that you install your cluster to yourself.
The installation program no longer creates the following components:
Internet gateways
NAT gateways
Subnets
Route tables
VPCs
VPC DHCP options
VPC endpoints
NOTE
If you use a custom VPC, you must correctly configure it and its subnets for the installation program and
the cluster to use. See Amazon VPC console wizard configurations and Work with VPCs and subnets in
the AWS documentation for more information on creating and managing an AWS VPC.
The installation program cannot:
Subdivide network ranges for the cluster to use.
Set route tables for the subnets.
Set VPC options like DHCP.
You must complete these tasks before you install the cluster. See VPC networking components and
Route tables for your VPC for more information on configuring networking in an AWS VPC.
361
Your VPC must meet the following characteristics:
The VPC must not use the kubernetes.io/cluster/.*: owned, Name, and openshift.io/cluster
tags.
The installation program modifies your subnets to add the kubernetes.io/cluster/.*: shared
tag, so your subnets must have at least one free tag slot available for it. See Tag Restrictions in
the AWS documentation to confirm that the installation program can add a tag to each subnet
that you specify. You cannot use a Name tag, because it overlaps with the EC2 Name field and
the installation fails.
If you want to extend your OpenShift Container Platform cluster into an AWS Outpost and have
an existing Outpost subnet, the existing subnet must use the
kubernetes.io/cluster/unmanaged: true tag. If you do not apply this tag, the installation might
fail due to the Cloud Controller Manager creating a service load balancer in the Outpost subnet,
which is an unsupported configuration.
You must enable the enableDnsSupport and enableDnsHostnames attributes in your VPC, so
that the cluster can use the Route 53 zones that are attached to the VPC to resolve cluster’s
internal DNS records. See DNS Support in Your VPC in the AWS documentation.
If you prefer to use your own Route 53 hosted private zone, you must associate the existing
hosted zone with your VPC prior to installing a cluster. You can define your hosted zone using
the platform.aws.hostedZone and platform.aws.hostedZoneRole fields in the install-
config.yaml file. You can use a private hosted zone from another account by sharing it with the
account where you install the cluster. If you use a private hosted zone from another account, you
must use the Passthrough or Manual credentials mode.
If you are working in a disconnected environment, you are unable to reach the public IP addresses for
EC2, ELB, and S3 endpoints. Depending on the level to which you want to restrict internet traffic during
the installation, the following configuration options are available:
Option 1: Create VPC endpoints

Create a VPC endpoint and attach it to the subnets that the clusters are using. Name the endpoints as
follows:
ec2.<aws_region>.amazonaws.com
elasticloadbalancing.<aws_region>.amazonaws.com
s3.<aws_region>.amazonaws.com
With this option, network traffic remains private between your VPC and the required AWS services.
Option 2: Create a proxy without VPC endpoints

As part of the installation process, you can configure an HTTP or HTTPS proxy. With this option, internet
traffic goes through the proxy to reach the required AWS services.
Option 3: Create a proxy with VPC endpoints

As part of the installation process, you can configure an HTTP or HTTPS proxy with VPC endpoints.
follows:
When configuring the proxy in the install-config.yaml file, add these endpoints to the noProxy field.
362
With this option, the proxy prevents the cluster from accessing the internet directly. However, network
traffic remains private between your VPC and the required AWS services.
Required VPC components

You must provide a suitable VPC and subnets that allow communication to your machines.
Compone AWS type Description

nt
VPC You must provide a public VPC for the

AWS::EC2::VPC
cluster to use. The VPC uses an endpoint
AWS::EC2::VPCEndpoint that references the route tables for each
subnet to improve communication with
the registry that is hosted in S3.
Public Your VPC must have public subnets for

AWS::EC2::Subnet
subnets between 1 and 3 availability zones and
AWS::EC2::SubnetNetworkAclAss associate them with appropriate Ingress
ociation rules.
Internet You must have a public internet gateway,

AWS::EC2::InternetGateway
gateway with public routes, attached to the VPC.
AWS::EC2::VPCGatewayAttachme In the provided templates, each public
nt subnet has a NAT gateway with an EIP
address. These NAT gateways allow
AWS::EC2::RouteTable cluster resources, like private subnet
instances, to reach the internet and are
AWS::EC2::Route
not required for some restricted network
AWS::EC2::SubnetRouteTableAss or proxy scenarios.
ociation
AWS::EC2::NatGateway
AWS::EC2::EIP
Network You must allow the VPC to access the

AWS::EC2::NetworkAcl
access following ports:
control AWS::EC2::NetworkAclEntry
Port Reason
80 Inbound HTTP
traffic
443 Inbound HTTPS

traffic
22 Inbound SSH
traffic
363

nt
1024 - 65535 Inbound

ephemeral traffic
0 - 65535 Outbound
ephemeral traffic
Private Your VPC can have private subnets. The

AWS::EC2::Subnet
subnets provided CloudFormation templates can
AWS::EC2::RouteTable create private subnets for between 1 and
3 availability zones. If you use private
AWS::EC2::SubnetRouteTableAss subnets, you must provide appropriate
ociation routes and tables for them.
To ensure that the subnets that you provide are suitable, the installation program confirms the following
data:
All the subnets that you specify exist.
You provide private subnets.
The subnet CIDRs belong to the machine CIDR that you specified.
You provide subnets for each availability zone. Each availability zone contains no more than one
public and one private subnet. If you use a private cluster, provide only a private subnet for each
availability zone. Otherwise, provide exactly one public and private subnet for each availability
zone.
You provide a public subnet for each private subnet availability zone. Machines are not
provisioned in availability zones that you do not provide private subnets for.
If you destroy a cluster that uses an existing VPC, the VPC is not deleted. When you remove the
OpenShift Container Platform cluster from a VPC, the kubernetes.io/cluster/.*: shared tag is removed
from the subnets that it used.
Starting with OpenShift Container Platform 4.3, you do not need all of the permissions that are required
for an installation program-provisioned infrastructure cluster to deploy a cluster. This change mimics
the division of permissions that you might have at your company: some individuals can create different
resource in your clouds than others. For example, you might be able to create application-specific items,
like instances, buckets, and load balancers, but not networking-related components such as VPCs,
subnets, or ingress rules.
The AWS credentials that you use when you create your cluster do not need the networking permissions
that are required to make VPCs and core networking components within the VPC, such as subnets,
routing tables, internet gateways, NAT, and VPN. You still need permission to make the application
resources that the machines within the cluster require, such as ELBs, security groups, S3 buckets, and
nodes.
364
If you deploy OpenShift Container Platform to an existing network, the isolation of cluster services is
ICMP ingress is allowed from the entire network.

In OpenShift Container Platform 4.15, you require access to the internet to obtain the images that are
necessary to install your cluster.

authentication.
user.
IMPORTANT
NOTE
365
Procedure
NOTE
ecdsa algorithm.
NOTE

task:
Example output
Agent pid 31874
NOTE

366
Example output
Next steps
program.

(AWS).
Prerequisites
cluster. For a restricted network installation, these files are on your mirror host.
You have the imageContentSources values that were generated during mirror registry
creation.
You have obtained the contents of the certificate for your mirror registry.
Procedure
command:
367
NOTE

2. Edit the install-config.yaml file to give the additional information that is required for an
installation in a restricted network.
a. Update the pullSecret value to contain the authentication information for your registry:
pullSecret: '{"auths":{"<mirror_host_name>:5000": {"auth": "<credentials>","email":

"you@example.com"}}}'
For <mirror_host_name>, specify the registry domain name that you specified in the
certificate for your mirror registry, and for <credentials>, specify the base64-encoded user
name and password for your mirror registry.
b. Add the additionalTrustBundle parameter and value.
additionalTrustBundle: |
ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ
The value must be the contents of the certificate file that you used for your mirror registry.
The certificate file can be an existing, trusted certificate authority, or the self-signed
certificate that you generated for the mirror registry.
c. Define the subnets for the VPC to install the cluster in:
subnets:
- subnet-1
- subnet-2
- subnet-3
d. Add the image content resources, which resemble the following YAML excerpt:
368
imageContentSources:
- mirrors:
- <mirror_host_name>:5000/<repo_name>/release
source: quay.io/openshift-release-dev/ocp-release
- mirrors:
source: registry.redhat.io/ocp/release
For these values, use the imageContentSources that you recorded during mirror registry
creation.
e. Optional: Set the publishing strategy to Internal:
publish: Internal
By setting this option, you create an internal Ingress Controller and a private load balancer.
3. Make any other modifications to the install-config.yaml file that you require.
For more information about the parameters, see "Installation configuration parameters".
IMPORTANT


System Per Second
(IOPS)[2]

8.6 and later
[3]
369
Optimizing storage
IMPORTANT
apiVersion: v1
controlPlane: 3 4
name: master
platform:
aws:
zones:
- us-west-2a
- us-west-2b
rootVolume:
iops: 4000
size: 500
type: io1 6
metadataService:
type: m6i.xlarge
replicas: 3
compute: 8
name: worker
platform:
aws:
rootVolume:
370
iops: 2000
size: 500
type: io1 10
metadataService:
type: c5.4xlarge
zones:
- us-west-2c
replicas: 3
metadata:
networking:
clusterNetwork:
- cidr: 10.128.0.0/14
hostPrefix: 23
machineNetwork:
- cidr: 10.0.0.0/16
serviceNetwork:
- 172.30.0.0/16
platform:
aws:
userTags:
adminContact: jdoe
costCenter: 7536
subnets: 16
- subnet-1
- subnet-2
- subnet-3
- name: ec2
hostedZone: Z3URY6TWQ91KVV 19
fips: false 20
pullSecret: '{"auths":{"<local_registry>": {"auth": "<credentials>","email": "you@example.com"}}}' 22
imageContentSources: 24
- mirrors:
- <local_registry>/<local_repository_name>/release
- mirrors:
source: quay.io/openshift-release-dev/ocp-v4.0-art-dev
1 12 14 Required. The installation program prompts you for this value.
371
default value.

IMPORTANT

set iops to 2000.
NOTE
value.
16 If you provide your own VPC, specify subnets for each availability zone that your cluster uses.
19 The ID of your existing Route 53 private hosted zone. Providing an existing hosted zone requires
that you supply your own VPC and the hosted zone is already associated with the VPC prior to
installing your cluster. If undefined, the installation program creates a new hosted zone.
IMPORTANT
372
IMPORTANT
NOTE
process uses.
22 For <local_registry>, specify the registry domain name, and optionally the port, that your mirror
registry uses to serve content. For example registry.example.com or
registry.example.com:5000. For <credentials>, specify the base64-encoded user name and
password for your mirror registry.
23 Provide the contents of the certificate file that you used for your mirror registry.
24 Provide the imageContentSources section from the output of the command to mirror the
repository.
Prerequisites
NOTE
(169.254.169.254).
373
Procedure
apiVersion: v1
proxy:
must be http.

NOTE
NOTE
374
NOTE
created.

IMPORTANT

Procedure
Portal.
$ tar xvf <file>

$ echo $PATH
Verification
$ oc <command>

375
Procedure
Portal.

C:\> path
Verification
C:\> oc <command>

Procedure
Portal.
NOTE

$ echo $PATH
Verification
$ oc <command>
376
credentials.
namespace.
Procedure
apiVersion: v1
# ...
command:

--included \ 1
377
metadata:
...
spec:
providerSpec:
statementEntries:
- effect: Allow
action:
- iam:GetUser
- iam:GetUserPolicy
resource: "*"
...
metadata:
...
spec:
providerSpec:
statementEntries:
- effect: Allow
action:
- s3:CreateBucket
- s3:DeleteBucket
resource: "*"
...
secretRef:
378
...
apiVersion: v1
kind: Secret
metadata:
data:
IMPORTANT
NOTE
Prerequisites
iam:CreateRole
iam:DeleteRole
379
iam:GetRole
iam:GetUser
iam:ListRoles
iam:PutRolePolicy
iam:TagRole
s3:CreateBucket
s3:DeleteBucket
s3:DeleteObject
s3:GetBucketAcl
s3:GetBucketTagging
s3:GetObject
s3:GetObjectAcl
s3:GetObjectTagging
s3:ListBucket
s3:PutBucketAcl
s3:PutBucketPolicy
s3:PutBucketTagging
s3:PutObject
s3:PutObjectAcl
s3:PutObjectTagging
380
NOTE

create-all command.
Procedure

NOTE

381
$ chmod 775 ccoctl
Verification
$ ccoctl --help
Usage:
ccoctl [command]
Available Commands:
Flags:
command.
resources.
382
NOTE
Prerequisites
You must have:
Procedure

--included \ 1
NOTE
command:

--name=<name> \ 1
383
parameter.
NOTE
preview parameter.
Verification
Example output
departments.
384
NOTE
Prerequisites
Procedure
Example output

signer.private
signer.public

key files.
command:

--name=<name> \ 1
385
Example output

updated

release image:

--included \ 1
2
command:

--name=<name> \
NOTE
386
NOTE
preview parameter.
Verification
Example output
Prerequisites
utility.
Procedure
387
apiVersion: v1
# ...
command:

IMPORTANT
Prerequisites
cluster.
Procedure
deployment:

--log-level=info 2
For <installation_directory>, specify the location of your customized ./install-

388

config.yaml file.
NOTE

Verification
IMPORTANT
Example output
...
IMPORTANT
389

Prerequisites
Procedure
$ oc whoami
Example output
system:admin
6.6.11. Disabling the default OperatorHub catalog sources

Operator catalogs that source content provided by Red Hat and community projects are configured for
OperatorHub by default during an OpenShift Container Platform installation. In a restricted network
environment, you must disable the default catalogs as a cluster administrator.
Procedure
Disable the sources for the default catalogs by adding disableAllDefaultSources: true to the
OperatorHub object:
$ oc patch OperatorHub cluster --type json \

-p '[{"op": "add", "path": "/spec/disableAllDefaultSources", "value": true}]'
TIP
Alternatively, you can use the web console to manage catalog sources. From the Administration →
Cluster Settings → Configuration → OperatorHub page, click the Sources tab, where you can create,
update, delete, disable, and enable individual sources.
390
6.6.13. Next steps

Configure image streams for the Cluster Samples Operator and the must-gather tool.
Learn how to use Operator Lifecycle Manager (OLM) on restricted networks .
If the mirror registry that you used to install your cluster has a trusted CA, add it to the cluster by
configuring additional trust stores.
6.7. INSTALLING A CLUSTER ON AWS INTO AN EXISTING VPC

In OpenShift Container Platform version 4.15, you can install a cluster into an existing Amazon Virtual
Private Cloud (VPC) on Amazon Web Services (AWS). The installation program provisions the rest of
the required infrastructure, which you can further customize. To customize the installation, you modify
processes.
users.
If the existing VPC is owned by a different account than the cluster, you shared the VPC
between accounts.
IMPORTANT
391
IMPORTANT
program.

Internet gateways
NAT gateways
Subnets
Route tables
VPCs
VPC DHCP options
VPC endpoints
NOTE
392
Create a public and private subnet for each availability zone that your cluster uses. Each
availability zone can contain no more than one public and one private subnet. For an example of
this type of configuration, see VPC with public and private subnets (NAT) in the AWS
documentation.
Record each subnet ID. Completing the installation requires that you enter these values in the
platform section of the install-config.yaml file. See Finding a subnet ID in the AWS
documentation.
The VPC’s CIDR block must contain the Networking.MachineCIDR range, which is the IP
address pool for cluster machines. The subnet CIDR blocks must belong to the machine CIDR
that you specify.
The VPC must have a public internet gateway attached to it. For each availability zone:
The public subnet requires a route to the internet gateway.
The public subnet requires a NAT gateway with an EIP address.
The private subnet requires a route to the NAT gateway in public subnet.
tags.
393

follows:


follows:


nt

AWS::EC2::VPC

AWS::EC2::Subnet
ociation rules.
394

nt

AWS::EC2::Route
ociation
AWS::EC2::EIP

Port Reason
80 Inbound HTTP
traffic
443 Inbound HTTPS

traffic
22 Inbound SSH
traffic
1024 - 65535 Inbound

ephemeral traffic
0 - 65535 Outbound
ephemeral traffic

AWS::EC2::Subnet
data:
395
zone.
nodes.
6.7.2.5. Optional: AWS security groups
By default, the installation program creates and attaches security groups to control plane and compute
machines. The rules associated with the default security groups cannot be modified.
However, you can apply additional existing AWS security groups, which are associated with your existing
VPC, to control plane and compute machines. Applying custom security groups can help you meet the
security needs of your organization, in such cases where you need to control the incoming or outgoing
traffic of these machines.
As part of the installation process, you apply custom security groups by modifying the install-
396
config.yaml file before deploying the cluster.
For more information, see "Applying existing AWS security groups to the cluster".
6.7.2.6. Modifying trust policy when installing into a shared VPC
If you install your cluster using a shared VPC, you can use the Passthrough or Manual credentials
mode. You must add the IAM role used to install the cluster as a principal in the trust policy of the
account that owns the VPC.
If you use Passthrough mode, add the Amazon Resource Name (ARN) of the account that creates the
cluster, such as arn:aws:iam::123456789012:user/clustercreator, to the trust policy as a principal.
If you use Manual mode, add the ARN of the account that creates the cluster as well as the ARN of the
ingress operator role in the cluster owner account, such as arn:aws:iam::123456789012:role/<cluster-
name>-openshift-ingress-operator-cloud-credentials, to the trust policy as principals.
You must add the following actions to the policy:
Example 6.27. Required actions for shared VPC installation
route53:GetAccountLimit
route53:GetChange
tag:GetResources
tag:UntagResources

397
IMPORTANT

authentication.
user.
IMPORTANT
NOTE
Procedure
NOTE
398
NOTE
ecdsa algorithm.
NOTE

task:
Example output
Agent pid 31874
NOTE

Example output
Next steps
399
program.

for installation.
Prerequisites
Procedure
IMPORTANT
IMPORTANT
components.

(AWS).
Prerequisites
400
cluster.
Procedure
command:
NOTE

IMPORTANT
401
IMPORTANT


System Per Second
(IOPS)[2]

8.6 and later
[3]
Optimizing storage
Platform.
402
NOTE
c4.*
c5.*
c5a.*
i3.*
m4.*
m5.*
m5a.*
m6a.*
m6i.*
r4.*
r5.*
r5a.*
r6i.*
t3.*
t3a.*
NOTE
c6g.*
403
m6g.*
IMPORTANT
apiVersion: v1
controlPlane: 3 4
name: master
platform:
aws:
zones:
- us-west-2a
- us-west-2b
rootVolume:
iops: 4000
size: 500
type: io1 6
metadataService:
type: m6i.xlarge
replicas: 3
compute: 8
name: worker
platform:
aws:
rootVolume:
iops: 2000
size: 500
type: io1 10
metadataService:
type: c5.4xlarge
zones:
- us-west-2c
replicas: 3
metadata:
networking:
clusterNetwork:
- cidr: 10.128.0.0/14
hostPrefix: 23
404
machineNetwork:
- cidr: 10.0.0.0/16
serviceNetwork:
- 172.30.0.0/16
platform:
aws:
userTags:
adminContact: jdoe
costCenter: 7536
subnets: 16
- subnet-1
- subnet-2
- subnet-3
- name: ec2
fips: false 20
default value.

IMPORTANT

set iops to 2000.
405
NOTE
value.
IMPORTANT
NOTE
process uses.
406
Prerequisites
NOTE
(169.254.169.254).
Procedure
apiVersion: v1
proxy:
must be http.

407
NOTE
NOTE
NOTE
created.
6.7.6.6. Applying existing AWS security groups to the cluster
Applying existing AWS security groups to your control plane and compute machines can help you meet
the security needs of your organization, in such cases where you need to control the incoming or
outgoing traffic of these machines.
Prerequisites
You have created the security groups in AWS. For more information, see the AWS
documentation about working with security groups.
The security groups must be associated with the existing VPC that you are deploying the cluster
to. The security groups cannot be associated with another VPC.
Procedure
1. In the install-config.yaml file, edit the compute.platform.aws.additionalSecurityGroupIDs

parameter to specify one or more custom security groups for your compute machines.
2. Edit the controlPlane.platform.aws.additionalSecurityGroupIDs parameter to specify one or

more custom security groups for your control plane machines.
3. Save the file and reference it when deploying the cluster.
Sample install-config.yaml file that specifies custom security groups
408
# ...
compute:
name: worker
platform:
aws:
additionalSecurityGroupIDs:
- sg-1 1
- sg-2
replicas: 3
controlPlane:
name: master
platform:
aws:
- sg-3
- sg-4
replicas: 3
platform:
aws:
region: us-east-1
subnets: 2
- subnet-1
- subnet-2
- subnet-3
1 Specify the name of the security group as it appears in the Amazon EC2 console, including the sg
prefix.
2 Specify subnets for each availability zone that your cluster uses.

IMPORTANT

Procedure
Portal.
409
$ tar xvf <file>

$ echo $PATH
Verification
$ oc <command>

Procedure
Portal.

C:\> path
Verification
C:\> oc <command>

Procedure
Portal.
410
NOTE

$ echo $PATH
Verification
$ oc <command>

credentials.
namespace.
Procedure
apiVersion: v1
# ...
command:
411

--included \ 1
metadata:
...
spec:
providerSpec:
statementEntries:
- effect: Allow
action:
- iam:GetUser
- iam:GetUserPolicy
resource: "*"
...
412
metadata:
...
spec:
providerSpec:
statementEntries:
- effect: Allow
action:
- s3:CreateBucket
- s3:DeleteBucket
resource: "*"
...
secretRef:
...
apiVersion: v1
kind: Secret
metadata:
data:
IMPORTANT
NOTE
Prerequisites
413
iam:CreateRole
iam:DeleteRole
iam:GetRole
iam:GetUser
iam:ListRoles
iam:PutRolePolicy
iam:TagRole
s3:CreateBucket
s3:DeleteBucket
s3:DeleteObject
s3:GetBucketAcl
s3:GetBucketTagging
s3:GetObject
s3:GetObjectAcl
s3:GetObjectTagging
s3:ListBucket
s3:PutBucketAcl
414
s3:PutBucketPolicy
s3:PutBucketTagging
s3:PutObject
s3:PutObjectAcl
s3:PutObjectTagging
NOTE

create-all command.
Procedure
415

NOTE

$ chmod 775 ccoctl
Verification
$ ccoctl --help
Usage:
ccoctl [command]
Available Commands:
Flags:
416
command.
resources.
NOTE
Prerequisites
You must have:
Procedure

--included \ 1
417
NOTE
command:

--name=<name> \ 1
parameter.
NOTE
preview parameter.
Verification
Example output
418
departments.
NOTE
Prerequisites
Procedure
Example output

signer.private
signer.public

key files.
command:

--name=<name> \ 1
419
Example output

updated

release image:

--included \ 1
2
420
command:

--name=<name> \
NOTE
preview parameter.
Verification
Example output
Prerequisites
421
utility.
Procedure
apiVersion: v1
# ...
command:

IMPORTANT
Prerequisites
cluster.
422
Procedure
deployment:

--log-level=info 2

config.yaml file.
NOTE

Verification
IMPORTANT
Example output
...
IMPORTANT
423
IMPORTANT

Prerequisites
Procedure
$ oc whoami
Example output
system:admin

Prerequisites
424
Procedure
installation host:
NOTE

NOTE
Example output


425
6.7.13. Next steps

After installing a cluster on AWS into an existing VPC, you can extend the AWS VPC cluster into
an AWS Outpost.
6.8. INSTALLING A PRIVATE CLUSTER ON AWS

In OpenShift Container Platform version 4.15, you can install a private cluster into an existing VPC on
Amazon Web Services (AWS). The installation program provisions the rest of the required
infrastructure, which you can further customize. To customize the installation, you modify parameters in
the install-config.yaml file before you install the cluster.
processes.
users.
IMPORTANT
program.
6.8.2. Private clusters

You can deploy a private OpenShift Container Platform cluster that does not expose external
endpoints. Private clusters are accessible from only an internal network and are not visible to the
internet.
By default, OpenShift Container Platform is provisioned to use publicly-accessible DNS and endpoints.
A private cluster sets the DNS, Ingress Controller, and API server to private when you deploy your
cluster. This means that the cluster resources are only accessible from your internal network and are not
visible to the internet.
IMPORTANT
426
IMPORTANT
If the cluster has any public subnets, load balancer services created by administrators
might be publicly accessible. To ensure cluster security, verify that these services are
explicitly annotated as private.
To deploy a private cluster, you must:
Use existing networking that meets your requirements. Your cluster resources might be shared
between other clusters on the network.
Deploy from a machine that has access to:
The API services for the cloud to which you provision.
The hosts on the network that you provision.
The internet to obtain installation media.
You can use any machine that meets these access requirements and follows your company’s guidelines.
For example, this machine can be a bastion host on your cloud network or a machine that has access to
the network through a VPN.
6.8.2.1. Private clusters in AWS
To create a private cluster on Amazon Web Services (AWS), you must provide an existing private VPC
and subnets to host the cluster. The installation program must also be able to resolve the DNS records
that the cluster requires. The installation program configures the Ingress Operator and API server for
access from only the private network.
The cluster still requires access to internet to access the AWS APIs.
The following items are not required or created when you install a private cluster:
Public subnets
Public load balancers, which support public ingress
A public Route 53 zone that matches the baseDomain for the cluster
The installation program does use the baseDomain that you specify to create a private Route 53 zone
and the required records for the cluster. The cluster is configured so that the Operators do not create
public records for the cluster and all cluster machines are placed in the private subnets that you specify.
6.8.2.1.1. Limitations
The ability to add public functionality to a private cluster is limited.
You cannot make the Kubernetes API endpoints public after installation without taking
additional actions, including creating public subnets in the VPC for each availability zone in use,
creating a public load balancer, and configuring the control plane security groups to allow traffic
from the internet on 6443 (Kubernetes API port).
If you use a public Service type load balancer, you must tag a public subnet in each availability
zone with kubernetes.io/cluster/<cluster-infra-id>: shared so that AWS can use them to
create public load balancers.
427

Internet gateways
NAT gateways
Subnets
Route tables
VPCs
VPC DHCP options
VPC endpoints
NOTE
tags.
428

follows:


follows:
429

nt

AWS::EC2::VPC

AWS::EC2::Subnet
ociation rules.

AWS::EC2::Route
ociation
AWS::EC2::EIP

Port Reason
80 Inbound HTTP
traffic
443 Inbound HTTPS

traffic
22 Inbound SSH
traffic
1024 - 65535 Inbound

ephemeral traffic
430

nt
0 - 65535 Outbound
ephemeral traffic

AWS::EC2::Subnet
data:
zone.
nodes.
431

IMPORTANT

432
authentication.
user.
IMPORTANT
NOTE
Procedure
NOTE
ecdsa algorithm.
433
NOTE

task:
Example output
Agent pid 31874
NOTE

Example output
Next steps
program.

for installation.
Prerequisites
Procedure
434
IMPORTANT
IMPORTANT
components.
6.8.7. Manually creating the installation configuration file

Installing the cluster requires that you manually create the installation configuration file.
Prerequisites
You have an SSH public key on your local machine to provide to the installation program. The
key will be used for SSH authentication onto your cluster nodes for debugging and disaster
recovery.
You have obtained the OpenShift Container Platform installation program and the pull secret
for your cluster.
Procedure
1. Create an installation directory to store your required installation assets in:
$ mkdir <installation_directory>
IMPORTANT
435
IMPORTANT
You must create a directory. Some installation assets, like bootstrap X.509
certificates have short expiration intervals, so you must not reuse an installation
directory. If you want to reuse individual files from another cluster installation,
you can copy them into your directory. However, the file names for the
installation assets might change between releases. Use caution when copying
installation files from an earlier OpenShift Container Platform version.
2. Customize the sample install-config.yaml file template that is provided and save it in the
<installation_directory>.
NOTE
You must name this configuration file install-config.yaml.
IMPORTANT
The install-config.yaml file is consumed during the next step of the installation
process. You must back it up now.

System Per Second
(IOPS)[2]

8.6 and later
[3]
436
Optimizing storage
Platform.
NOTE
c4.*
c5.*
c5a.*
i3.*
m4.*
m5.*
m5a.*
m6a.*
m6i.*
r4.*
r5.*
r5a.*
r6i.*
t3.*
437
t3a.*
NOTE
c6g.*
m6g.*
IMPORTANT
apiVersion: v1
controlPlane: 3 4
name: master
platform:
aws:
zones:
- us-west-2a
- us-west-2b
rootVolume:
iops: 4000
size: 500
type: io1 6
metadataService:
type: m6i.xlarge
replicas: 3
compute: 8
438
name: worker
platform:
aws:
rootVolume:
iops: 2000
size: 500
type: io1 10
metadataService:
type: c5.4xlarge
zones:
- us-west-2c
replicas: 3
metadata:
networking:
clusterNetwork:
- cidr: 10.128.0.0/14
hostPrefix: 23
machineNetwork:
- cidr: 10.0.0.0/16
serviceNetwork:
- 172.30.0.0/16
platform:
aws:
userTags:
adminContact: jdoe
costCenter: 7536
subnets: 16
- subnet-1
- subnet-2
- subnet-3
- name: ec2
fips: false 20
publish: Internal 22
3 8 15
If you do not provide these parameters and values, the installation program provides the default
439
If you do not provide these parameters and values, the installation program provides the default
value.

IMPORTANT

set iops to 2000.
NOTE
value.
IMPORTANT
440
IMPORTANT
NOTE
process uses.
22 How to publish the user-facing endpoints of your cluster. Set publish to Internal to deploy a
private cluster, which cannot be accessed from the internet. The default value is External.
Prerequisites
NOTE
(169.254.169.254).
Procedure
apiVersion: v1
441
proxy:
must be http.

NOTE
NOTE
442
NOTE
created.
Prerequisites
Procedure


# ...
compute:
name: worker
platform:
aws:
- sg-1 1
- sg-2
replicas: 3
controlPlane:
name: master
platform:
aws:
- sg-3
- sg-4
replicas: 3
platform:
aws:
region: us-east-1
443
subnets: 2
- subnet-1
- subnet-2
- subnet-3
prefix.

IMPORTANT

Procedure
Portal.
$ tar xvf <file>

$ echo $PATH
Verification
$ oc <command>

Procedure
444
Procedure
Portal.

C:\> path
Verification
C:\> oc <command>

Procedure
Portal.
NOTE

$ echo $PATH
Verification
$ oc <command>
445
credentials.
namespace.
Procedure
apiVersion: v1
# ...
command:

--included \ 1
446
metadata:
...
spec:
providerSpec:
statementEntries:
- effect: Allow
action:
- iam:GetUser
- iam:GetUserPolicy
resource: "*"
...
metadata:
...
spec:
providerSpec:
statementEntries:
- effect: Allow
action:
- s3:CreateBucket
- s3:DeleteBucket
resource: "*"
...
secretRef:
447
...
apiVersion: v1
kind: Secret
metadata:
data:
IMPORTANT
NOTE
Prerequisites
iam:CreateRole
iam:DeleteRole
448
iam:GetRole
iam:GetUser
iam:ListRoles
iam:PutRolePolicy
iam:TagRole
s3:CreateBucket
s3:DeleteBucket
s3:DeleteObject
s3:GetBucketAcl
s3:GetBucketTagging
s3:GetObject
s3:GetObjectAcl
s3:GetObjectTagging
s3:ListBucket
s3:PutBucketAcl
s3:PutBucketPolicy
s3:PutBucketTagging
s3:PutObject
s3:PutObjectAcl
s3:PutObjectTagging
449
NOTE

create-all command.
Procedure

NOTE

450
$ chmod 775 ccoctl
Verification
$ ccoctl --help
Usage:
ccoctl [command]
Available Commands:
Flags:
command.
resources.
451
NOTE
Prerequisites
You must have:
Procedure

--included \ 1
NOTE
command:

--name=<name> \ 1
452
parameter.
NOTE
preview parameter.
Verification
Example output
departments.
453
NOTE
Prerequisites
Procedure
Example output

signer.private
signer.public

key files.
command:

--name=<name> \ 1
454
Example output

updated

release image:

--included \ 1
2
command:

--name=<name> \
NOTE
455
NOTE
preview parameter.
Verification
Example output
Prerequisites
utility.
Procedure
456
apiVersion: v1
# ...
command:

IMPORTANT
Prerequisites
cluster.
Procedure
deployment:

--log-level=info 2

457

config.yaml file.
NOTE

Verification
IMPORTANT
Example output
...
IMPORTANT
458

Prerequisites
Procedure
$ oc whoami
Example output
system:admin

Prerequisites
Procedure
installation host:
NOTE

459
NOTE
Example output


6.8.14. Next steps

6.9. INSTALLING A CLUSTER ON AWS INTO A GOVERNMENT REGION

into a government region. To configure the region, modify parameters in the install-config.yaml file
before you install the cluster.
460
processes.
users.
IMPORTANT
program.
6.9.2. AWS government regions

OpenShift Container Platform supports deploying a cluster to an AWS GovCloud (US) region.
The following AWS GovCloud partitions are supported:
us-gov-east-1
us-gov-west-1
6.9.3. Installation requirements

Before you can install the cluster, you must:
Provide an existing private AWS VPC and subnets to host the cluster.
Public zones are not supported in Route 53 in AWS GovCloud. As a result, clusters must be
private when you deploy to an AWS government region.
Manually create the installation configuration file (install-config.yaml).

internet.
NOTE
Public zones are not supported in Route 53 in an AWS GovCloud Region. Therefore,
clusters must be private if they are deployed to an AWS GovCloud Region.
461
IMPORTANT
Public subnets
462

Internet gateways
NAT gateways
Subnets
Route tables
VPCs
VPC DHCP options
VPC endpoints
NOTE
463
tags.

follows:


follows:
464


nt

AWS::EC2::VPC

AWS::EC2::Subnet
ociation rules.

AWS::EC2::Route
ociation
AWS::EC2::EIP

Port Reason
80 Inbound HTTP
traffic
443 Inbound HTTPS

traffic
22 Inbound SSH
traffic
465

nt
1024 - 65535 Inbound

ephemeral traffic
0 - 65535 Outbound
ephemeral traffic

AWS::EC2::Subnet
data:
zone.
nodes.
466

IMPORTANT

467
authentication.
user.
IMPORTANT
NOTE
Procedure
NOTE
ecdsa algorithm.
468
NOTE

task:
Example output
Agent pid 31874
NOTE

Example output
Next steps
program.

Prerequisites
Procedure
469
apiVersion: v1
compute:
name: worker
platform:
aws:
type: m5.4xlarge
replicas: 3
metadata:
name: test-cluster
platform:
aws:
region: us-east-2 2
configuration file, ensure that you select the same AWS Region that you specified when
configuring your subscription.

for installation.
Prerequisites
Procedure
IMPORTANT
470
IMPORTANT
IMPORTANT
components.

Prerequisites
recovery.
for your cluster.
Procedure
IMPORTANT
471
NOTE
IMPORTANT

System Per Second
(IOPS)[2]

8.6 and later
[3]
472
Optimizing storage
Platform.
NOTE
c4.*
c5.*
c5a.*
i3.*
m4.*
m5.*
m5a.*
m6a.*
m6i.*
r4.*
r5.*
r5a.*
r6i.*
t3.*
t3a.*
473
NOTE
c6g.*
m6g.*
IMPORTANT
This sample YAML file is provided for reference only. Use it as a resource to enter
parameter values into the installation configuration file that you created manually.
apiVersion: v1
controlPlane: 3 4
name: master
platform:
aws:
zones:
- us-gov-west-1a
- us-gov-west-1b
rootVolume:
iops: 4000
size: 500
type: io1 6
metadataService:
type: m6i.xlarge
replicas: 3
compute: 8
name: worker
platform:
aws:
rootVolume:
iops: 2000
474
size: 500
type: io1 10
metadataService:
type: c5.4xlarge
zones:
- us-gov-west-1c
replicas: 3
metadata:
networking:
clusterNetwork:
- cidr: 10.128.0.0/14
hostPrefix: 23
machineNetwork:
- cidr: 10.0.0.0/16
serviceNetwork:
- 172.30.0.0/16
platform:
aws:
region: us-gov-west-1 14
userTags:
adminContact: jdoe
costCenter: 7536
subnets: 16
- subnet-1
- subnet-2
- subnet-3
- name: ec2
fips: false 20
1 12 14 23 Required.
default value.
475

IMPORTANT

set iops to 2000.
NOTE
value.
IMPORTANT
476
NOTE
process uses.
Prerequisites
NOTE
(169.254.169.254).
Procedure
apiVersion: v1
proxy:
477
must be http.

NOTE
NOTE
NOTE
created.
478
Prerequisites
Procedure


# ...
compute:
name: worker
platform:
aws:
- sg-1 1
- sg-2
replicas: 3
controlPlane:
name: master
platform:
aws:
- sg-3
- sg-4
replicas: 3
platform:
aws:
region: us-east-1
subnets: 2
- subnet-1
- subnet-2
- subnet-3
prefix.
479

IMPORTANT

Procedure
Portal.
$ tar xvf <file>

$ echo $PATH
Verification
$ oc <command>

Procedure
Portal.
480
C:\> path
Verification
C:\> oc <command>

Procedure
Portal.
NOTE

$ echo $PATH
Verification
$ oc <command>
6.9.12. Alternatives to storing administrator-level secrets in the kube-system

project
components, follow the procedures in Incorporating the Cloud Credential Operator utility
manifests.
481
namespace.
Procedure
apiVersion: v1
# ...
command:

--included \ 1
482
metadata:
...
spec:
providerSpec:
statementEntries:
- effect: Allow
action:
- iam:GetUser
- iam:GetUserPolicy
resource: "*"
...
metadata:
...
spec:
providerSpec:
statementEntries:
- effect: Allow
action:
- s3:CreateBucket
- s3:DeleteBucket
resource: "*"
...
secretRef:
...
apiVersion: v1
kind: Secret
metadata:
483
data:
IMPORTANT
NOTE
Prerequisites
iam:CreateRole
iam:DeleteRole
iam:GetRole
iam:GetUser
iam:ListRoles
484
iam:PutRolePolicy
iam:TagRole
s3:CreateBucket
s3:DeleteBucket
s3:DeleteObject
s3:GetBucketAcl
s3:GetBucketTagging
s3:GetObject
s3:GetObjectAcl
s3:GetObjectTagging
s3:ListBucket
s3:PutBucketAcl
s3:PutBucketPolicy
s3:PutBucketTagging
s3:PutObject
s3:PutObjectAcl
s3:PutObjectTagging
485
NOTE

create-all command.
Procedure

NOTE

$ chmod 775 ccoctl
Verification
$ ccoctl --help
486
Usage:
ccoctl [command]
Available Commands:
Flags:
command.
resources.
NOTE
Prerequisites
You must have:
487
Procedure

--included \ 1
NOTE
command:

--name=<name> \ 1
parameter.
488
NOTE
preview parameter.
Verification
Example output
departments.
NOTE
Prerequisites
Procedure
489
Example output

signer.private
signer.public

key files.
command:

--name=<name> \ 1
Example output

updated

490
release image:

--included \ 1
2
command:

--name=<name> \
NOTE
preview parameter.
Verification
491
Example output
Prerequisites
utility.
Procedure
apiVersion: v1
# ...
command:
492

IMPORTANT
Prerequisites
cluster.
Procedure
deployment:

--log-level=info 2

config.yaml file.
NOTE

Verification
493
IMPORTANT
Example output
...
IMPORTANT

Prerequisites
Procedure
494
$ oc whoami
Example output
system:admin

Prerequisites
Procedure
installation host:
NOTE

NOTE
Example output

495

6.9.17. Next steps

6.10. INSTALLING A CLUSTER ON AWS INTO A SECRET OR TOP

SECRET REGION
into the following secret regions:
Secret Commercial Cloud Services (SC2S)
Commercial Cloud Services (C2S)
To configure a cluster in either region, you change parameters in the install config.yaml file before you
processes.
users.
IMPORTANT
496
IMPORTANT
session token that you generated while using a multifactor authentication device.
The cluster continues to use your current AWS credentials to create AWS
program.
6.10.2. AWS secret regions

The following AWS secret partitions are supported:
us-isob-east-1 (SC2S)
us-iso-east-1 (C2S)
NOTE
The maximum supported MTU in an AWS SC2S and C2S Regions is not the same as AWS
commercial. For more information about configuring MTU during installation, see the
Cluster Network Operator configuration object section in Installing a cluster on AWS with
network customizations

Red Hat does not publish a Red Hat Enterprise Linux CoreOS (RHCOS) Amzaon Machine Image for the
AWS Secret and Top Secret Regions.
Upload a custom RHCOS AMI.
Specify the AWS region, and the accompanying custom AMI, in the installation configuration
file.
You cannot use the OpenShift Container Platform installation program to create the installation
configuration file. The installer does not list an AWS region without native support for an RHCOS AMI.
IMPORTANT
You must also define a custom CA certificate in the additionalTrustBundle field of the
install-config.yaml file because the AWS API requires a custom CA trust bundle. To
allow the installation program to access the AWS API, the CA certificates must also be
defined on the machine that runs the installation program. You must add the CA bundle
to the trust store on the machine, use the AWS_CA_BUNDLE environment variable, or
define the CA bundle in the ca_bundle field of the AWS config file.
497
internet.
NOTE
Public zones are not supported in Route 53 in an AWS Top Secret Region. Therefore,
clusters must be private if they are deployed to an AWS Top Secret Region.
IMPORTANT
Public subnets
498

Internet gateways
NAT gateways
Subnets
Route tables
VPCs
VPC DHCP options
VPC endpoints
NOTE
499
tags.
A cluster in an SC2S or C2S Region is unable to reach the public IP addresses for the EC2, ELB, and S3
endpoints. Depending on the level to which you want to restrict internet traffic during the installation,
the following configuration options are available:

follows:
SC2S
elasticloadbalancing.<aws_region>.sc2s.sgov.gov
ec2.<aws_region>.sc2s.sgov.gov
s3.<aws_region>.sc2s.sgov.gov
500
C2S
elasticloadbalancing.<aws_region>.c2s.ic.gov
ec2.<aws_region>.c2s.ic.gov
s3.<aws_region>.c2s.ic.gov


follows:
SC2S
elasticloadbalancing.<aws_region>.sc2s.sgov.gov
ec2.<aws_region>.sc2s.sgov.gov
s3.<aws_region>.sc2s.sgov.gov
C2S
elasticloadbalancing.<aws_region>.c2s.ic.gov
ec2.<aws_region>.c2s.ic.gov
s3.<aws_region>.c2s.ic.gov


nt

AWS::EC2::VPC
501

nt

AWS::EC2::Subnet
ociation rules.

AWS::EC2::Route
ociation
AWS::EC2::EIP

Port Reason
80 Inbound HTTP
traffic
443 Inbound HTTPS

traffic
22 Inbound SSH
traffic
1024 - 65535 Inbound

ephemeral traffic
0 - 65535 Outbound
ephemeral traffic

AWS::EC2::Subnet
502
data:
zone.
nodes.
503

IMPORTANT
6.10.7. Uploading a custom RHCOS AMI in AWS

If you are deploying to a custom Amazon Web Services (AWS) region, you must upload a custom Red
Hat Enterprise Linux CoreOS (RHCOS) Amazon Machine Image (AMI) that belongs to that region.
Prerequisites
You configured an AWS account.
You created an Amazon S3 bucket with the required IAM service role.
You uploaded your RHCOS VMDK file to Amazon S3. The RHCOS VMDK file must be the
highest version that is less than or equal to the OpenShift Container Platform version you are
installing.
the Bundled Installer.
Procedure
504
1. Export your AWS profile as an environment variable:
$ export AWS_PROFILE=<aws_profile> 1
2. Export the region to associate with your custom AMI as an environment variable:
$ export AWS_DEFAULT_REGION=<aws_region> 1
3. Export the version of RHCOS you uploaded to Amazon S3 as an environment variable:
$ export RHCOS_VERSION=<version> 1
1 1 1 The RHCOS VMDK version, like 4.15.0.
4. Export the Amazon S3 bucket name as an environment variable:
$ export VMIMPORT_BUCKET_NAME=<s3_bucket_name>
5. Create the containers.json file and define your RHCOS VMDK file:
$ cat <<EOF > containers.json

{
"Description": "rhcos-${RHCOS_VERSION}-x86_64-aws.x86_64",
"Format": "vmdk",
"UserBucket": {
"S3Bucket": "${VMIMPORT_BUCKET_NAME}",
"S3Key": "rhcos-${RHCOS_VERSION}-x86_64-aws.x86_64.vmdk"
}
}
EOF
6. Import the RHCOS disk as an Amazon EBS snapshot:
$ aws ec2 import-snapshot --region ${AWS_DEFAULT_REGION} \

--description "<description>" \ 1
--disk-container "file://<file_path>/containers.json" 2
1 The description of your RHCOS disk being imported, like rhcos-${RHCOS_VERSION}-

x86_64-aws.x86_64.
2 The file path to the JSON file describing your RHCOS disk. The JSON file should contain
your Amazon S3 bucket name and key.
7. Check the status of the image import:
$ watch -n 5 aws ec2 describe-import-snapshot-tasks --region ${AWS_DEFAULT_REGION}
Example output
{
"ImportSnapshotTasks": [
505
{
"Description": "rhcos-4.7.0-x86_64-aws.x86_64",
"ImportTaskId": "import-snap-fh6i8uil",
"SnapshotTaskDetail": {
"DiskImageSize": 819056640.0,
"Format": "VMDK",
"SnapshotId": "snap-06331325870076318",
"Status": "completed",
"UserBucket": {
"S3Bucket": "external-images",
"S3Key": "rhcos-4.7.0-x86_64-aws.x86_64.vmdk"
}
}
}
]
}
Copy the SnapshotId to register the image.
8. Create a custom RHCOS AMI from the RHCOS snapshot:
$ aws ec2 register-image \

--region ${AWS_DEFAULT_REGION} \
--architecture x86_64 \ 1
--description "rhcos-${RHCOS_VERSION}-x86_64-aws.x86_64" \ 2
--ena-support \
--name "rhcos-${RHCOS_VERSION}-x86_64-aws.x86_64" \ 3
--virtualization-type hvm \
--root-device-name '/dev/xvda' \
--block-device-mappings 'DeviceName=/dev/xvda,Ebs=
{DeleteOnTermination=true,SnapshotId=<snapshot_ID>}' 4
1 The RHCOS VMDK architecture type, like x86_64, aarch64, s390x, or ppc64le.
2 The Description from the imported snapshot.
3 The name of the RHCOS AMI.
4 The SnapshotID from the imported snapshot.
To learn more about these APIs, see the AWS documentation for importing snapshots and creating
EBS-backed AMIs.

authentication.
506
user.
IMPORTANT
NOTE
Procedure
NOTE
ecdsa algorithm.
NOTE
507
NOTE

task:
Example output
Agent pid 31874
NOTE

Example output
Next steps
program.

for installation.
Prerequisites
Procedure
508
IMPORTANT
IMPORTANT
components.

Prerequisites
You have uploaded a custom RHCOS AMI.
recovery.
for your cluster.
Procedure
IMPORTANT
509
IMPORTANT
NOTE
IMPORTANT
Platform.
NOTE
Example 6.40. Machine types based on 64-bit x86 architecture for secret regions
c4.*
c5.*
i3.*
m4.*
m5.*
r4.*
r5.*
510
t3.*
IMPORTANT
apiVersion: v1
controlPlane: 3 4
name: master
platform:
aws:
zones:
- us-iso-east-1a
- us-iso-east-1b
rootVolume:
iops: 4000
size: 500
type: io1 6
metadataService:
type: m6i.xlarge
replicas: 3
compute: 8
name: worker
platform:
aws:
rootVolume:
iops: 2000
size: 500
type: io1 10
metadataService:
type: c5.4xlarge
zones:
- us-iso-east-1a
- us-iso-east-1b
replicas: 3
metadata:
networking:
clusterNetwork:
- cidr: 10.128.0.0/14
511
hostPrefix: 23
machineNetwork:
- cidr: 10.0.0.0/16
serviceNetwork:
- 172.30.0.0/16
platform:
aws:
region: us-iso-east-1 14
userTags:
adminContact: jdoe
costCenter: 7536
subnets: 16
- subnet-1
- subnet-2
- subnet-3
amiID: ami-96c6f8f7 17 18
- name: ec2
fips: false 21
1 12 14 17 24Required.
default value.

IMPORTANT
512
IMPORTANT

set iops to 2000.
NOTE
value.
IMPORTANT
NOTE
513
NOTE
process uses.
25 The custom CA certificate. This is required when deploying to the SC2S or C2S Regions because
the AWS API requires a custom CA trust bundle.
Prerequisites
NOTE
(169.254.169.254).
Procedure
apiVersion: v1
proxy:
514
must be http.

NOTE
NOTE
NOTE
created.
515
Prerequisites
Procedure


# ...
compute:
name: worker
platform:
aws:
- sg-1 1
- sg-2
replicas: 3
controlPlane:
name: master
platform:
aws:
- sg-3
- sg-4
replicas: 3
platform:
aws:
region: us-east-1
subnets: 2
- subnet-1
- subnet-2
- subnet-3
prefix.
516

IMPORTANT

Procedure
Portal.
$ tar xvf <file>

$ echo $PATH
Verification
$ oc <command>

Procedure
Portal.
517

C:\> path
Verification
C:\> oc <command>

Procedure
Portal.
NOTE

$ echo $PATH
Verification
$ oc <command>

project
518
credentials.
namespace.
Procedure
apiVersion: v1
# ...
command:

--included \ 1
519
metadata:
...
spec:
providerSpec:
statementEntries:
- effect: Allow
action:
- iam:GetUser
- iam:GetUserPolicy
resource: "*"
...
metadata:
...
spec:
providerSpec:
statementEntries:
- effect: Allow
action:
- s3:CreateBucket
- s3:DeleteBucket
resource: "*"
...
secretRef:
...
apiVersion: v1
520
kind: Secret
metadata:
data:
IMPORTANT
NOTE
Prerequisites
iam:CreateRole
iam:DeleteRole
iam:GetRole
iam:GetUser
521
iam:ListRoles
iam:PutRolePolicy
iam:TagRole
s3:CreateBucket
s3:DeleteBucket
s3:DeleteObject
s3:GetBucketAcl
s3:GetBucketTagging
s3:GetObject
s3:GetObjectAcl
s3:GetObjectTagging
s3:ListBucket
s3:PutBucketAcl
s3:PutBucketPolicy
s3:PutBucketTagging
s3:PutObject
s3:PutObjectAcl
s3:PutObjectTagging
522
NOTE

create-all command.
Procedure

NOTE

$ chmod 775 ccoctl
523
Verification
$ ccoctl --help
Usage:
ccoctl [command]
Available Commands:
Flags:
command.
resources.
NOTE
524
Prerequisites
You must have:
Procedure

--included \ 1
NOTE
command:

--name=<name> \ 1
525
NOTE
preview parameter.
Verification
Example output
departments.
NOTE
Prerequisites
526
Procedure
Example output

signer.private
signer.public

key files.
command:

--name=<name> \ 1
Example output

updated

527
release image:

--included \ 1
2
command:

--name=<name> \
NOTE
preview parameter.
Verification
528
Example output
Prerequisites
utility.
Procedure
apiVersion: v1
# ...
command:
529

IMPORTANT
Prerequisites
cluster.
Procedure
deployment:

--log-level=info 2

config.yaml file.
NOTE

Verification
530
IMPORTANT
Example output
...
IMPORTANT

Prerequisites
Procedure
531
$ oc whoami
Example output
system:admin

Prerequisites
Procedure
installation host:
NOTE

NOTE
Example output
532

Accessing the web console

About remote health monitoring
6.10.17. Next steps

6.11. INSTALLING A CLUSTER ON AWS CHINA

In OpenShift Container Platform version 4.15, you can install a cluster to the following Amazon Web
Services (AWS) China regions:
cn-north-1 (Beijing)
cn-northwest-1 (Ningxia)
You have an Internet Content Provider (ICP) license.
processes.
users.
533
IMPORTANT
If you have an AWS profile stored on your computer, it must not use a temporary session
token that you generated while using a multi-factor authentication device. The cluster
continues to use your current AWS credentials to create AWS resources for the entire life
of the cluster, so you must use long-term credentials. To generate appropriate keys, see
Managing Access Keys for IAM Users in the AWS documentation. You can supply the keys
when you run the installation program.

Red Hat does not publish a Red Hat Enterprise Linux CoreOS (RHCOS) Amazon Machine Image (AMI)
for the AWS China regions.
Upload a custom RHCOS AMI.
Specify the AWS region, and the accompanying custom AMI, in the installation configuration
file.
You cannot use the OpenShift Container Platform installation program to create the installation
configuration file. The installer does not list an AWS region without native support for an RHCOS AMI.

IMPORTANT
534
internet.
IMPORTANT
For example, this machine can be a bastion host on your cloud network.
NOTE
AWS China does not support a VPN connection between the VPC and your network. For
more information about the Amazon VPC service in the Beijing and Ningxia regions, see
Amazon Virtual Private Cloud in the AWS China documentation.
Public subnets
535

Internet gateways
NAT gateways
Subnets
Route tables
VPCs
VPC DHCP options
VPC endpoints
NOTE
536
tags.

follows:
ec2.<aws_region>.amazonaws.com.cn
537


follows:
ec2.<aws_region>.amazonaws.com.cn


nt

AWS::EC2::VPC

AWS::EC2::Subnet
ociation rules.

AWS::EC2::Route
ociation
AWS::EC2::EIP
538

nt

Port Reason
80 Inbound HTTP
traffic
443 Inbound HTTPS

traffic
22 Inbound SSH
traffic
1024 - 65535 Inbound

ephemeral traffic
0 - 65535 Outbound
ephemeral traffic

AWS::EC2::Subnet
data:
zone.
539
nodes.

authentication.
540
user.
IMPORTANT
NOTE
Procedure
NOTE
ecdsa algorithm.
NOTE
541
NOTE

task:
Example output
Agent pid 31874
NOTE

Example output
Next steps
program.
6.11.7. Uploading a custom RHCOS AMI in AWS

Prerequisites
installing.
542
Procedure
1 The AWS profile name that holds your AWS credentials, like beijingadmin.
1 The AWS region, like cn-north-1.
1 The RHCOS VMDK version, like 4.15.0.

{
"Format": "vmdk",
"UserBucket": {
}
}
EOF


x86_64-aws.x86_64.
543
Example output
{
{
"Format": "VMDK",
"SnapshotId": "snap-06331325870076318",
"UserBucket": {
}
}
}
]
}

--ena-support \
EBS-backed AMIs.
544
for installation.
Prerequisites
Procedure
IMPORTANT
IMPORTANT
components.

Prerequisites
You have uploaded a custom RHCOS AMI.
recovery.
545
for your cluster.
Procedure
IMPORTANT
NOTE
IMPORTANT
IMPORTANT
apiVersion: v1
controlPlane: 3 4
546
name: master
platform:
aws:
zones:
- cn-north-1a
- cn-north-1b
rootVolume:
iops: 4000
size: 500
type: io1 6
metadataService:
type: m6i.xlarge
replicas: 3
compute: 8
name: worker
platform:
aws:
rootVolume:
iops: 2000
size: 500
type: io1 10
metadataService:
type: c5.4xlarge
zones:
- cn-north-1a
replicas: 3
metadata:
networking:
clusterNetwork:
- cidr: 10.128.0.0/14
hostPrefix: 23
machineNetwork:
- cidr: 10.0.0.0/16
serviceNetwork:
- 172.30.0.0/16
platform:
aws:
region: cn-north-1 14
userTags:
adminContact: jdoe
costCenter: 7536
subnets: 16
- subnet-1
- subnet-2
- subnet-3
amiID: ami-96c6f8f7 17 18
- name: ec2
547
url: https://vpce-id.ec2.cn-north-1.vpce.amazonaws.com.cn
fips: false 21
1 12 14 17 24Required.
default value.

IMPORTANT

set iops to 2000.
NOTE
value.
548
IMPORTANT
NOTE
process uses.

System Per Second
(IOPS)[2]

8.6 and later
[3]
549
Optimizing storage
Platform.
NOTE
c4.*
c5.*
c5a.*
i3.*
m4.*
m5.*
m5a.*
m6a.*
m6i.*
r4.*
550
r5.*
r5a.*
r6i.*
t3.*
t3a.*
NOTE
c6g.*
m6g.*
Prerequisites
NOTE
551
NOTE
(169.254.169.254).
Procedure
apiVersion: v1
proxy:
must be http.

NOTE
552
NOTE
NOTE
NOTE
created.
Prerequisites
Procedure


# ...
compute:
name: worker
553
platform:
aws:
- sg-1 1
- sg-2
replicas: 3
controlPlane:
name: master
platform:
aws:
- sg-3
- sg-4
replicas: 3
platform:
aws:
region: us-east-1
subnets: 2
- subnet-1
- subnet-2
- subnet-3
prefix.

IMPORTANT

Procedure
Portal.
554
$ tar xvf <file>

$ echo $PATH
Verification
$ oc <command>

Procedure
Portal.

C:\> path
Verification
C:\> oc <command>

Procedure
Portal.
NOTE
555
NOTE

$ echo $PATH
Verification
$ oc <command>

project
credentials.
namespace.
Procedure
apiVersion: v1
# ...
command:
556

--included \ 1
metadata:
...
spec:
providerSpec:
statementEntries:
- effect: Allow
action:
- iam:GetUser
- iam:GetUserPolicy
resource: "*"
...
557
metadata:
...
spec:
providerSpec:
statementEntries:
- effect: Allow
action:
- s3:CreateBucket
- s3:DeleteBucket
resource: "*"
...
secretRef:
...
apiVersion: v1
kind: Secret
metadata:
data:
IMPORTANT
NOTE
558
Prerequisites
iam:CreateRole
iam:DeleteRole
iam:GetRole
iam:GetUser
iam:ListRoles
iam:PutRolePolicy
iam:TagRole
s3:CreateBucket
s3:DeleteBucket
s3:DeleteObject
s3:GetBucketAcl
s3:GetBucketTagging
s3:GetObject
s3:GetObjectAcl
s3:GetObjectTagging
s3:ListBucket
559
s3:PutBucketAcl
s3:PutBucketPolicy
s3:PutBucketTagging
s3:PutObject
s3:PutObjectAcl
s3:PutObjectTagging
NOTE

create-all command.
Procedure
560

NOTE

$ chmod 775 ccoctl
Verification
$ ccoctl --help
Usage:
ccoctl [command]
Available Commands:
Flags:
561
command.
resources.
NOTE
Prerequisites
You must have:
Procedure

--included \ 1
Specify the path to the directory where you want to store the CredentialsRequest
562
NOTE
command:

--name=<name> \ 1
parameter.
NOTE
preview parameter.
Verification
Example output
563
departments.
NOTE
Prerequisites
Procedure
Example output

signer.private
signer.public

key files.
564
command:

--name=<name> \ 1
Example output

updated

release image:

--included \ 1
2
565
command:

--name=<name> \
NOTE
preview parameter.
Verification
Example output
566
Prerequisites
utility.
Procedure
apiVersion: v1
# ...
command:

IMPORTANT
Prerequisites
567
cluster.
Procedure
deployment:

--log-level=info 2

config.yaml file.
NOTE

Verification
IMPORTANT
Example output
...
568
IMPORTANT

Prerequisites
Procedure
$ oc whoami
Example output
system:admin

Prerequisites
569
Procedure
installation host:
NOTE

NOTE
Example output


6.11.16. Next steps
570
6.12. INSTALLING A CLUSTER ON USER-PROVISIONED

INFRASTRUCTURE IN AWS BY USING CLOUDFORMATION
TEMPLATES
that uses infrastructure that you provide.
One way to create this infrastructure is to use the provided CloudFormation templates. You can modify
the templates to customize your infrastructure or use the information that they contain to create AWS
objects according to your company’s policies.
IMPORTANT
The steps for performing a user-provisioned infrastructure installation are provided as an

example only. Installing a cluster with infrastructure you provide requires knowledge of
the cloud provider and the installation process of OpenShift Container Platform. Several
CloudFormation templates are provided to assist in completing these steps or to help
model your own. You are also free to create the required resources through other
methods; the templates are just an example.
processes.
users.
IMPORTANT
the Bundled Installer (Linux, macOS, or UNIX) in the AWS documentation.
NOTE
571
NOTE
Be sure to also review this site list if you are configuring a proxy.
If the cloud identity and access management (IAM) APIs are not accessible in your environment,
namespace, you can manually create and maintain long-term credentials .

IMPORTANT
6.12.3. Requirements for a cluster with user-provisioned infrastructure

For a cluster that contains user-provisioned infrastructure, you must deploy all of the required machines.
This section describes the requirements for deploying OpenShift Container Platform on user-
provisioned infrastructure.
6.12.3.1. Required machines for cluster installation
The smallest OpenShift Container Platform clusters require the following hosts:
Table 6.15. Minimum required hosts
Hosts Description
One temporary bootstrap machine The cluster requires the bootstrap machine to deploy
the OpenShift Container Platform cluster on the
three control plane machines. You can remove the
bootstrap machine after you install the cluster.
572
Hosts Description
Three control plane machines The control plane machines run the Kubernetes and
OpenShift Container Platform services that form the
control plane.
At least two compute machines, which are also The workloads requested by OpenShift Container
known as worker machines. Platform users run on the compute machines.
IMPORTANT
To maintain high availability of your cluster, use separate physical hosts for these cluster
machines.
The bootstrap and control plane machines must use Red Hat Enterprise Linux CoreOS (RHCOS) as the
operating system. However, the compute machines can choose between Red Hat Enterprise Linux
CoreOS (RHCOS), Red Hat Enterprise Linux (RHEL) 8.6 and later.
Note that RHCOS is based on Red Hat Enterprise Linux (RHEL) 9.2 and inherits all of its hardware
certifications and requirements. See Red Hat Enterprise Linux technology capabilities and limits .

System Per Second
(IOPS)[2]

8.6 and later
[3]
573
Optimizing storage
Platform.
NOTE
c4.*
c5.*
c5a.*
i3.*
m4.*
m5.*
m5a.*
m6a.*
m6i.*
r4.*
r5.*
r5a.*
r6i.*
t3.*
t3a.*
574
NOTE
c6g.*
m6g.*
6.12.3.5. Certificate signing requests management
Because your cluster has limited access to automatic machine management when you use infrastructure
that you provision, you must provide a mechanism for approving cluster certificate signing requests
(CSRs) after installation. The kube-controller-manager only approves the kubelet client CSRs. The
machine-approver cannot guarantee the validity of a serving certificate that is requested by using
kubelet credentials because it cannot confirm that the correct machine issued the request. You must
determine and implement a method of verifying the validity of the kubelet serving certificate requests
and approving them.
6.12.4. Required AWS infrastructure components

To install OpenShift Container Platform on user-provisioned infrastructure in Amazon Web Services
(AWS), you must manually create both the machines and their supporting infrastructure.
For more information about the integration testing for different platforms, see the OpenShift Container
Platform 4.x Tested Integrations page.
By using the provided CloudFormation templates, you can create stacks of AWS resources that
represent the following components:
An AWS Virtual Private Cloud (VPC)
Networking and load balancing components
Security groups and roles
An OpenShift Container Platform bootstrap node
OpenShift Container Platform control plane nodes
An OpenShift Container Platform compute node
Alternatively, you can manually create the components or you can reuse existing infrastructure that
meets the cluster requirements. Review the CloudFormation templates for more details about how the
components interrelate.
575
6.12.4.1. Other infrastructure components
A VPC
DNS entries
Load balancers (classic or network) and listeners
A public and a private Route 53 zone
Security groups
IAM roles
S3 buckets

follows:


follows:

576

nt

AWS::EC2::VPC

AWS::EC2::Subnet
ociation rules.

AWS::EC2::Route
ociation
AWS::EC2::EIP

Port Reason
80 Inbound HTTP
traffic
443 Inbound HTTPS

traffic
22 Inbound SSH
traffic
1024 - 65535 Inbound

ephemeral traffic
0 - 65535 Outbound
ephemeral traffic
577

nt

AWS::EC2::Subnet
Required DNS and load balancing components

Your DNS and load balancer configuration needs to use a public hosted zone and can use a private
hosted zone similar to the one that the installation program uses if it provisions the cluster’s
infrastructure. You must create a DNS entry that resolves to your load balancer. An entry for api.
<cluster_name>.<domain> must point to the external load balancer, and an entry for api-int.
<cluster_name>.<domain> must point to the internal load balancer.
The cluster also requires load balancers and listeners for port 6443, which are required for the
Kubernetes API and its extensions, and port 22623, which are required for the Ignition config files for
new machines. The targets will be the control plane nodes. Port 6443 must be accessible to both clients
external to the cluster and nodes within the cluster. Port 22623 must be accessible to nodes within the
cluster.
Component AWS type Description
DNS AWS::Route The hosted zone for your internal DNS.

53::HostedZ
one
Public load AWS::Elastic The load balancer for your public subnets.
balancer LoadBalanci
ngV2::LoadB
alancer
External API AWS::Route Alias records for the external API server.
server record 53::RecordS
etGroup
External AWS::Elastic A listener on port 6443 for the external load balancer.
listener LoadBalanci
ngV2::Listen
er
External target AWS::Elastic The target group for the external load balancer.
group LoadBalanci
ngV2::Target
Group
578
Private load AWS::Elastic The load balancer for your private subnets.
ngV2::LoadB
alancer
Internal API AWS::Route Alias records for the internal API server.
etGroup
Internal listener AWS::Elastic A listener on port 22623 for the internal load balancer.
LoadBalanci
ngV2::Listen
er
Internal target AWS::Elastic The target group for the internal load balancer.
group LoadBalanci
ngV2::Target
Group
LoadBalanci
ngV2::Listen
er
group LoadBalanci
ngV2::Target
Group
Security groups
The control plane and worker machines require access to the following ports:
Group Type IP Protocol Port range
MasterSecurityGrou AWS::EC2::Security icmp 0

p Group
tcp 22
tcp 6443
tcp 22623
WorkerSecurityGrou AWS::EC2::Security icmp 0

p Group
tcp 22
579
BootstrapSecurityGr AWS::EC2::Security tcp 22

oup Group
tcp 19531
Control plane Ingress

The control plane machines require the following Ingress groups. Each Ingress group is a
AWS::EC2::SecurityGroupIngress resource.
Ingress group Description IP protocol Port range
MasterIngress etcd tcp 2379- 2380

Etcd
MasterIngress Vxlan packets udp 4789

Vxlan

WorkerVxlan
MasterIngress Internal cluster communication and Kubernetes tcp 9000 - 9999

Internal proxy metrics
MasterIngress Internal cluster communication tcp 9000 - 9999

WorkerInterna
l
MasterIngress Kubernetes kubelet, scheduler and controller tcp 10250 - 10259

Kube manager

WorkerKube manager
MasterIngress Kubernetes Ingress services tcp 30000 - 32767

IngressServic
es

WorkerIngress
Services
MasterIngress Geneve packets udp 6081

Geneve

WorkerGenev
e
580
MasterIngress IPsec IKE packets udp 500

IpsecIke

WorkerIpsecIk
e
MasterIngress IPsec NAT-T packets udp 4500

IpsecNat

WorkerIpsecN
at
MasterIngress IPsec ESP packets 50 All

IpsecEsp

WorkerIpsecE
sp
MasterIngress Internal cluster communication udp 9000 - 9999

InternalUDP

WorkerInterna
lUDP
MasterIngress Kubernetes Ingress services udp 30000 - 32767

IngressServic
esUDP

WorkerIngress
ServicesUDP
Worker Ingress
The worker machines require the following Ingress groups. Each Ingress group is a
WorkerIngress Vxlan packets udp 4789

Vxlan
581

WorkerVxlan
WorkerIngress Internal cluster communication tcp 9000 - 9999

Internal

WorkerInterna
l
WorkerIngress Kubernetes kubelet, scheduler, and controller tcp 10250

Kube manager

WorkerKube manager
WorkerIngress Kubernetes Ingress services tcp 30000 - 32767

IngressServic
es

WorkerIngress
Services
WorkerIngress Geneve packets udp 6081

Geneve

MasterGeneve
WorkerIngress IPsec IKE packets udp 500

IpsecIke

MasterIpsecIk
e
WorkerIngress IPsec NAT-T packets udp 4500

IpsecNat

MasterIpsecN
at
WorkerIngress IPsec ESP packets 50 All

IpsecEsp
582

MasterIpsecEs
p
WorkerIngress Internal cluster communication udp 9000 - 9999

InternalUDP

MasterInternal
UDP
WorkerIngress Kubernetes Ingress services udp 30000 - 32767

IngressServic
esUDP

MasterIngress
ServicesUDP
Roles and instance profiles

You must grant the machines permissions in AWS. The provided CloudFormation templates grant the
machines Allow permissions for the following AWS::IAM::Role objects and provide a
AWS::IAM::InstanceProfile for each set of roles. If you do not use the templates, you can grant the
machines the following broad permissions or the following individual permissions.
Role Effect Action Resource
Master Allow ec2:* *
Allow elasticloadbalancing *
:*
Allow iam:PassRole *
Allow s3:GetObject *
Worker Allow ec2:Describe* *
Bootstrap Allow ec2:Describe* *
Allow ec2:AttachVolume *
Allow ec2:DetachVolume *
6.12.4.2. Cluster machines
583
You need AWS::EC2::Instance objects for the following machines:
A bootstrap machine. This machine is required during installation, but you can remove it after
your cluster deploys.
Three control plane machines. The control plane machines are not governed by a control plane
machine set.
Compute machines. You must create at least two compute machines, which are also known as
worker machines, during installation. These machines are not governed by a compute machine
set.
6.12.4.3. Required AWS permissions for the IAM user
NOTE
ec2:CopyImage
ec2:CreateTags
ec2:CreateVolume
ec2:DeleteSnapshot
ec2:DeleteTags
ec2:DeregisterImage
584
ec2:DescribeImages
ec2:DescribeRegions
ec2:DescribeSubnets
ec2:DescribeTags
ec2:DescribeVolumes
ec2:DescribeVpcs
585
ec2:RunInstances
ec2:AllocateAddress
ec2:CreateRoute
ec2:CreateSubnet
ec2:CreateVpc
NOTE
586
587
iam:CreateRole
iam:DeleteRole
iam:GetRole
iam:GetRolePolicy
iam:GetUser
iam:ListRoles
iam:ListUsers
iam:PassRole
iam:PutRolePolicy
iam:TagRole
NOTE
route53:GetChange
588
s3:CreateBucket
s3:DeleteBucket
s3:GetBucketAcl
s3:GetBucketCors
s3:GetBucketLogging
s3:GetBucketPolicy
s3:GetBucketTagging
s3:GetBucketWebsite
s3:ListBucket
s3:PutBucketAcl
s3:PutBucketTagging
s3:DeleteObject
s3:GetObject
s3:GetObjectAcl
s3:GetObjectTagging
589
s3:GetObjectVersion
s3:PutObject
s3:PutObjectAcl
s3:PutObjectTagging
ec2:DeleteVolume
iam:DeleteAccessKey
iam:DeleteUser
s3:DeleteObject
tag:GetResources
ec2:DeleteRoute
ec2:DeleteSubnet
ec2:DeleteVpc
590
ec2:ReleaseAddress
NOTE
iam:UntagRole
iam:DeleteAccessKey
iam:DeleteUser
iam:GetUserPolicy
iam:ListAccessKeys
iam:PutUserPolicy
iam:TagUser
s3:ListBucket
NOTE
591
shared VPC
sts:AssumeRole

Prerequisites
Procedure
2. Record the AMI ID for your specific AWS Region. If you use the CloudFormation template to
deploy your compute nodes, you must update the worker0.type.properties.ImageID parameter
with the AMI ID value.

for installation.
Prerequisites
Procedure
IMPORTANT
592
IMPORTANT
IMPORTANT
components.

authentication.
user.
IMPORTANT
NOTE
Procedure
593
NOTE
ecdsa algorithm.
NOTE

task:
Example output
Agent pid 31874
NOTE

594
Example output
Next steps
program. If you install a cluster on infrastructure that you provision, you must provide the key to
the installation program.
6.12.8. Creating the installation files for AWS

To install OpenShift Container Platform on Amazon Web Services (AWS) using user-provisioned
infrastructure, you must generate the files that the installation program needs to deploy your cluster
and modify them so that the cluster creates only the machines that it will use. You generate and
customize the install-config.yaml file, Kubernetes manifests, and Ignition config files. You also have the
option to first set up a separate var partition during the preparation phases of installation.
6.12.8.1. Optional: Creating a separate /var partition
It is recommended that disk partitioning for OpenShift Container Platform be left to the installer.
However, there are cases where you might want to create separate partitions in a part of the filesystem
that you expect to grow.
OpenShift Container Platform supports the addition of a single partition to attach storage to either the
/var partition or a subdirectory of /var. For example:
/var/lib/containers: Holds container-related content that can grow as more images and
containers are added to a system.
/var/lib/etcd: Holds data that you might want to keep separate for purposes such as
performance optimization of etcd storage.
/var: Holds data that you might want to keep separate for purposes such as auditing.
Storing the contents of a /var directory separately makes it easier to grow storage for those areas as
needed and reinstall OpenShift Container Platform at a later date and keep that data intact. With this
method, you will not have to pull all your containers again, nor will you have to copy massive log files
when you update systems.
Because /var must be in place before a fresh installation of Red Hat Enterprise Linux CoreOS (RHCOS),
the following procedure sets up the separate /var partition by creating a machine config manifest that is
inserted during the openshift-install preparation phases of an OpenShift Container Platform
installation.
IMPORTANT
595
IMPORTANT
If you follow the steps to create a separate /var partition in this procedure, it is not
necessary to create the Kubernetes manifest and Ignition config files again as described
later in this section.
Procedure
1. Create a directory to hold the OpenShift Container Platform installation files:
$ mkdir $HOME/clusterconfig
2. Run openshift-install to create a set of files in the manifest and openshift subdirectories.
Answer the system questions as you are prompted:
$ openshift-install create manifests --dir $HOME/clusterconfig
Example output
? SSH Public Key ...

INFO Credentials loaded from the "myprofile" profile in file "/home/myuser/.aws/credentials"
INFO Consuming Install Config from target directory
INFO Manifests created in: $HOME/clusterconfig/manifests and
$HOME/clusterconfig/openshift
3. Optional: Confirm that the installation program created manifests in the

clusterconfig/openshift directory:
$ ls $HOME/clusterconfig/openshift/
Example output
99_kubeadmin-password-secret.yaml
99_openshift-cluster-api_master-machines-0.yaml
...
4. Create a Butane config that configures the additional partition. For example, name the file
$HOME/clusterconfig/98-var-partition.bu, change the disk device name to the name of the
storage device on the worker systems, and set the storage size as appropriate. This example
places the /var directory on a separate partition:
variant: openshift
version: 4.15.0
metadata:
labels:
machineconfiguration.openshift.io/role: worker
name: 98-var-partition
storage:
disks:
- device: /dev/disk/by-id/<device_name> 1
partitions:
596
- label: var
start_mib: <partition_start_offset> 2
size_mib: <partition_size> 3
number: 5
filesystems:
- device: /dev/disk/by-partlabel/var
path: /var
format: xfs
mount_options: [defaults, prjquota] 4
with_mount_unit: true
1 The storage device name of the disk that you want to partition.
2 When adding a data partition to the boot disk, a minimum value of 25000 MiB (Mebibytes)
is recommended. The root file system is automatically resized to fill all available space up
to the specified offset. If no value is specified, or if the specified value is smaller than the
recommended minimum, the resulting root file system will be too small, and future
reinstalls of RHCOS might overwrite the beginning of the data partition.
3 The size of the data partition in mebibytes.
4 The prjquota mount option must be enabled for filesystems used for container storage.
NOTE
When creating a separate /var partition, you cannot use different instance types
for worker nodes, if the different instance types do not have the same device
name.
5. Create a manifest from the Butane config and save it to the clusterconfig/openshift directory.
For example, run the following command:
$ butane $HOME/clusterconfig/98-var-partition.bu -o $HOME/clusterconfig/openshift/98-var-

partition.yaml
6. Run openshift-install again to create Ignition configs from a set of files in the manifest and
openshift subdirectories:
$ openshift-install create ignition-configs --dir $HOME/clusterconfig

$ ls $HOME/clusterconfig/
auth bootstrap.ign master.ign metadata.json worker.ign
Now you can use the Ignition config files as input to the installation procedures to install Red Hat
Enterprise Linux CoreOS (RHCOS) systems.
Generate and customize the installation configuration file that the installation program needs to deploy
your cluster.
Prerequisites
You obtained the OpenShift Container Platform installation program for user-provisioned
597
infrastructure and the pull secret for your cluster.
You checked that you are deploying your cluster to an AWS Region with an accompanying Red
Hat Enterprise Linux CoreOS (RHCOS) AMI published by Red Hat. If you are deploying to an
AWS Region that requires a custom AMI, such as an AWS GovCloud Region, you must create the
install-config.yaml file manually.
Procedure
command:
IMPORTANT
Specify an empty directory. Some installation assets, like bootstrap X.509

certificates have short expiration intervals, so you must not reuse an
installation directory. If you want to reuse individual files from another cluster
installation, you can copy them into your directory. However, the file names
for the installation assets might change between releases. Use caution when
copying installation files from an earlier OpenShift Container Platform
version.
NOTE

ii. Select aws as the platform to target.
iii. If you do not have an AWS profile stored on your computer, enter the AWS access key
ID and secret access key for the user that you configured to run the installation
program.
NOTE
598
NOTE
program if the credentials for the exported profile are not present in the
file. Any credentials that you provide to the installation program are
stored in the file.
iv. Select the AWS Region to deploy the cluster to.
vii. Paste the pull secret from Red Hat OpenShift Cluster Manager .
2. If you are installing a three-node cluster, modify the install-config.yaml file by setting the
compute.replicas parameter to 0. This ensures that the cluster’s control planes are schedulable.
For more information, see "Installing a three-node cluster on AWS".
3. Optional: Back up the install-config.yaml file.
IMPORTANT

Prerequisites
NOTE
599
NOTE
(169.254.169.254).
Procedure
apiVersion: v1
proxy:
must be http.

NOTE
600
NOTE
NOTE
NOTE
created.
6.12.8.4. Creating the Kubernetes manifest and Ignition config files
Because you must modify some cluster definition files and manually start the cluster machines, you must
generate the Kubernetes manifest and Ignition config files that the cluster needs to configure the
machines.
The installation configuration file transforms into the Kubernetes manifests. The manifests wrap into the
Ignition configuration files, which are later used to configure the cluster machines.
IMPORTANT
The Ignition config files that the OpenShift Container Platform installation
program generates contain certificates that expire after 24 hours, which are then
renewed at that time. If the cluster is shut down before renewing the certificates
and the cluster is later restarted after the 24 hours have elapsed, the cluster
automatically recovers the expired certificates. The exception is that you must
manually approve the pending node-bootstrapper certificate signing requests
(CSRs) to recover kubelet certificates. See the documentation for Recovering
from expired control plane certificates for more information.
Prerequisites
You obtained the OpenShift Container Platform installation program.
You created the install-config.yaml installation configuration file.
601
Procedure
1. Change to the directory that contains the OpenShift Container Platform installation program
and generate the Kubernetes manifests for the cluster:
1 For <installation_directory>, specify the installation directory that contains the install-
config.yaml file you created.
2. Remove the Kubernetes manifest files that define the control plane machines:
$ rm -f <installation_directory>/openshift/99_openshift-cluster-api_master-machines-*.yaml
By removing these files, you prevent the cluster from automatically generating control plane
machines.
3. Remove the Kubernetes manifest files that define the control plane machine set:
$ rm -f <installation_directory>/openshift/99_openshift-machine-api_master-control-plane-
machine-set.yaml
$ rm -f <installation_directory>/openshift/99_openshift-cluster-api_worker-machineset-*.yaml
IMPORTANT
If you disabled the MachineAPI capability when installing a cluster on user-

provisioned infrastructure, you must remove the Kubernetes manifest files that
define the worker machines. Otherwise, your cluster fails to install.
Because you create and manage the worker machines yourself, you do not need to initialize
these machines.

WARNING
If you are installing a three-node cluster, skip the following step to allow the
control plane nodes to be schedulable.
IMPORTANT
When you configure control plane nodes from the default unschedulable to
schedulable, additional subscriptions are required. This is because control plane
nodes then become compute nodes.
4. Check that the mastersSchedulable parameter in the

<installation_directory>/manifests/cluster-scheduler-02-config.yml Kubernetes manifest
file is set to false. This setting prevents pods from being scheduled on the control plane
602
machines:
a. Open the <installation_directory>/manifests/cluster-scheduler-02-config.yml file.
b. Locate the mastersSchedulable parameter and ensure that it is set to false.
c. Save and exit the file.
5. Optional: If you do not want the Ingress Operator to create DNS records on your behalf, remove
the privateZone and publicZone sections from the
<installation_directory>/manifests/cluster-dns-02-config.yml DNS configuration file:
apiVersion: config.openshift.io/v1
kind: DNS
metadata:
name: cluster
spec:
baseDomain: example.openshift.com
privateZone: 1
id: mycluster-100419-private-zone
publicZone: 2
id: example.openshift.com
status: {}
1 2 Remove this section completely.
If you do so, you must add ingress DNS records manually in a later step.
6. To create the Ignition configuration files, run the following command from the directory that
contains the installation program:
$ ./openshift-install create ignition-configs --dir <installation_directory> 1
1 For <installation_directory>, specify the same installation directory.
Ignition config files are created for the bootstrap, control plane, and compute nodes in the
installation directory. The kubeadmin-password and kubeconfig files are created in the
./<installation_directory>/auth directory:
.
├── auth
│ ├── kubeadmin-password
│ └── kubeconfig
├── bootstrap.ign
├── master.ign
├── metadata.json
└── worker.ign
6.12.9. Extracting the infrastructure name

The Ignition config files contain a unique cluster identifier that you can use to uniquely identify your
cluster in Amazon Web Services (AWS). The infrastructure name is also used to locate the appropriate
603
AWS resources during an OpenShift Container Platform installation. The provided CloudFormation
templates contain references to this infrastructure name, so you must extract it.
Prerequisites
You obtained the OpenShift Container Platform installation program and the pull secret for
your cluster.
You generated the Ignition config files for your cluster.
You installed the jq package.
Procedure
To extract and view the infrastructure name from the Ignition config file metadata, run the
following command:
$ jq -r .infraID <installation_directory>/metadata.json 1
Example output
openshift-vw9j6 1
1 The output of this command is your cluster name and a random string.
6.12.10. Creating a VPC in AWS

You must create a Virtual Private Cloud (VPC) in Amazon Web Services (AWS) for your OpenShift
Container Platform cluster to use. You can customize the VPC to meet your requirements, including
VPN and route tables.
You can use the provided CloudFormation template and a custom parameter file to create a stack of
AWS resources that represent the VPC.
NOTE
If you do not use the provided CloudFormation template to create your AWS
infrastructure, you must review the provided information and manually create the
infrastructure. If your cluster does not initialize correctly, you might have to contact Red
Hat support with your installation logs.
Prerequisites
You added your AWS keys and region to your local AWS profile by running aws configure.
Procedure
604
Procedure
1. Create a JSON file that contains the parameter values that the template requires:
[
{
"ParameterKey": "VpcCidr", 1
"ParameterValue": "10.0.0.0/16" 2
},
{
"ParameterKey": "AvailabilityZoneCount", 3
"ParameterValue": "1" 4
},
{
"ParameterKey": "SubnetBits", 5
}
]
1 The CIDR block for the VPC.
2 Specify a CIDR block in the format x.x.x.x/16-24.
3 The number of availability zones to deploy the VPC in.
4 Specify an integer between 1 and 3.
5 The size of each subnet in each availability zone.
6 Specify an integer between 5 and 13, where 5 is /27 and 13 is /19.
2. Copy the template from the CloudFormation template for the VPCsection of this topic and
save it as a YAML file on your computer. This template describes the VPC that your cluster
requires.
3. Launch the CloudFormation template to create a stack of AWS resources that represent the
VPC:
IMPORTANT
You must enter the command on a single line.
$ aws cloudformation create-stack --stack-name <name> 1

--template-body file://<template>.yaml 2
--parameters file://<parameters>.json 3
1 <name> is the name for the CloudFormation stack, such as cluster-vpc. You need the
name of this stack if you remove the cluster.
2 <template> is the relative path to and name of the CloudFormation template YAML file
that you saved.
3 <parameters> is the relative path to and name of the CloudFormation parameters JSON
605
Example output
arn:aws:cloudformation:us-east-1:269333783861:stack/cluster-vpc/dbedae40-2fd3-11eb-
820e-12a48460849f
4. Confirm that the template components exist:
$ aws cloudformation describe-stacks --stack-name <name>
After the StackStatus displays CREATE_COMPLETE, the output displays values for the
following parameters. You must provide these parameter values to the other CloudFormation
templates that you run to create your cluster:
VpcId The ID of your VPC.
PublicSub The IDs of the new public subnets.

netIds
PrivateSu The IDs of the new private subnets.

bnetIds
6.12.10.1. CloudFormation template for the VPC
You can use the following CloudFormation template to deploy the VPC that you need for your
OpenShift Container Platform cluster.
Example 6.63. CloudFormation template for the VPC
AWSTemplateFormatVersion: 2010-09-09
Description: Template for Best Practice VPC with 1-3 AZs
Parameters:
VpcCidr:
AllowedPattern: ^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-
4][0-9]|25[0-5])(\/(1[6-9]|2[0-4]))$
ConstraintDescription: CIDR block parameter must be in the form x.x.x.x/16-24.
Default: 10.0.0.0/16
Description: CIDR block for VPC.
Type: String
AvailabilityZoneCount:
ConstraintDescription: "The number of availability zones. (Min: 1, Max: 3)"
MinValue: 1
MaxValue: 3
Default: 1
Description: "How many AZs to create VPC subnets for. (Min: 1, Max: 3)"
Type: Number
SubnetBits:
MinValue: 5
MaxValue: 13
Default: 12
Description: "Size of each subnet to create within the availability zones. (Min: 5 = /27, Max: 13 =
606
/19)"
Type: Number
Metadata:
AWS::CloudFormation::Interface:
ParameterGroups:
- Label:
default: "Network Configuration"
Parameters:
- VpcCidr
- SubnetBits
- Label:
default: "Availability Zones"
Parameters:
- AvailabilityZoneCount
ParameterLabels:
default: "Availability Zone Count"
VpcCidr:
default: "VPC CIDR"
SubnetBits:
default: "Bits Per Subnet"
Conditions:
DoAz3: !Equals [3, !Ref AvailabilityZoneCount]
DoAz2: !Or [!Equals [2, !Ref AvailabilityZoneCount], Condition: DoAz3]
Resources:
VPC:
Type: "AWS::EC2::VPC"
Properties:
EnableDnsSupport: "true"
EnableDnsHostnames: "true"
CidrBlock: !Ref VpcCidr
PublicSubnet:
Type: "AWS::EC2::Subnet"
Properties:
VpcId: !Ref VPC
CidrBlock: !Select [0, !Cidr [!Ref VpcCidr, 6, !Ref SubnetBits]]
AvailabilityZone: !Select
-0
- Fn::GetAZs: !Ref "AWS::Region"
PublicSubnet2:
Condition: DoAz2
Properties:
VpcId: !Ref VPC
-1
PublicSubnet3:
Condition: DoAz3
Properties:
VpcId: !Ref VPC
607

-2
InternetGateway:
Type: "AWS::EC2::InternetGateway"
GatewayToInternet:
Type: "AWS::EC2::VPCGatewayAttachment"
Properties:
VpcId: !Ref VPC
InternetGatewayId: !Ref InternetGateway
PublicRouteTable:
Type: "AWS::EC2::RouteTable"
Properties:
VpcId: !Ref VPC
PublicRoute:
Type: "AWS::EC2::Route"
DependsOn: GatewayToInternet
Properties:
RouteTableId: !Ref PublicRouteTable
DestinationCidrBlock: 0.0.0.0/0
GatewayId: !Ref InternetGateway
PublicSubnetRouteTableAssociation:
Type: "AWS::EC2::SubnetRouteTableAssociation"
Properties:
SubnetId: !Ref PublicSubnet
PublicSubnetRouteTableAssociation2:
Condition: DoAz2
Properties:
SubnetId: !Ref PublicSubnet2
Condition: DoAz3
Properties:
PrivateSubnet:
Properties:
VpcId: !Ref VPC
-0
PrivateRouteTable:
Properties:
VpcId: !Ref VPC
PrivateSubnetRouteTableAssociation:
Properties:
SubnetId: !Ref PrivateSubnet
RouteTableId: !Ref PrivateRouteTable
608
NAT:
DependsOn:
- GatewayToInternet
Type: "AWS::EC2::NatGateway"
Properties:
AllocationId:
"Fn::GetAtt":
- EIP
- AllocationId
EIP:
Type: "AWS::EC2::EIP"
Properties:
Domain: vpc
Route:
Properties:
RouteTableId:
Ref: PrivateRouteTable
NatGatewayId:
Ref: NAT
PrivateSubnet2:
Condition: DoAz2
Properties:
VpcId: !Ref VPC
-1
PrivateRouteTable2:
Condition: DoAz2
Properties:
VpcId: !Ref VPC
PrivateSubnetRouteTableAssociation2:
Condition: DoAz2
Properties:
SubnetId: !Ref PrivateSubnet2
RouteTableId: !Ref PrivateRouteTable2
NAT2:
DependsOn:
- GatewayToInternet
Condition: DoAz2
Properties:
AllocationId:
"Fn::GetAtt":
- EIP2
- AllocationId
EIP2:
Condition: DoAz2
609
Properties:
Domain: vpc
Route2:
Condition: DoAz2
Properties:
RouteTableId:
Ref: PrivateRouteTable2
NatGatewayId:
Ref: NAT2
PrivateSubnet3:
Condition: DoAz3
Properties:
VpcId: !Ref VPC
-2
PrivateRouteTable3:
Condition: DoAz3
Properties:
VpcId: !Ref VPC
Condition: DoAz3
Properties:
NAT3:
DependsOn:
- GatewayToInternet
Condition: DoAz3
Properties:
AllocationId:
"Fn::GetAtt":
- EIP3
- AllocationId
EIP3:
Condition: DoAz3
Properties:
Domain: vpc
Route3:
Condition: DoAz3
Properties:
RouteTableId:
NatGatewayId:
Ref: NAT3
610
S3Endpoint:
Type: AWS::EC2::VPCEndpoint
Properties:
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal: '*'
Action:
- '*'
Resource:
- '*'
RouteTableIds:
- !Ref PublicRouteTable
- !Ref PrivateRouteTable
- !If [DoAz2, !Ref PrivateRouteTable2, !Ref "AWS::NoValue"]
ServiceName: !Join
- ''
- - com.amazonaws.
- !Ref 'AWS::Region'
- .s3
VpcId: !Ref VPC
Outputs:
VpcId:
Description: ID of the new VPC.
Value: !Ref VPC
PublicSubnetIds:
Description: Subnet IDs of the public subnets.
Value:
!Join [
",",
[!Ref PublicSubnet, !If [DoAz2, !Ref PublicSubnet2, !Ref "AWS::NoValue"], !If [DoAz3, !Ref
PublicSubnet3, !Ref "AWS::NoValue"]]
]
PrivateSubnetIds:
Description: Subnet IDs of the private subnets.
Value:
!Join [
",",
[!Ref PrivateSubnet, !If [DoAz2, !Ref PrivateSubnet2, !Ref "AWS::NoValue"], !If [DoAz3, !Ref
PrivateSubnet3, !Ref "AWS::NoValue"]]
]
PublicRouteTableId:
Description: Public Route table ID
Value: !Ref PublicRouteTable
PrivateRouteTableIds:
Description: Private Route table IDs
Value:
!Join [
",",
[
!Join ["=", [
!Select [0, "Fn::GetAZs": !Ref "AWS::Region"],
!Ref PrivateRouteTable
611
]],
!If [DoAz2,
!Join ["=", [!Select [1, "Fn::GetAZs": !Ref "AWS::Region"], !Ref PrivateRouteTable2]],
!Ref "AWS::NoValue"
],
!If [DoAz3,
!Ref "AWS::NoValue"
]
]
]
You can view details about the CloudFormation stacks that you create by navigating to the AWS
CloudFormation console.
6.12.11. Creating networking and load balancing components in AWS

You must configure networking and classic or network load balancing in Amazon Web Services (AWS)
that your OpenShift Container Platform cluster can use.
AWS resources. The stack represents the networking and load balancing components that your
OpenShift Container Platform cluster requires. The template also creates a hosted zone and subnet
tags.
You can run the template multiple times within a single Virtual Private Cloud (VPC).
NOTE
Prerequisites
You created and configured a VPC and associated subnets in AWS.
Procedure
1. Obtain the hosted zone ID for the Route 53 base domain that you specified in the install-
config.yaml file for your cluster. You can obtain details about your hosted zone by running the
following command:
$ aws route53 list-hosted-zones-by-name --dns-name <route53_domain> 1
612
1 For the <route53_domain>, specify the Route 53 base domain that you used when you
generated the install-config.yaml file for the cluster.
Example output
mycluster.example.com. False 100

HOSTEDZONES 65F8F38E-2268-B835-E15C-AB55336FCBFA
/hostedzone/Z21IXYZABCZ2A4 mycluster.example.com. 10
In the example output, the hosted zone ID is Z21IXYZABCZ2A4.
[
{
"ParameterKey": "ClusterName", 1
"ParameterValue": "mycluster" 2
},
{
"ParameterKey": "InfrastructureName", 3
"ParameterValue": "mycluster-<random_string>" 4
},
{
"ParameterKey": "HostedZoneId", 5
"ParameterValue": "<random_string>" 6
},
{
"ParameterKey": "HostedZoneName", 7
"ParameterValue": "example.com" 8
},
{
"ParameterKey": "PublicSubnets", 9
"ParameterValue": "subnet-<random_string>" 10
},
{
"ParameterKey": "PrivateSubnets", 11
},
{
"ParameterKey": "VpcId", 13
"ParameterValue": "vpc-<random_string>" 14
}
]
1 A short, representative cluster name to use for hostnames, etc.
2 Specify the cluster name that you used when you generated the install-config.yaml file
for the cluster.
3 The name for your cluster infrastructure that is encoded in your Ignition config files for the
cluster.
613
4 Specify the infrastructure name that you extracted from the Ignition config file metadata,
which has the format <cluster-name>-<random-string>.
5 The Route 53 public zone ID to register the targets with.
6 Specify the Route 53 public zone ID, which as a format similar to Z21IXYZABCZ2A4. You
can obtain this value from the AWS console.
7 The Route 53 zone to register the targets with.
8 Specify the Route 53 base domain that you used when you generated the install-
config.yaml file for the cluster. Do not include the trailing period (.) that is displayed in the
AWS console.
9 The public subnets that you created for your VPC.
10 Specify the PublicSubnetIds value from the output of the CloudFormation template for
the VPC.
11 The private subnets that you created for your VPC.
12 Specify the PrivateSubnetIds value from the output of the CloudFormation template for
the VPC.
13 The VPC that you created for the cluster.
14 Specify the VpcId value from the output of the CloudFormation template for the VPC.
3. Copy the template from the CloudFormation template for the network and load balancers
section of this topic and save it as a YAML file on your computer. This template describes the
networking and load balancing objects that your cluster requires.
IMPORTANT
If you are deploying your cluster to an AWS government or secret region, you
must update the InternalApiServerRecord in the CloudFormation template to
use CNAME records. Records of type ALIAS are not supported for AWS
government regions.
4. Launch the CloudFormation template to create a stack of AWS resources that provide the
networking and load balancing components:
IMPORTANT

--capabilities CAPABILITY_NAMED_IAM 4
1 <name> is the name for the CloudFormation stack, such as cluster-dns. You need the
614
that you saved.
file.
4 You must explicitly declare the CAPABILITY_NAMED_IAM capability because the

provided template creates some AWS::IAM::Role resources.
Example output
arn:aws:cloudformation:us-east-1:269333783861:stack/cluster-dns/cd3e5de0-2fd4-11eb-
5cf0-12be5c33a183
PrivateHo Hosted zone ID for the private DNS.

stedZoneI
d
ExternalA Full name of the external API load balancer.

piLoadBal
ancerNam
e
InternalAp Full name of the internal API load balancer.

iLoadBala
ncerName
ApiServer Full hostname of the API server.

DnsName
RegisterN Lambda ARN useful to help register/deregister IP targets for these load balancers.
lbIpTarget
sLambda
ExternalA ARN of external API target group.

piTargetG
roupArn
InternalAp ARN of internal API target group.

iTargetGr
oupArn
615
InternalSe ARN of internal service target group.

rviceTarg
etGroupA
rn
6.12.11.1. CloudFormation template for the network and load balancers
You can use the following CloudFormation template to deploy the networking objects and load
balancers that you need for your OpenShift Container Platform cluster.
Example 6.64. CloudFormation template for the network and load balancers
Description: Template for OpenShift Cluster Network Elements (Route53 & LBs)
Parameters:
ClusterName:
AllowedPattern: ^([a-zA-Z][a-zA-Z0-9\-]{0,26})$
MaxLength: 27
MinLength: 1
ConstraintDescription: Cluster name must be alphanumeric, start with a letter, and have a
maximum of 27 characters.
Description: A short, representative cluster name to use for host names and other identifying
names.
Type: String
InfrastructureName:
MaxLength: 27
MinLength: 1
ConstraintDescription: Infrastructure name must be alphanumeric, start with a letter, and have a
Description: A short, unique cluster ID used to tag cloud resources and identify items owned or
used by the cluster.
Type: String
HostedZoneId:
Description: The Route53 public zone ID to register the targets with, such as
Z21IXYZABCZ2A4.
Type: String
HostedZoneName:
Description: The Route53 zone to register the targets with, such as example.com. Omit the
trailing period.
Type: String
Default: "example.com"
PublicSubnets:
Description: The internet-facing subnets.
Type: List<AWS::EC2::Subnet::Id>
PrivateSubnets:
Description: The internal subnets.
VpcId:
Description: The VPC-scoped resources will belong to this VPC.
Type: AWS::EC2::VPC::Id
Metadata:
616
ParameterGroups:
- Label:
default: "Cluster Information"
Parameters:
- ClusterName
- InfrastructureName
- Label:
Parameters:
- VpcId
- PublicSubnets
- PrivateSubnets
- Label:
default: "DNS"
Parameters:
- HostedZoneName
- HostedZoneId
ParameterLabels:
ClusterName:
default: "Cluster Name"
InfrastructureName:
default: "Infrastructure Name"
VpcId:
default: "VPC ID"
PublicSubnets:
default: "Public Subnets"
PrivateSubnets:
default: "Private Subnets"
HostedZoneName:
default: "Public Hosted Zone Name"
HostedZoneId:
default: "Public Hosted Zone ID"
Resources:
ExtApiElb:
Type: AWS::ElasticLoadBalancingV2::LoadBalancer
Properties:
Name: !Join ["-", [!Ref InfrastructureName, "ext"]]
IpAddressType: ipv4
Subnets: !Ref PublicSubnets
Type: network
IntApiElb:
Properties:
Name: !Join ["-", [!Ref InfrastructureName, "int"]]
Scheme: internal
IpAddressType: ipv4
Subnets: !Ref PrivateSubnets
Type: network
IntDns:
Type: "AWS::Route53::HostedZone"
Properties:
HostedZoneConfig:
617
Comment: "Managed by CloudFormation"

Name: !Join [".", [!Ref ClusterName, !Ref HostedZoneName]]
HostedZoneTags:
- Key: Name
Value: !Join ["-", [!Ref InfrastructureName, "int"]]
- Key: !Join ["", ["kubernetes.io/cluster/", !Ref InfrastructureName]]
Value: "owned"
VPCs:
- VPCId: !Ref VpcId
VPCRegion: !Ref "AWS::Region"
ExternalApiServerRecord:
Type: AWS::Route53::RecordSetGroup
Properties:
Comment: Alias record for the API server
HostedZoneId: !Ref HostedZoneId
RecordSets:
- Name:
!Join [
".",
["api", !Ref ClusterName, !Join ["", [!Ref HostedZoneName, "."]]],
]
Type: A
AliasTarget:
HostedZoneId: !GetAtt ExtApiElb.CanonicalHostedZoneID
DNSName: !GetAtt ExtApiElb.DNSName
InternalApiServerRecord:
Properties:
HostedZoneId: !Ref IntDns
RecordSets:
- Name:
!Join [
".",
]
Type: A
AliasTarget:
HostedZoneId: !GetAtt IntApiElb.CanonicalHostedZoneID
DNSName: !GetAtt IntApiElb.DNSName
- Name:
!Join [
".",
["api-int", !Ref ClusterName, !Join ["", [!Ref HostedZoneName, "."]]],
]
Type: A
AliasTarget:
ExternalApiListener:
Type: AWS::ElasticLoadBalancingV2::Listener
Properties:
DefaultActions:
618
- Type: forward
TargetGroupArn:
Ref: ExternalApiTargetGroup
LoadBalancerArn:
Ref: ExtApiElb
Port: 6443
Protocol: TCP
ExternalApiTargetGroup:
Type: AWS::ElasticLoadBalancingV2::TargetGroup
Properties:
HealthCheckIntervalSeconds: 10
HealthCheckPath: "/readyz"
HealthCheckPort: 6443
HealthCheckProtocol: HTTPS
HealthyThresholdCount: 2
UnhealthyThresholdCount: 2
Port: 6443
Protocol: TCP
TargetType: ip
VpcId:
Ref: VpcId
TargetGroupAttributes:
- Key: deregistration_delay.timeout_seconds
Value: 60
InternalApiListener:
Properties:
DefaultActions:
- Type: forward
TargetGroupArn:
Ref: InternalApiTargetGroup
LoadBalancerArn:
Ref: IntApiElb
Port: 6443
Protocol: TCP
InternalApiTargetGroup:
Properties:
Port: 6443
Protocol: TCP
TargetType: ip
VpcId:
Ref: VpcId
Value: 60
619
InternalServiceInternalListener:
Properties:
DefaultActions:
- Type: forward
TargetGroupArn:
Ref: InternalServiceTargetGroup
LoadBalancerArn:
Ref: IntApiElb
Port: 22623
Protocol: TCP
InternalServiceTargetGroup:
Properties:
HealthCheckPath: "/healthz"
Port: 22623
Protocol: TCP
TargetType: ip
VpcId:
Ref: VpcId
Value: 60
RegisterTargetLambdaIamRole:
Type: AWS::IAM::Role
Properties:
RoleName: !Join ["-", [!Ref InfrastructureName, "nlb", "lambda", "role"]]
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: "Allow"
Principal:
Service:
- "lambda.amazonaws.com"
Action:
- "sts:AssumeRole"
Path: "/"
Policies:
- PolicyName: !Join ["-", [!Ref InfrastructureName, "master", "policy"]]
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: "Allow"
Action:
[
"elasticloadbalancing:RegisterTargets",
"elasticloadbalancing:DeregisterTargets",
]
Resource: !Ref InternalApiTargetGroup
620
- Effect: "Allow"
Action:
[
]
Resource: !Ref InternalServiceTargetGroup
- Effect: "Allow"
Action:
[
]
Resource: !Ref ExternalApiTargetGroup
RegisterNlbIpTargets:
Type: "AWS::Lambda::Function"
Properties:
Handler: "index.handler"
Role:
Fn::GetAtt:
- "RegisterTargetLambdaIamRole"
- "Arn"
Code:
ZipFile: |
import json
import boto3
import cfnresponse
def handler(event, context):
elb = boto3.client('elbv2')
if event['RequestType'] == 'Delete':
elb.deregister_targets(TargetGroupArn=event['ResourceProperties']
['TargetArn'],Targets=[{'Id': event['ResourceProperties']['TargetIp']}])
elif event['RequestType'] == 'Create':
elb.register_targets(TargetGroupArn=event['ResourceProperties']['TargetArn'],Targets=
[{'Id': event['ResourceProperties']['TargetIp']}])
responseData = {}
cfnresponse.send(event, context, cfnresponse.SUCCESS, responseData,
event['ResourceProperties']['TargetArn']+event['ResourceProperties']['TargetIp'])
Runtime: "python3.8"
Timeout: 120
RegisterSubnetTagsLambdaIamRole:
Properties:
RoleName: !Join ["-", [!Ref InfrastructureName, "subnet-tags-lambda-role"]]
Version: "2012-10-17"
Statement:
- Effect: "Allow"
Principal:
Service:
Action:
- "sts:AssumeRole"
Path: "/"
621
Policies:
- PolicyName: !Join ["-", [!Ref InfrastructureName, "subnet-tagging-policy"]]
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: "Allow"
Action:
[
"ec2:DeleteTags",
"ec2:CreateTags"
]
Resource: "arn:aws:ec2:*:*:subnet/*"
- Effect: "Allow"
Action:
[
"ec2:DescribeSubnets",
"ec2:DescribeTags"
]
Resource: "*"
RegisterSubnetTags:
Properties:
Role:
Fn::GetAtt:
- "RegisterSubnetTagsLambdaIamRole"
- "Arn"
Code:
ZipFile: |
import json
import boto3
import cfnresponse
ec2_client = boto3.client('ec2')
for subnet_id in event['ResourceProperties']['Subnets']:
ec2_client.delete_tags(Resources=[subnet_id], Tags=[{'Key': 'kubernetes.io/cluster/' +
event['ResourceProperties']['InfrastructureName']}]);
ec2_client.create_tags(Resources=[subnet_id], Tags=[{'Key': 'kubernetes.io/cluster/' +
event['ResourceProperties']['InfrastructureName'], 'Value': 'shared'}]);
responseData = {}
event['ResourceProperties']['InfrastructureName']+event['ResourceProperties']['Subnets'][0])
Timeout: 120
RegisterPublicSubnetTags:
Type: Custom::SubnetRegister
Properties:
ServiceToken: !GetAtt RegisterSubnetTags.Arn
InfrastructureName: !Ref InfrastructureName
622
RegisterPrivateSubnetTags:
Properties:
Outputs:
PrivateHostedZoneId:
Description: Hosted zone ID for the private DNS, which is required for private records.
Value: !Ref IntDns
ExternalApiLoadBalancerName:
Description: Full name of the external API load balancer.
Value: !GetAtt ExtApiElb.LoadBalancerFullName
InternalApiLoadBalancerName:
Description: Full name of the internal API load balancer.
Value: !GetAtt IntApiElb.LoadBalancerFullName
ApiServerDnsName:
Description: Full hostname of the API server, which is required for the Ignition config files.
Value: !Join [".", ["api-int", !Ref ClusterName, !Ref HostedZoneName]]
RegisterNlbIpTargetsLambda:
Description: Lambda ARN useful to help register or deregister IP targets for these load
balancers.
Value: !GetAtt RegisterNlbIpTargets.Arn
ExternalApiTargetGroupArn:
Description: ARN of the external API target group.
Value: !Ref ExternalApiTargetGroup
InternalApiTargetGroupArn:
Description: ARN of the internal API target group.
Value: !Ref InternalApiTargetGroup
InternalServiceTargetGroupArn:
Description: ARN of the internal service target group.
Value: !Ref InternalServiceTargetGroup
IMPORTANT
If you are deploying your cluster to an AWS government or secret region, you must
update the InternalApiServerRecord to use CNAME records. Records of type ALIAS
are not supported for AWS government regions. For example:
Type: CNAME
TTL: 10
ResourceRecords:
- !GetAtt IntApiElb.DNSName
You can view details about your hosted zones by navigating to the AWS Route 53 console .
See Listing public hosted zones in the AWS documentation for more information about listing
623
public hosted zones.
6.12.12. Creating security group and roles in AWS

You must create security groups and roles in Amazon Web Services (AWS) for your OpenShift Container
Platform cluster to use.
AWS resources. The stack represents the security groups and roles that your OpenShift Container
Platform cluster requires.
NOTE
Prerequisites
Procedure
[
{
},
{
},
{
},
{
}
]
cluster.
624
4 Specify the CIDR block parameter that you used for the VPC that you defined in the form
x.x.x.x/16-24.
the VPC.
2. Copy the template from the CloudFormation template for security objectssection of this
topic and save it as a YAML file on your computer. This template describes the security groups
and roles that your cluster requires.
security groups and roles:
IMPORTANT

1 <name> is the name for the CloudFormation stack, such as cluster-sec. You need the
that you saved.
file.

provided template creates some AWS::IAM::Role and AWS::IAM::InstanceProfile
resources.
Example output
arn:aws:cloudformation:us-east-1:269333783861:stack/cluster-sec/03bd4210-2ed7-11eb-
6d7a-13fc0b61e9db
625
MasterSec Master Security Group ID

urityGrou
pId
WorkerSe Worker Security Group ID

curityGro
upId
MasterIns Master IAM Instance Profile

tanceProfi
le
WorkerIns Worker IAM Instance Profile

tanceProfi
le
6.12.12.1. CloudFormation template for security objects
You can use the following CloudFormation template to deploy the security objects that you need for
your OpenShift Container Platform cluster.
Example 6.65. CloudFormation template for security objects
Description: Template for OpenShift Cluster Security Elements (Security Groups & IAM)
Parameters:
InfrastructureName:
MaxLength: 27
MinLength: 1
Type: String
VpcCidr:
AllowedPattern: ^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-
4][0-9]|25[0-5])(\/(1[6-9]|2[0-4]))$
Default: 10.0.0.0/16
Type: String
VpcId:
PrivateSubnets:
626

Metadata:
ParameterGroups:
- Label:
Parameters:
- Label:
Parameters:
- VpcId
- VpcCidr
- PrivateSubnets
ParameterLabels:
InfrastructureName:
VpcId:
default: "VPC ID"
VpcCidr:
default: "VPC CIDR"
PrivateSubnets:
Resources:
MasterSecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: Cluster Master Security Group
SecurityGroupIngress:
- IpProtocol: icmp
FromPort: 0
ToPort: 0
CidrIp: !Ref VpcCidr
- IpProtocol: tcp
FromPort: 22
ToPort: 22
- IpProtocol: tcp
ToPort: 6443
FromPort: 6443
- IpProtocol: tcp
FromPort: 22623
ToPort: 22623
VpcId: !Ref VpcId
WorkerSecurityGroup:
Properties:
GroupDescription: Cluster Worker Security Group
- IpProtocol: icmp
627
FromPort: 0
ToPort: 0
- IpProtocol: tcp
FromPort: 22
ToPort: 22
VpcId: !Ref VpcId
MasterIngressEtcd:
Type: AWS::EC2::SecurityGroupIngress
Properties:
GroupId: !GetAtt MasterSecurityGroup.GroupId
SourceSecurityGroupId: !GetAtt MasterSecurityGroup.GroupId
Description: etcd
FromPort: 2379
ToPort: 2380
IpProtocol: tcp
MasterIngressVxlan:
Properties:
Description: Vxlan packets
FromPort: 4789
ToPort: 4789
IpProtocol: udp
MasterIngressWorkerVxlan:
Properties:
SourceSecurityGroupId: !GetAtt WorkerSecurityGroup.GroupId
FromPort: 4789
ToPort: 4789
IpProtocol: udp
MasterIngressGeneve:
Properties:
Description: Geneve packets
FromPort: 6081
ToPort: 6081
IpProtocol: udp
MasterIngressWorkerGeneve:
Properties:
FromPort: 6081
628
ToPort: 6081
IpProtocol: udp
MasterIngressIpsecIke:
Properties:
Description: IPsec IKE packets
FromPort: 500
ToPort: 500
IpProtocol: udp
MasterIngressIpsecNat:
Properties:
Description: IPsec NAT-T packets
FromPort: 4500
ToPort: 4500
IpProtocol: udp
MasterIngressIpsecEsp:
Properties:
Description: IPsec ESP packets
IpProtocol: 50
MasterIngressWorkerIpsecIke:
Properties:
FromPort: 500
ToPort: 500
IpProtocol: udp
MasterIngressWorkerIpsecNat:
Properties:
FromPort: 4500
ToPort: 4500
IpProtocol: udp
MasterIngressWorkerIpsecEsp:
Properties:
629

IpProtocol: 50
MasterIngressInternal:
Properties:
Description: Internal cluster communication
FromPort: 9000
ToPort: 9999
IpProtocol: tcp
MasterIngressWorkerInternal:
Properties:
FromPort: 9000
ToPort: 9999
IpProtocol: tcp
MasterIngressInternalUDP:
Properties:
FromPort: 9000
ToPort: 9999
IpProtocol: udp
MasterIngressWorkerInternalUDP:
Properties:
FromPort: 9000
ToPort: 9999
IpProtocol: udp
MasterIngressKube:
Properties:
Description: Kubernetes kubelet, scheduler and controller manager
FromPort: 10250
ToPort: 10259
IpProtocol: tcp
MasterIngressWorkerKube:
Properties:
630

FromPort: 10250
ToPort: 10259
IpProtocol: tcp
MasterIngressIngressServices:
Properties:
Description: Kubernetes ingress services
FromPort: 30000
ToPort: 32767
IpProtocol: tcp
MasterIngressWorkerIngressServices:
Properties:
FromPort: 30000
ToPort: 32767
IpProtocol: tcp
MasterIngressIngressServicesUDP:
Properties:
FromPort: 30000
ToPort: 32767
IpProtocol: udp
MasterIngressWorkerIngressServicesUDP:
Properties:
FromPort: 30000
ToPort: 32767
IpProtocol: udp
WorkerIngressVxlan:
Properties:
GroupId: !GetAtt WorkerSecurityGroup.GroupId
FromPort: 4789
ToPort: 4789
IpProtocol: udp
631
WorkerIngressMasterVxlan:
Properties:
FromPort: 4789
ToPort: 4789
IpProtocol: udp
WorkerIngressGeneve:
Properties:
FromPort: 6081
ToPort: 6081
IpProtocol: udp
WorkerIngressMasterGeneve:
Properties:
FromPort: 6081
ToPort: 6081
IpProtocol: udp
WorkerIngressIpsecIke:
Properties:
FromPort: 500
ToPort: 500
IpProtocol: udp
WorkerIngressIpsecNat:
Properties:
FromPort: 4500
ToPort: 4500
IpProtocol: udp
WorkerIngressIpsecEsp:
Properties:
632

IpProtocol: 50
WorkerIngressMasterIpsecIke:
Properties:
FromPort: 500
ToPort: 500
IpProtocol: udp
WorkerIngressMasterIpsecNat:
Properties:
FromPort: 4500
ToPort: 4500
IpProtocol: udp
WorkerIngressMasterIpsecEsp:
Properties:
IpProtocol: 50
WorkerIngressInternal:
Properties:
FromPort: 9000
ToPort: 9999
IpProtocol: tcp
WorkerIngressMasterInternal:
Properties:
FromPort: 9000
ToPort: 9999
IpProtocol: tcp
WorkerIngressInternalUDP:
Properties:
633

FromPort: 9000
ToPort: 9999
IpProtocol: udp
WorkerIngressMasterInternalUDP:
Properties:
FromPort: 9000
ToPort: 9999
IpProtocol: udp
WorkerIngressKube:
Properties:
Description: Kubernetes secure kubelet port
FromPort: 10250
ToPort: 10250
IpProtocol: tcp
WorkerIngressWorkerKube:
Properties:
Description: Internal Kubernetes communication
FromPort: 10250
ToPort: 10250
IpProtocol: tcp
WorkerIngressIngressServices:
Properties:
FromPort: 30000
ToPort: 32767
IpProtocol: tcp
WorkerIngressMasterIngressServices:
Properties:
FromPort: 30000
ToPort: 32767
IpProtocol: tcp
WorkerIngressIngressServicesUDP:
634
Properties:
FromPort: 30000
ToPort: 32767
IpProtocol: udp
WorkerIngressMasterIngressServicesUDP:
Properties:
FromPort: 30000
ToPort: 32767
IpProtocol: udp
MasterIamRole:
Properties:
Version: "2012-10-17"
Statement:
- Effect: "Allow"
Principal:
Service:
- "ec2.amazonaws.com"
Action:
- "sts:AssumeRole"
Policies:
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: "Allow"
Action:
- "ec2:AttachVolume"
- "ec2:AuthorizeSecurityGroupIngress"
- "ec2:CreateSecurityGroup"
- "ec2:CreateTags"
- "ec2:CreateVolume"
- "ec2:DeleteSecurityGroup"
- "ec2:DeleteVolume"
- "ec2:Describe*"
- "ec2:DetachVolume"
- "ec2:ModifyInstanceAttribute"
- "ec2:ModifyVolume"
- "ec2:RevokeSecurityGroupIngress"
- "elasticloadbalancing:AddTags"
- "elasticloadbalancing:AttachLoadBalancerToSubnets"
- "elasticloadbalancing:ApplySecurityGroupsToLoadBalancer"
- "elasticloadbalancing:CreateListener"
- "elasticloadbalancing:CreateLoadBalancer"
- "elasticloadbalancing:CreateLoadBalancerPolicy"
635
- "elasticloadbalancing:CreateLoadBalancerListeners"
- "elasticloadbalancing:CreateTargetGroup"
- "elasticloadbalancing:ConfigureHealthCheck"
- "elasticloadbalancing:DeleteListener"
- "elasticloadbalancing:DeleteLoadBalancer"
- "elasticloadbalancing:DeleteLoadBalancerListeners"
- "elasticloadbalancing:DeleteTargetGroup"
- "elasticloadbalancing:DeregisterInstancesFromLoadBalancer"
- "elasticloadbalancing:DeregisterTargets"
- "elasticloadbalancing:Describe*"
- "elasticloadbalancing:DetachLoadBalancerFromSubnets"
- "elasticloadbalancing:ModifyListener"
- "elasticloadbalancing:ModifyLoadBalancerAttributes"
- "elasticloadbalancing:ModifyTargetGroup"
- "elasticloadbalancing:ModifyTargetGroupAttributes"
- "elasticloadbalancing:RegisterInstancesWithLoadBalancer"
- "elasticloadbalancing:RegisterTargets"
- "elasticloadbalancing:SetLoadBalancerPoliciesForBackendServer"
- "elasticloadbalancing:SetLoadBalancerPoliciesOfListener"
- "kms:DescribeKey"
Resource: "*"
MasterInstanceProfile:
Type: "AWS::IAM::InstanceProfile"
Properties:
Roles:
- Ref: "MasterIamRole"
WorkerIamRole:
Properties:
Version: "2012-10-17"
Statement:
- Effect: "Allow"
Principal:
Service:
Action:
- "sts:AssumeRole"
Policies:
- PolicyName: !Join ["-", [!Ref InfrastructureName, "worker", "policy"]]
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: "Allow"
Action:
- "ec2:DescribeInstances"
- "ec2:DescribeRegions"
Resource: "*"
WorkerInstanceProfile:
Properties:
Roles:
- Ref: "WorkerIamRole"
636
Outputs:
MasterSecurityGroupId:
Description: Master Security Group ID
Value: !GetAtt MasterSecurityGroup.GroupId
WorkerSecurityGroupId:
Description: Worker Security Group ID
Value: !GetAtt WorkerSecurityGroup.GroupId
Description: Master IAM Instance Profile
Value: !Ref MasterInstanceProfile
Description: Worker IAM Instance Profile
Value: !Ref WorkerInstanceProfile
6.12.13. Accessing RHCOS AMIs with stream metadata

In OpenShift Container Platform, stream metadata provides standardized metadata about RHCOS in
the JSON format and injects the metadata into the cluster. Stream metadata is a stable format that
supports multiple architectures and is intended to be self-documenting for maintaining automation.
You can use the coreos print-stream-json sub-command of openshift-install to access information
about the boot images in the stream metadata format. This command provides a method for printing
stream metadata in a scriptable, machine-readable format.
For user-provisioned installations, the openshift-install binary contains references to the version of
RHCOS boot images that are tested for use with OpenShift Container Platform, such as the AWS AMI.
Procedure
To parse the stream metadata, use one of the following methods:
From a Go program, use the official stream-metadata-go library at

https://github.com/coreos/stream-metadata-go. You can also view example code in the library.
From another programming language, such as Python or Ruby, use the JSON library of your
preferred programming language.
From a command-line utility that handles JSON data, such as jq:
Print the current x86_64 or aarch64 AMI for an AWS region, such as us-west-1:
For x86_64
$ openshift-install coreos print-stream-json | jq -r

'.architectures.x86_64.images.aws.regions["us-west-1"].image'
637
Example output
ami-0d3e625f84626bbda
For aarch64

'.architectures.aarch64.images.aws.regions["us-west-1"].image'
Example output
ami-0af1d3b7fa5be2131
The output of this command is the AWS AMI ID for your designated architecture and the us-
west-1 region. The AMI must belong to the same region as the cluster.
6.12.14. RHCOS AMIs for the AWS infrastructure

Red Hat provides Red Hat Enterprise Linux CoreOS (RHCOS) AMIs that are valid for the various AWS
regions and instance architectures that you can manually specify for your OpenShift Container Platform
nodes.
NOTE
By importing your own AMI, you can also install to regions that do not have a published
RHCOS AMI.
Table 6.17. x86_64 RHCOS AMIs
AWS zone AWS AMI
af-south-1 ami-0493ec0f0a451f83b
ap-east-1 ami-050a6d164705e7f62
ap-northeast-1 ami-00910c337e0f52cff
ap-northeast-2 ami-07e98d33de2b93ac0
ap-northeast-3 ami-09bc0a599f4b3c483
ap-south-1 ami-0ba603a7f9d41228e
ap-south-2 ami-03130aecb5d7459cc
ap-southeast-1 ami-026c056e0a25e5a04
ap-southeast-2 ami-0d471f504ff6d9a0f
638
AWS zone AWS AMI
ap-southeast-3 ami-0c1b9a0721cbb3291
ap-southeast-4 ami-0ef23bfe787efe11e
ca-central-1 ami-0163965a05b75f976
eu-central-1 ami-01edb54011f870f0c
eu-central-2 ami-0bc500d6056a3b104
eu-north-1 ami-0ab155e935177f16a
eu-south-1 ami-051b4c06b21f5a328
eu-south-2 ami-096644e5555c23b19
eu-west-1 ami-0faeeeb3d2b1aa07c
eu-west-2 ami-00bb1522dc71b604f
eu-west-3 ami-01e5397bd2b795bd3
il-central-1 ami-0b32feb5d77c64e61
me-central-1 ami-0a5158a3e68ab7e88
me-south-1 ami-024864ad1b799dbba
sa-east-1 ami-0c402ffb0c4b7edc0
us-east-1 ami-057df4d0cb8cbae0d
us-east-2 ami-07566e5da1fd297f8
us-gov-east-1 ami-0fe03a7e289354670
us-gov-west-1 ami-06b7cc6445c5da732
us-west-1 ami-02d20001c5b9df1e9
us-west-2 ami-0dfba457127fba98c
Table 6.18. aarch64 RHCOS AMIs
639
AWS zone AWS AMI
af-south-1 ami-06c7b4e42179544df
ap-east-1 ami-07b6a37fa6d2d2e99
ap-northeast-1 ami-056d2eef4a3638246
ap-northeast-2 ami-0bd5a7684f0ff4e02
ap-northeast-3 ami-0fd08063da50de1da
ap-south-1 ami-08f1ae2cef8f9690e
ap-south-2 ami-020ba25cc1ec53b1c
ap-southeast-1 ami-0020a1c0964ac8e48
ap-southeast-2 ami-07013a63289350c3c
ap-southeast-3 ami-041d6ca1d57e3190f
ap-southeast-4 ami-06539e9cbefc28702
ca-central-1 ami-0bb3991641f2b40f6
eu-central-1 ami-0908d117c26059e39
eu-central-2 ami-0e48c82ffbde67ed2
eu-north-1 ami-016614599b38d515e
eu-south-1 ami-01b6cc1f0fd7b431f
eu-south-2 ami-0687e1d98e55e402d
eu-west-1 ami-0bf0b7b1cb052d68d
eu-west-2 ami-0ba0bf567caa63731
eu-west-3 ami-0eab6a7956a66deda
il-central-1 ami-03b3cb1f4869bf21d
me-central-1 ami-0a6e1ade3c9e206a1
me-south-1 ami-0aa0775c68eac9f6f
640
AWS zone AWS AMI
sa-east-1 ami-07235eee0bb930c78
us-east-1 ami-005808ca73e7b36ff
us-east-2 ami-0c5c9420f6b992e9e
us-gov-east-1 ami-08c9b2b8d578caf92
us-gov-west-1 ami-0bdff65422ba7d95d
us-west-1 ami-017ad4dd030a04233
us-west-2 ami-068d0af5e3c08e618
6.12.14.1. AWS regions without a published RHCOS AMI
You can deploy an OpenShift Container Platform cluster to Amazon Web Services (AWS) regions
without native support for a Red Hat Enterprise Linux CoreOS (RHCOS) Amazon Machine Image (AMI)
or the AWS software development kit (SDK). If a published AMI is not available for an AWS region, you
can upload a custom AMI prior to installing the cluster.
If you are deploying to a region not supported by the AWS SDK and you do not specify a custom AMI,
the installation program copies the us-east-1 AMI to the user account automatically. Then the
installation program creates the control plane machines with encrypted EBS volumes using the default
or user-specified Key Management Service (KMS) key. This allows the AMI to follow the same process
workflow as published RHCOS AMIs.
A region without native support for an RHCOS AMI is not available to select from the terminal during
cluster creation because it is not published. However, you can install to this region by configuring the
custom AMI in the install-config.yaml file.
6.12.14.2. Uploading a custom RHCOS AMI in AWS
Prerequisites
installing.
641
Procedure
1 1 1 The RHCOS VMDK version, like 4.15.0.

{
"Format": "vmdk",
"UserBucket": {
}
}
EOF


x86_64-aws.x86_64.
Example output
642
{
{
"Format": "VMDK",
"SnapshotId": "snap-06331325870076318",
"UserBucket": {
}
}
}
]
}

--ena-support \
EBS-backed AMIs.
6.12.15. Creating the bootstrap node in AWS

You must create the bootstrap node in Amazon Web Services (AWS) to use during OpenShift Container
Platform cluster initialization. You do this by:
Providing a location to serve the bootstrap.ign Ignition config file to your cluster. This file is
located in your installation directory. The provided CloudFormation Template assumes that the
Ignition config files for your cluster are served from an S3 bucket. If you choose to serve the
643
files from another location, you must modify the templates.
Using the provided CloudFormation template and a custom parameter file to create a stack of
AWS resources. The stack represents the bootstrap node that your OpenShift Container
Platform installation requires.
NOTE
If you do not use the provided CloudFormation template to create your bootstrap node,
you must review the provided information and manually create the infrastructure. If your
cluster does not initialize correctly, you might have to contact Red Hat support with your
installation logs.
Prerequisites
You created and configured DNS, load balancers, and listeners in AWS.
You created the security groups and roles required for your cluster in AWS.
Procedure
1. Create the bucket by running the following command:
$ aws s3 mb s3://<cluster-name>-infra 1
1 <cluster-name>-infra is the bucket name. When creating the install-config.yaml file,

replace <cluster-name> with the name specified for the cluster.
You must use a presigned URL for your S3 bucket, instead of the s3:// schema, if you are:
Deploying to a region that has endpoints that differ from the AWS SDK.
Deploying a proxy.
Providing your own custom endpoints.
2. Upload the bootstrap.ign Ignition config file to the bucket by running the following command:
$ aws s3 cp <installation_directory>/bootstrap.ign s3://<cluster-name>-infra/bootstrap.ign 1
3. Verify that the file uploaded by running the following command:
644
$ aws s3 ls s3://<cluster-name>-infra/
Example output
2019-04-03 16:15:16 314878 bootstrap.ign
NOTE
The bootstrap Ignition config file does contain secrets, like X.509 keys. The
following steps provide basic security for the S3 bucket. To provide additional
security, you can enable an S3 bucket policy to allow only certain users, such as
the OpenShift IAM user, to access objects that the bucket contains. You can
avoid S3 entirely and serve your bootstrap Ignition config file from any address
that the bootstrap machine can reach.
[
{
},
{
"ParameterKey": "RhcosAmi", 3
"ParameterValue": "ami-<random_string>" 4
},
{
"ParameterKey": "AllowedBootstrapSshCidr", 5
},
{
"ParameterKey": "PublicSubnet", 7
},
{
"ParameterKey": "MasterSecurityGroupId", 9
"ParameterValue": "sg-<random_string>" 10
},
{
},
{
"ParameterKey": "BootstrapIgnitionLocation", 13
"ParameterValue": "s3://<bucket_name>/bootstrap.ign" 14
},
{
"ParameterKey": "AutoRegisterELB", 15
"ParameterValue": "yes" 16
},
{
"ParameterKey": "RegisterNlbIpTargetsLambdaArn", 17
645
"ParameterValue": "arn:aws:lambda:<aws_region>:<account_number>:function:
<dns_stack_name>-RegisterNlbIpTargets-<random_string>" 18
},
{
"ParameterKey": "ExternalApiTargetGroupArn", 19
"ParameterValue": "arn:aws:elasticloadbalancing:<aws_region>:
<account_number>:targetgroup/<dns_stack_name>-Exter-<random_string>" 20
},
{
"ParameterKey": "InternalApiTargetGroupArn", 21
<account_number>:targetgroup/<dns_stack_name>-Inter-<random_string>" 22
},
{
"ParameterKey": "InternalServiceTargetGroupArn", 23
}
]
cluster.
3 Current Red Hat Enterprise Linux CoreOS (RHCOS) AMI to use for the bootstrap node
based on your selected architecture.
4 Specify a valid AWS::EC2::Image::Id value.
5 CIDR block to allow SSH access to the bootstrap node.
7 The public subnet that is associated with your VPC to launch the bootstrap node into.
the VPC.
9 The master security group ID (for registering temporary rules)
10 Specify the MasterSecurityGroupId value from the output of the CloudFormation

template for the security group and roles.
11 The VPC created resources will belong to.
13 Location to fetch bootstrap Ignition config file from.
14 Specify the S3 bucket and file name in the form s3://<bucket_name>/bootstrap.ign.
15 Whether or not to register a network load balancer (NLB).
16 Specify yes or no. If you specify yes, you must provide a Lambda Amazon Resource Name
(ARN) value.
646
(ARN) value.
17 The ARN for NLB IP target registration lambda group.
18 Specify the RegisterNlbIpTargetsLambda value from the output of the CloudFormation

template for DNS and load balancing. Use arn:aws-us-gov if deploying the cluster to an
AWS GovCloud region.
19 The ARN for external API load balancer target group.
20 Specify the ExternalApiTargetGroupArn value from the output of the CloudFormation

21 The ARN for internal API load balancer target group.
22 Specify the InternalApiTargetGroupArn value from the output of the CloudFormation

23 The ARN for internal service load balancer target group.
24 Specify the InternalServiceTargetGroupArn value from the output of the

CloudFormation template for DNS and load balancing. Use arn:aws-us-gov if deploying
the cluster to an AWS GovCloud region.
5. Copy the template from the CloudFormation template for the bootstrap machinesection of
this topic and save it as a YAML file on your computer. This template describes the bootstrap
machine that your cluster requires.
6. Optional: If you are deploying the cluster with a proxy, you must update the ignition in the
template to add the ignition.config.proxy fields. Additionally, If you have added the Amazon
EC2, Elastic Load Balancing, and S3 VPC endpoints to your VPC, you must add these endpoints
to the noProxy field.
bootstrap node:
IMPORTANT

1 <name> is the name for the CloudFormation stack, such as cluster-bootstrap. You need
the name of this stack if you remove the cluster.
that you saved.
file.
647

Example output
arn:aws:cloudformation:us-east-1:269333783861:stack/cluster-bootstrap/12944486-2add-
11eb-9dee-12dace8e3a83
Bootstrap The bootstrap Instance ID.

InstanceId
Bootstrap The bootstrap node public IP address.

PublicIp
Bootstrap The bootstrap node private IP address.

PrivateIp
6.12.15.1. CloudFormation template for the bootstrap machine
You can use the following CloudFormation template to deploy the bootstrap machine that you need for
Example 6.66. CloudFormation template for the bootstrap machine
Description: Template for OpenShift Cluster Bootstrap (EC2 Instance, Security Groups and IAM)
Parameters:
InfrastructureName:
MaxLength: 27
MinLength: 1
Type: String
RhcosAmi:
Description: Current Red Hat Enterprise Linux CoreOS AMI to use for bootstrap.
Type: AWS::EC2::Image::Id
AllowedBootstrapSshCidr:
AllowedPattern: ^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-
4][0-9]|25[0-5])(\/([0-9]|1[0-9]|2[0-9]|3[0-2]))$
648
Default: 0.0.0.0/0
Description: CIDR block to allow SSH access to the bootstrap node.
Type: String
PublicSubnet:
Description: The public subnet to launch the bootstrap node into.
Type: AWS::EC2::Subnet::Id
Description: The master security group ID for registering temporary rules.
Type: AWS::EC2::SecurityGroup::Id
VpcId:
BootstrapIgnitionLocation:
Default: s3://my-s3-bucket/bootstrap.ign
Description: Ignition config file location.
Type: String
AutoRegisterELB:
Default: "yes"
AllowedValues:
- "yes"
- "no"
Description: Do you want to invoke NLB registration, which requires a Lambda ARN parameter?
Type: String
RegisterNlbIpTargetsLambdaArn:
Description: ARN for NLB IP target registration lambda.
Type: String
Description: ARN for external API load balancer target group.
Type: String
Description: ARN for internal API load balancer target group.
Type: String
Description: ARN for internal service load balancer target group.
Type: String
BootstrapInstanceType:
Description: Instance type for the bootstrap EC2 instance
Default: "i3.large"
Type: String
Metadata:
ParameterGroups:
- Label:
Parameters:
- Label:
default: "Host Information"
Parameters:
- RhcosAmi
- BootstrapIgnitionLocation
- MasterSecurityGroupId
- Label:
Parameters:
649
- VpcId
- AllowedBootstrapSshCidr
- PublicSubnet
- Label:
default: "Load Balancer Automation"
Parameters:
- AutoRegisterELB
- RegisterNlbIpTargetsLambdaArn
- ExternalApiTargetGroupArn
- InternalApiTargetGroupArn
- InternalServiceTargetGroupArn
ParameterLabels:
InfrastructureName:
VpcId:
default: "VPC ID"
default: "Allowed SSH Source"
PublicSubnet:
default: "Public Subnet"
RhcosAmi:
default: "Red Hat Enterprise Linux CoreOS AMI ID"
default: "Bootstrap Ignition Source"
default: "Master Security Group ID"
AutoRegisterELB:
default: "Use Provided ELB Automation"
Conditions:
DoRegistration: !Equals ["yes", !Ref AutoRegisterELB]
Resources:
BootstrapIamRole:
Properties:
Version: "2012-10-17"
Statement:
- Effect: "Allow"
Principal:
Service:
Action:
- "sts:AssumeRole"
Path: "/"
Policies:
- PolicyName: !Join ["-", [!Ref InfrastructureName, "bootstrap", "policy"]]
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: "Allow"
Action: "ec2:Describe*"
Resource: "*"
- Effect: "Allow"
Action: "ec2:AttachVolume"
650
Resource: "*"
- Effect: "Allow"
Action: "ec2:DetachVolume"
Resource: "*"
- Effect: "Allow"
Action: "s3:GetObject"
Resource: "*"
BootstrapInstanceProfile:
Properties:
Path: "/"
Roles:
- Ref: "BootstrapIamRole"
BootstrapSecurityGroup:
Properties:
GroupDescription: Cluster Bootstrap Security Group
- IpProtocol: tcp
FromPort: 22
ToPort: 22
CidrIp: !Ref AllowedBootstrapSshCidr
- IpProtocol: tcp
ToPort: 19531
FromPort: 19531
CidrIp: 0.0.0.0/0
VpcId: !Ref VpcId
BootstrapInstance:
Type: AWS::EC2::Instance
Properties:
ImageId: !Ref RhcosAmi
IamInstanceProfile: !Ref BootstrapInstanceProfile
InstanceType: !Ref BootstrapInstanceType
NetworkInterfaces:
- AssociatePublicIpAddress: "true"
DeviceIndex: "0"
GroupSet:
- !Ref "BootstrapSecurityGroup"
- !Ref "MasterSecurityGroupId"
SubnetId: !Ref "PublicSubnet"
UserData:
Fn::Base64: !Sub
- '{"ignition":{"config":{"replace":{"source":"${S3Loc}"}},"version":"3.1.0"}}'
-{
S3Loc: !Ref BootstrapIgnitionLocation
}
RegisterBootstrapApiTarget:
Condition: DoRegistration
Type: Custom::NLBRegister
Properties:
ServiceToken: !Ref RegisterNlbIpTargetsLambdaArn
TargetArn: !Ref ExternalApiTargetGroupArn
651
TargetIp: !GetAtt BootstrapInstance.PrivateIp
RegisterBootstrapInternalApiTarget:
Properties:
TargetArn: !Ref InternalApiTargetGroupArn
RegisterBootstrapInternalServiceTarget:
Properties:
TargetArn: !Ref InternalServiceTargetGroupArn
Outputs:
BootstrapInstanceId:
Description: Bootstrap Instance ID.
Value: !Ref BootstrapInstance
BootstrapPublicIp:
Description: The bootstrap node public IP address.
Value: !GetAtt BootstrapInstance.PublicIp
BootstrapPrivateIp:
Description: The bootstrap node private IP address.
Value: !GetAtt BootstrapInstance.PrivateIp
See RHCOS AMIs for the AWS infrastructure for details about the Red Hat Enterprise Linux
CoreOS (RHCOS) AMIs for the AWS zones.
6.12.16. Creating the control plane machines in AWS

You must create the control plane machines in Amazon Web Services (AWS) that your cluster will use.
AWS resources that represent the control plane nodes.
IMPORTANT
The CloudFormation template creates a stack that represents three control plane nodes.
NOTE
652
NOTE
If you do not use the provided CloudFormation template to create your control plane
nodes, you must review the provided information and manually create the infrastructure.
If your cluster does not initialize correctly, you might have to contact Red Hat support
with your installation logs.
Prerequisites
You created the bootstrap machine.
Procedure
[
{
},
{
},
{
"ParameterKey": "AutoRegisterDNS", 5
},
{
"ParameterKey": "PrivateHostedZoneId", 7
},
{
"ParameterKey": "PrivateHostedZoneName", 9
"ParameterValue": "mycluster.example.com" 10
},
{
"ParameterKey": "Master0Subnet", 11
},
{
653
},
{
},
{
},
{
"ParameterKey": "IgnitionLocation", 19
"ParameterValue": "https://api-int.<cluster_name>.<domain_name>:22623/config/master"
20
},
{
"ParameterKey": "CertificateAuthorities", 21
"ParameterValue": "data:text/plain;charset=utf-8;base64,ABC...xYz==" 22
},
{
"ParameterKey": "MasterInstanceProfileName", 23
"ParameterValue": "<roles_stack>-MasterInstanceProfile-<random_string>" 24
},
{
"ParameterKey": "MasterInstanceType", 25
"ParameterValue": "" 26
},
{
},
{
},
{
},
{
},
{
}
]
654
cluster.
3 Current Red Hat Enterprise Linux CoreOS (RHCOS) AMI to use for the control plane
machines based on your selected architecture.
4 Specify an AWS::EC2::Image::Id value.
5 Whether or not to perform DNS etcd registration.
6 Specify yes or no. If you specify yes, you must provide hosted zone information.
7 The Route 53 private zone ID to register the etcd targets with.
8 Specify the PrivateHostedZoneId value from the output of the CloudFormation template
for DNS and load balancing.
10 Specify <cluster_name>.<domain_name> where <domain_name> is the Route 53 base

domain that you used when you generated install-config.yaml file for the cluster. Do not
include the trailing period (.) that is displayed in the AWS console.
11 13 15 A subnet, preferably private, to launch the control plane machines on.
12 14 16 Specify a subnet from the PrivateSubnets value from the output of the
CloudFormation template for DNS and load balancing.
17 The master security group ID to associate with control plane nodes.

19 The location to fetch control plane Ignition config file from.
20 Specify the generated Ignition config file location, https://api-int.<cluster_name>.

<domain_name>:22623/config/master.
21 The base64 encoded certificate authority string to use.
22 Specify the value from the master.ign file that is in the installation directory. This value is
the long string with the format data:text/plain;charset=utf-8;base64,ABC…xYz==.
23 The IAM profile to associate with control plane nodes.
24 Specify the MasterInstanceProfile parameter value from the output of the

CloudFormation template for the security group and roles.
25 The type of AWS instance to use for the control plane machines based on your selected
architecture.
26 The instance type value corresponds to the minimum resource requirements for control
plane machines. For example m6i.xlarge is a type for AMD64 and m6g.xlarge is a type for
ARM64.
655
(ARN) value.




2. Copy the template from the CloudFormation template for control plane machinessection of
this topic and save it as a YAML file on your computer. This template describes the control plane
machines that your cluster requires.
3. If you specified an m5 instance type as the value for MasterInstanceType, add that instance
type to the MasterInstanceType.AllowedValues parameter in the CloudFormation template.
control plane nodes:
IMPORTANT

1 <name> is the name for the CloudFormation stack, such as cluster-control-plane. You
need the name of this stack if you remove the cluster.
that you saved.
656
Example output
arn:aws:cloudformation:us-east-1:269333783861:stack/cluster-control-plane/21c7e2b0-2ee2-
11eb-c6f6-0aa34627df4b
NOTE
The CloudFormation template creates a stack that represents three control plane
nodes.
6.12.16.1. CloudFormation template for control plane machines
You can use the following CloudFormation template to deploy the control plane machines that you need
for your OpenShift Container Platform cluster.
Example 6.67. CloudFormation template for control plane machines
Description: Template for OpenShift Cluster Node Launch (EC2 master instances)
Parameters:
InfrastructureName:
MaxLength: 27
MinLength: 1
Description: A short, unique cluster ID used to tag nodes for the kubelet cloud provider.
Type: String
RhcosAmi:
AutoRegisterDNS:
Default: ""
Description: unused
Type: String
Default: ""
Description: unused
Type: String
PrivateHostedZoneName:
Default: ""
Description: unused
Type: String
Master0Subnet:
Description: The subnets, recommend private, to launch the master nodes into.
Master1Subnet:
657
Master2Subnet:
Description: The master security group ID to associate with master nodes.
IgnitionLocation:
Default: https://api-int.$CLUSTER_NAME.$DOMAIN:22623/config/master
Type: String
CertificateAuthorities:
Default: data:text/plain;charset=utf-8;base64,ABC...xYz==
Description: Base64 encoded certificate authority string to use.
Type: String
MasterInstanceProfileName:
Description: IAM profile to associate with master nodes.
Type: String
MasterInstanceType:
Default: m5.xlarge
Type: String
AutoRegisterELB:
Default: "yes"
AllowedValues:
- "yes"
- "no"
Type: String
Description: ARN for NLB IP target registration lambda. Supply the value from the cluster
infrastructure or select "no" for AutoRegisterELB.
Type: String
Description: ARN for external API load balancer target group. Supply the value from the cluster
Type: String
Description: ARN for internal API load balancer target group. Supply the value from the cluster
Type: String
Description: ARN for internal service load balancer target group. Supply the value from the
cluster infrastructure or select "no" for AutoRegisterELB.
Type: String
Metadata:
ParameterGroups:
- Label:
Parameters:
- Label:
Parameters:
658
- MasterInstanceType
- RhcosAmi
- IgnitionLocation
- CertificateAuthorities
- MasterInstanceProfileName
- Label:
Parameters:
- VpcId
- Master0Subnet
- Master1Subnet
- Master2Subnet
- Label:
Parameters:
- AutoRegisterELB
ParameterLabels:
InfrastructureName:
VpcId:
default: "VPC ID"
Master0Subnet:
default: "Master-0 Subnet"
Master1Subnet:
Master2Subnet:
MasterInstanceType:
default: "Master Instance Type"
default: "Master Instance Profile Name"
RhcosAmi:
default: "Master Ignition Source"
default: "Ignition CA String"
AutoRegisterELB:
Conditions:
Resources:
Master0:
Properties:
659
BlockDeviceMappings:
- DeviceName: /dev/xvda
Ebs:
VolumeSize: "120"
VolumeType: "gp2"
IamInstanceProfile: !Ref MasterInstanceProfileName
InstanceType: !Ref MasterInstanceType
NetworkInterfaces:
- AssociatePublicIpAddress: "false"
DeviceIndex: "0"
GroupSet:
SubnetId: !Ref "Master0Subnet"
UserData:
Fn::Base64: !Sub
- '{"ignition":{"config":{"merge":[{"source":"${SOURCE}"}]},"security":{"tls":
{"certificateAuthorities":[{"source":"${CA_BUNDLE}"}]}},"version":"3.1.0"}}'
-{
SOURCE: !Ref IgnitionLocation,
CA_BUNDLE: !Ref CertificateAuthorities,
}
Tags:
Value: "shared"
RegisterMaster0:
Properties:
TargetIp: !GetAtt Master0.PrivateIp
RegisterMaster0InternalApiTarget:
Properties:
RegisterMaster0InternalServiceTarget:
Properties:
Master1:
Properties:
Ebs:
660
VolumeSize: "120"
VolumeType: "gp2"
NetworkInterfaces:
DeviceIndex: "0"
GroupSet:
UserData:
Fn::Base64: !Sub
-{
}
Tags:
Value: "shared"
RegisterMaster1:
Properties:
Properties:
Properties:
Master2:
Properties:
Ebs:
VolumeSize: "120"
VolumeType: "gp2"
661

NetworkInterfaces:
DeviceIndex: "0"
GroupSet:
UserData:
Fn::Base64: !Sub
-{
}
Tags:
Value: "shared"
RegisterMaster2:
Properties:
Properties:
Properties:
Outputs:
PrivateIPs:
Description: The control-plane node private IP addresses.
Value:
!Join [
",",
[!GetAtt Master0.PrivateIp, !GetAtt Master1.PrivateIp, !GetAtt Master2.PrivateIp]
]
662
6.12.17. Creating the worker nodes in AWS

You can create worker nodes in Amazon Web Services (AWS) for your cluster to use.
NOTE
If you are installing a three-node cluster, skip this step. A three-node cluster consists of
three control plane machines, which also act as compute machines.
AWS resources that represent a worker node.
IMPORTANT
The CloudFormation template creates a stack that represents one worker node. You
must create a stack for each worker node.
NOTE
If you do not use the provided CloudFormation template to create your worker nodes,
installation logs.
Prerequisites
You created the control plane machines.
Procedure
1. Create a JSON file that contains the parameter values that the CloudFormation template
requires:
[
{
663
},
{
},
{
"ParameterKey": "Subnet", 5
},
{
"ParameterKey": "WorkerSecurityGroupId", 7
},
{
"ParameterValue": "https://api-int.<cluster_name>.<domain_name>:22623/config/worker"
10
},
{
},
{
"ParameterKey": "WorkerInstanceProfileName", 13
},
{
"ParameterKey": "WorkerInstanceType", 15
}
]
cluster.
3 Current Red Hat Enterprise Linux CoreOS (RHCOS) AMI to use for the worker nodes
5 A subnet, preferably private, to start the worker nodes on.
6 Specify a subnet from the PrivateSubnets value from the output of the CloudFormation
template for DNS and load balancing.
7 The worker security group ID to associate with worker nodes.
8 Specify the WorkerSecurityGroupId value from the output of the CloudFormation

9 The location to fetch the bootstrap Ignition config file from.
664
10 Specify the generated Ignition config location, https://api-int.<cluster_name>.

<domain_name>:22623/config/worker.
11 Base64 encoded certificate authority string to use.
12 Specify the value from the worker.ign file that is in the installation directory. This value is
13 The IAM profile to associate with worker nodes.
14 Specify the WorkerInstanceProfile parameter value from the output of the

15 The type of AWS instance to use for the compute machines based on your selected
architecture.
16 The instance type value corresponds to the minimum resource requirements for compute
machines. For example m6i.large is a type for AMD64 and m6g.large is a type for ARM64.
2. Copy the template from the CloudFormation template for worker machines section of this
topic and save it as a YAML file on your computer. This template describes the networking
objects and load balancers that your cluster requires.
3. Optional: If you specified an m5 instance type as the value for WorkerInstanceType, add that
instance type to the WorkerInstanceType.AllowedValues parameter in the CloudFormation
template.
4. Optional: If you are deploying with an AWS Marketplace image, update the
Worker0.type.properties.ImageID parameter with the AMI ID that you obtained from your
subscription.
5. Use the CloudFormation template to create a stack of AWS resources that represent a worker
node:
IMPORTANT

--template-body file://<template>.yaml \ 2
1 <name> is the name for the CloudFormation stack, such as cluster-worker-1. You need
that you saved.
file.
Example output
665
arn:aws:cloudformation:us-east-1:269333783861:stack/cluster-worker-1/729ee301-1c2a-
11eb-348f-sd9888c65b59
NOTE
The CloudFormation template creates a stack that represents one worker node.
7. Continue to create worker stacks until you have created enough worker machines for your
cluster. You can create additional worker stacks by referencing the same template and
parameter files and specifying a different stack name.
IMPORTANT
You must create at least two worker machines, so you must create at least two
stacks that use this CloudFormation template.
6.12.17.1. CloudFormation template for worker machines
You can use the following CloudFormation template to deploy the worker machines that you need for
Example 6.68. CloudFormation template for worker machines
Description: Template for OpenShift Cluster Node Launch (EC2 worker instance)
Parameters:
InfrastructureName:
MaxLength: 27
MinLength: 1
Type: String
RhcosAmi:
Subnet:
Description: The subnets, recommend private, to launch the worker nodes into.
Description: The worker security group ID to associate with worker nodes.
IgnitionLocation:
Default: https://api-int.$CLUSTER_NAME.$DOMAIN:22623/config/worker
Type: String
666
Type: String
WorkerInstanceProfileName:
Description: IAM profile to associate with worker nodes.
Type: String
WorkerInstanceType:
Default: m5.large
Type: String
Metadata:
ParameterGroups:
- Label:
Parameters:
- Label:
Parameters:
- WorkerInstanceType
- RhcosAmi
- IgnitionLocation
- WorkerSecurityGroupId
- WorkerInstanceProfileName
- Label:
Parameters:
- Subnet
ParameterLabels:
Subnet:
default: "Subnet"
InfrastructureName:
WorkerInstanceType:
default: "Worker Instance Type"
default: "Worker Instance Profile Name"
RhcosAmi:
IgnitionLocation:
default: "Worker Ignition Source"
default: "Worker Security Group ID"
Resources:
Worker0:
Properties:
Ebs:
667
VolumeSize: "120"
VolumeType: "gp2"
IamInstanceProfile: !Ref WorkerInstanceProfileName
InstanceType: !Ref WorkerInstanceType
NetworkInterfaces:
DeviceIndex: "0"
GroupSet:
- !Ref "WorkerSecurityGroupId"
SubnetId: !Ref "Subnet"
UserData:
Fn::Base64: !Sub
-{
}
Tags:
Value: "shared"
Outputs:
PrivateIP:
Description: The compute node private IP address.
Value: !GetAtt Worker0.PrivateIp
6.12.18. Initializing the bootstrap sequence on AWS with user-provisioned

infrastructure
After you create all of the required infrastructure in Amazon Web Services (AWS), you can start the
bootstrap sequence that initializes the OpenShift Container Platform control plane.
Prerequisites
668
You created the worker nodes.
Procedure
1. Change to the directory that contains the installation program and start the bootstrap process
that initializes the OpenShift Container Platform control plane:
$ ./openshift-install wait-for bootstrap-complete --dir <installation_directory> \ 1

--log-level=info 2
Example output
INFO Waiting up to 20m0s for the Kubernetes API at

https://api.mycluster.example.com:6443...
INFO API v1.28.5 up
INFO Waiting up to 30m0s for bootstrapping to complete...
INFO It is now safe to remove the bootstrap resources
INFO Time elapsed: 1s
If the command exits without a FATAL warning, your OpenShift Container Platform control
plane has initialized.
NOTE
After the control plane initializes, it sets up the compute nodes and installs
additional services in the form of Operators.
See Monitoring installation progress for details about monitoring the installation, bootstrap, and
control plane logs as an OpenShift Container Platform installation progresses.
See Gathering bootstrap node diagnostic data for information about troubleshooting issues
related to the bootstrap process.
You can view details about the running instances that are created by using the AWS EC2
console.

IMPORTANT
669
IMPORTANT

Procedure
Portal.
$ tar xvf <file>

$ echo $PATH
Verification
$ oc <command>

Procedure
Portal.

C:\> path
Verification
670
Verification
C:\> oc <command>

Procedure
Portal.
NOTE

$ echo $PATH
Verification
$ oc <command>

Prerequisites
Procedure
671
$ oc whoami
Example output
system:admin
6.12.21. Approving the certificate signing requests for your machines

When you add machines to a cluster, two pending certificate signing requests (CSRs) are generated for
each machine that you added. You must confirm that these CSRs are approved or, if necessary, approve
them yourself. The client requests must be approved first, followed by the server requests.
Prerequisites
You added machines to your cluster.
Procedure
1. Confirm that the cluster recognizes the machines:
$ oc get nodes
Example output

master-0 Ready master 63m v1.28.5
The output lists all of the machines that you created.
NOTE
The preceding output might not include the compute nodes, also known as
worker nodes, until some CSRs are approved.
2. Review the pending CSRs and ensure that you see the client requests with the Pending or
Approved status for each machine that you added to the cluster:
$ oc get csr
Example output
NAME AGE REQUESTOR CONDITION

csr-8b2br 15m system:serviceaccount:openshift-machine-config-operator:node-
bootstrapper Pending
672
csr-8vnps 15m system:serviceaccount:openshift-machine-config-operator:node-

...
In this example, two machines are joining the cluster. You might see more approved CSRs in the
list.
3. If the CSRs were not approved, after all of the pending CSRs for the machines you added are in
Pending status, approve the CSRs for your cluster machines:
NOTE
Because the CSRs rotate automatically, approve your CSRs within an hour of
adding the machines to the cluster. If you do not approve them within an hour, the
certificates will rotate, and more than two certificates will be present for each
node. You must approve all of these certificates. After the client CSR is
approved, the Kubelet creates a secondary CSR for the serving certificate, which
requires manual approval. Then, subsequent serving certificate renewal requests
are automatically approved by the machine-approver if the Kubelet requests a
new certificate with identical parameters.
NOTE
For clusters running on platforms that are not machine API enabled, such as bare
metal and other user-provisioned infrastructure, you must implement a method
of automatically approving the kubelet serving certificate requests (CSRs). If a
request is not approved, then the oc exec, oc rsh, and oc logs commands
cannot succeed, because a serving certificate is required when the API server
connects to the kubelet. Any operation that contacts the Kubelet endpoint
requires this certificate approval to be in place. The method must watch for new
CSRs, confirm that the CSR was submitted by the node-bootstrapper service
account in the system:node or system:admin groups, and confirm the identity
of the node.
To approve them individually, run the following command for each valid CSR:
$ oc adm certificate approve <csr_name> 1
1 <csr_name> is the name of a CSR from the list of current CSRs.
To approve all pending CSRs, run the following command:
$ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}

{{end}}{{end}}' | xargs --no-run-if-empty oc adm certificate approve
NOTE
Some Operators might not become available until some CSRs are approved.
4. Now that your client requests are approved, you must review the server requests for each
machine that you added to the cluster:
673
$ oc get csr
Example output

csr-bfd72 5m26s system:node:ip-10-0-50-126.us-east-2.compute.internal
Pending
csr-c57lv 5m26s system:node:ip-10-0-95-157.us-east-2.compute.internal
Pending
...
5. If the remaining CSRs are not approved, and are in the Pending status, approve the CSRs for
your cluster machines:

{{end}}{{end}}' | xargs oc adm certificate approve
6. After all client and server CSRs have been approved, the machines have the Ready status.
Verify this by running the following command:
$ oc get nodes
Example output

worker-0 Ready worker 11m v1.28.5
NOTE
It can take a few minutes after approval of the server CSRs for the machines to
transition to the Ready status.
For more information on CSRs, see Certificate Signing Requests .
6.12.22. Initial Operator configuration
After the control plane initializes, you must immediately configure some Operators so that they all
674
become available.
Prerequisites
Your control plane has initialized.
Procedure
1. Watch the cluster components come online:
$ watch -n5 oc get clusteroperators
Example output
NAME VERSION AVAILABLE PROGRESSING DEGRADED

SINCE
authentication 4.15.0 True False False 19m
baremetal 4.15.0 True False False 37m
cloud-credential 4.15.0 True False False 40m
cluster-autoscaler 4.15.0 True False False 37m
config-operator 4.15.0 True False False 38m
console 4.15.0 True False False 26m
csi-snapshot-controller 4.15.0 True False False 37m
dns 4.15.0 True False False 37m
etcd 4.15.0 True False False 36m
image-registry 4.15.0 True False False 31m
ingress 4.15.0 True False False 30m
insights 4.15.0 True False False 31m
kube-apiserver 4.15.0 True False False 26m
kube-controller-manager 4.15.0 True False False 36m
kube-scheduler 4.15.0 True False False 36m
kube-storage-version-migrator 4.15.0 True False False 37m
machine-api 4.15.0 True False False 29m
machine-approver 4.15.0 True False False 37m
machine-config 4.15.0 True False False 36m
marketplace 4.15.0 True False False 37m
monitoring 4.15.0 True False False 29m
network 4.15.0 True False False 38m
node-tuning 4.15.0 True False False 37m
openshift-apiserver 4.15.0 True False False 32m
openshift-controller-manager 4.15.0 True False False 30m
openshift-samples 4.15.0 True False False 32m
operator-lifecycle-manager 4.15.0 True False False 37m
operator-lifecycle-manager-catalog 4.15.0 True False False 37m
operator-lifecycle-manager-packageserver 4.15.0 True False False 32m
service-ca 4.15.0 True False False 38m
storage 4.15.0 True False False 37m
2. Configure the Operators that are not available.
6.12.22.1. Image registry storage configuration
Amazon Web Services provides default storage, which means the Image Registry Operator is available
675
after installation. However, if the Registry Operator cannot create an S3 bucket and automatically
configure storage, you must manually configure registry storage.
Instructions are shown for configuring a persistent volume, which is required for production clusters.
Where applicable, instructions are shown for configuring an empty directory as the storage location,
which is available for only non-production clusters.
Additional instructions are provided for allowing the image registry to use block storage types by using
the Recreate rollout strategy during upgrades.
You can configure registry storage for user-provisioned infrastructure in AWS to deploy OpenShift
Container Platform to hidden regions. See Configuring the registry for AWS user-provisioned
infrastructure for more information.
6.12.22.1.1. Configuring registry storage for AWS with user-provisioned infrastructure
During installation, your cloud credentials are sufficient to create an Amazon S3 bucket and the Registry
Operator will automatically configure storage.
If the Registry Operator cannot create an S3 bucket and automatically configure storage, you can
create an S3 bucket and configure storage with the following procedure.
Prerequisites
You have a cluster on AWS with user-provisioned infrastructure.
For Amazon S3 storage, the secret is expected to contain two keys:
REGISTRY_STORAGE_S3_ACCESSKEY
REGISTRY_STORAGE_S3_SECRETKEY
Procedure
Use the following procedure if the Registry Operator cannot create an S3 bucket and automatically
configure storage.
1. Set up a Bucket Lifecycle Policy to abort incomplete multipart uploads that are one day old.
2. Fill in the storage configuration in configs.imageregistry.operator.openshift.io/cluster:
$ oc edit configs.imageregistry.operator.openshift.io/cluster
Example configuration
storage:
s3:
bucket: <bucket-name>
region: <region-name>
676

WARNING
To secure your registry images in AWS, block public access to the S3 bucket.
6.12.22.1.2. Configuring storage for the image registry in non-production clusters
You must configure storage for the Image Registry Operator. For non-production clusters, you can set
the image registry to an empty directory. If you do so, all images are lost if you restart the registry.
Procedure
To set the image registry storage to an empty directory:
$ oc patch configs.imageregistry.operator.openshift.io cluster --type merge --patch '{"spec":

{"storage":{"emptyDir":{}}}}'

WARNING
Configure this option for only non-production clusters.
If you run this command before the Image Registry Operator initializes its components, the oc
patch command fails with the following error:
Error from server (NotFound): configs.imageregistry.operator.openshift.io "cluster" not found
Wait a few minutes and run the command again.
6.12.23. Deleting the bootstrap resources

After you complete the initial Operator configuration for the cluster, remove the bootstrap resources
from Amazon Web Services (AWS).
Prerequisites
You completed the initial Operator configuration for your cluster.
Procedure
1. Delete the bootstrap resources. If you used the CloudFormation template, delete its stack :
Delete the stack by using the AWS CLI:
$ aws cloudformation delete-stack --stack-name <name> 1
677
1 <name> is the name of your bootstrap stack.
Delete the stack by using the AWS CloudFormation console.
6.12.24. Creating the Ingress DNS Records

If you removed the DNS Zone configuration, manually create DNS records that point to the Ingress load
balancer. You can create either a wildcard record or specific records. While the following procedure uses
A records, you can use other record types that you require, such as CNAME or alias.
Prerequisites
You deployed an OpenShift Container Platform cluster on Amazon Web Services (AWS) that
uses infrastructure that you provisioned.
You installed the OpenShift CLI (oc).
the Bundled Installer (Linux, macOS, or Unix).
Procedure
1. Determine the routes to create.
To create a wildcard record, use *.apps.<cluster_name>.<domain_name>, where

<cluster_name> is your cluster name, and <domain_name> is the Route 53 base domain
To create specific records, you must create a record for each route that your cluster uses, as
shown in the output of the following command:
$ oc get --all-namespaces -o jsonpath='{range .items[*]}{range .status.ingress[*]}{.host}

{"\n"}{end}{end}' routes
Example output
oauth-openshift.apps.<cluster_name>.<domain_name>
console-openshift-console.apps.<cluster_name>.<domain_name>
downloads-openshift-console.apps.<cluster_name>.<domain_name>
alertmanager-main-openshift-monitoring.apps.<cluster_name>.<domain_name>
prometheus-k8s-openshift-monitoring.apps.<cluster_name>.<domain_name>
2. Retrieve the Ingress Operator load balancer status and note the value of the external IP address
that it uses, which is shown in the EXTERNAL-IP column:
$ oc -n openshift-ingress get service router-default
Example output
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)

AGE
678
router-default LoadBalancer 172.30.62.215 ab3...28.us-east-2.elb.amazonaws.com

80:31499/TCP,443:30693/TCP 5m
3. Locate the hosted zone ID for the load balancer:
$ aws elb describe-load-balancers | jq -r '.LoadBalancerDescriptions[] | select(.DNSName ==

"<external_ip>").CanonicalHostedZoneNameID' 1
1 For <external_ip>, specify the value of the external IP address of the Ingress Operator
load balancer that you obtained.
Example output
Z3AADJGX6KTTL2
The output of this command is the load balancer hosted zone ID.
4. Obtain the public hosted zone ID for your cluster’s domain:
$ aws route53 list-hosted-zones-by-name \

--dns-name "<domain_name>" \ 1
--query 'HostedZones[? Config.PrivateZone != `true` && Name ==
`<domain_name>.`].Id' 2
--output text
1 2 For <domain_name>, specify the Route 53 base domain for your OpenShift Container
Platform cluster.
Example output
/hostedzone/Z3URY6TWQ91KVV
The public hosted zone ID for your domain is shown in the command output. In this example, it is
Z3URY6TWQ91KVV.
5. Add the alias records to your private zone:
$ aws route53 change-resource-record-sets --hosted-zone-id "<private_hosted_zone_id>" --

change-batch '{ 1
> "Changes": [
> {
> "Action": "CREATE",
> "ResourceRecordSet": {
> "Name": "\\052.apps.<cluster_domain>", 2
> "Type": "A",
> "AliasTarget":{
> "HostedZoneId": "<hosted_zone_id>", 3
> "DNSName": "<external_ip>.", 4
> "EvaluateTargetHealth": false
> }
> }
679
> }
> ]
> }'
1 For <private_hosted_zone_id>, specify the value from the output of the CloudFormation
2 For <cluster_domain>, specify the domain or subdomain that you use with your
3 For <hosted_zone_id>, specify the public hosted zone ID for the load balancer that you
obtained.
load balancer. Ensure that you include the trailing period (.) in this parameter value.
6. Add the records to your public zone:
$ aws route53 change-resource-record-sets --hosted-zone-id "<public_hosted_zone_id>"" --

change-batch '{ 1
> "Changes": [
> {
> "Type": "A",
> "AliasTarget":{
> }
> }
> }
> ]
> }'
1 For <public_hosted_zone_id>, specify the public hosted zone for your domain.
obtained.
6.12.25. Completing an AWS installation on user-provisioned infrastructure

After you start the OpenShift Container Platform installation on Amazon Web Service (AWS) user-
provisioned infrastructure, monitor the deployment to completion.
Prerequisites
680
You removed the bootstrap node for an OpenShift Container Platform cluster on user-
provisioned AWS infrastructure.
Procedure
From the directory that contains the installation program, complete the cluster installation:
$ ./openshift-install --dir <installation_directory> wait-for install-complete 1
Example output
INFO Waiting up to 40m0s for the cluster at https://api.mycluster.example.com:6443 to

initialize...
INFO Waiting up to 10m0s for the openshift-console route to be created...
IMPORTANT
certificates that expire after 24 hours, which are then renewed at that time. If
the cluster is shut down before renewing the certificates and the cluster is
later restarted after the 24 hours have elapsed, the cluster automatically
recovers the expired certificates. The exception is that you must manually
approve the pending node-bootstrapper certificate signing requests (CSRs)
to recover kubelet certificates. See the documentation for Recovering from
expired control plane certificates for more information.
It is recommended that you use Ignition config files within 12 hours after they
are generated because the 24-hour certificate rotates from 16 to 22 hours
after the cluster is installed. By using the Ignition config files within 12 hours,
you can avoid installation failure if the certificate update runs during
installation.

Prerequisites
681
Procedure
installation host:
NOTE

NOTE
Example output


682
See Working with stacks in the AWS documentation for more information about AWS
CloudFormation stacks.
6.12.29. Next steps

6.13. INSTALLING A CLUSTER ON AWS IN A RESTRICTED NETWORK

WITH USER-PROVISIONED INFRASTRUCTURE
using infrastructure that you provide and an internal mirror of the installation release content.
IMPORTANT
While you can install an OpenShift Container Platform cluster by using mirrored
installation release content, your cluster still requires internet access to use the AWS
APIs.
One way to create this infrastructure is to use the provided CloudFormation templates. You can modify
the templates to customize your infrastructure or use the information that they contain to create AWS
objects according to your company’s policies.
IMPORTANT

CloudFormation templates are provided to assist in completing these steps or to help
model your own. You are also free to create the required resources through other
processes.
users.
You created a mirror registry on your mirror host and obtained the imageContentSources data
for your version of OpenShift Container Platform.
IMPORTANT
683
IMPORTANT
the Bundled Installer (Linux, macOS, or Unix) in the AWS documentation.
NOTE

your restrictions.
IMPORTANT
Because of the complexity of the configuration for user-provisioned installations,

consider completing a standard user-provisioned infrastructure installation before you
attempt a restricted network installation using user-provisioned infrastructure.
Completing this test installation might make it easier to isolate and troubleshoot any
issues that might arise during your installation in a restricted network.
684


Hosts Description
control plane.
IMPORTANT
machines.
685

System Per Second
(IOPS)[2]

8.6 and later
[3]
Optimizing storage
Platform.
NOTE
686
NOTE
c4.*
c5.*
c5a.*
i3.*
m4.*
m5.*
m5a.*
m6a.*
m6i.*
r4.*
r5.*
r5a.*
r6i.*
t3.*
t3a.*
NOTE
c6g.*
687
m6g.*
and approving them.
6.13.5. Required AWS infrastructure components

To install OpenShift Container Platform on user-provisioned infrastructure in Amazon Web Services
(AWS), you must manually create both the machines and their supporting infrastructure.
For more information about the integration testing for different platforms, see the OpenShift Container
Platform 4.x Tested Integrations page.
By using the provided CloudFormation templates, you can create stacks of AWS resources that
represent the following components:
An AWS Virtual Private Cloud (VPC)
Networking and load balancing components
Security groups and roles
An OpenShift Container Platform bootstrap node
OpenShift Container Platform control plane nodes
An OpenShift Container Platform compute node
Alternatively, you can manually create the components or you can reuse existing infrastructure that
meets the cluster requirements. Review the CloudFormation templates for more details about how the
components interrelate.
6.13.5.1. Other infrastructure components
A VPC
DNS entries
Load balancers (classic or network) and listeners
A public and a private Route 53 zone
Security groups
IAM roles
S3 buckets
688

follows:


follows:


nt

AWS::EC2::VPC

AWS::EC2::Subnet
ociation rules.
689

nt

AWS::EC2::Route
ociation
AWS::EC2::EIP

Port Reason
80 Inbound HTTP
traffic
443 Inbound HTTPS

traffic
22 Inbound SSH
traffic
1024 - 65535 Inbound

ephemeral traffic
0 - 65535 Outbound
ephemeral traffic

AWS::EC2::Subnet
Required DNS and load balancing components

Your DNS and load balancer configuration needs to use a public hosted zone and can use a private
hosted zone similar to the one that the installation program uses if it provisions the cluster’s
infrastructure. You must create a DNS entry that resolves to your load balancer. An entry for api.
690
<cluster_name>.<domain> must point to the external load balancer, and an entry for api-int.
<cluster_name>.<domain> must point to the internal load balancer.
The cluster also requires load balancers and listeners for port 6443, which are required for the
Kubernetes API and its extensions, and port 22623, which are required for the Ignition config files for
new machines. The targets will be the control plane nodes. Port 6443 must be accessible to both clients
external to the cluster and nodes within the cluster. Port 22623 must be accessible to nodes within the
cluster.
DNS AWS::Route The hosted zone for your internal DNS.

53::HostedZ
one
Public load AWS::Elastic The load balancer for your public subnets.
ngV2::LoadB
alancer
External API AWS::Route Alias records for the external API server.
etGroup
External AWS::Elastic A listener on port 6443 for the external load balancer.
listener LoadBalanci
ngV2::Listen
er
External target AWS::Elastic The target group for the external load balancer.
group LoadBalanci
ngV2::Target
Group
Private load AWS::Elastic The load balancer for your private subnets.
ngV2::LoadB
alancer
Internal API AWS::Route Alias records for the internal API server.
etGroup
LoadBalanci
ngV2::Listen
er
group LoadBalanci
ngV2::Target
Group
691
LoadBalanci
ngV2::Listen
er
group LoadBalanci
ngV2::Target
Group
Security groups
The control plane and worker machines require access to the following ports:
MasterSecurityGrou AWS::EC2::Security icmp 0

p Group
tcp 22
tcp 6443
tcp 22623
WorkerSecurityGrou AWS::EC2::Security icmp 0

p Group
tcp 22
BootstrapSecurityGr AWS::EC2::Security tcp 22

oup Group
tcp 19531
Control plane Ingress

The control plane machines require the following Ingress groups. Each Ingress group is a
MasterIngress etcd tcp 2379- 2380

Etcd

Vxlan
692

WorkerVxlan
MasterIngress Internal cluster communication and Kubernetes tcp 9000 - 9999

Internal proxy metrics
MasterIngress Internal cluster communication tcp 9000 - 9999

WorkerInterna
l

Kube manager

WorkerKube manager

IngressServic
es

WorkerIngress
Services

Geneve

WorkerGenev
e

IpsecIke

WorkerIpsecIk
e

IpsecNat

WorkerIpsecN
at
693

IpsecEsp

WorkerIpsecE
sp

InternalUDP

WorkerInterna
lUDP

IngressServic
esUDP

WorkerIngress
ServicesUDP
Worker Ingress
The worker machines require the following Ingress groups. Each Ingress group is a

Vxlan

WorkerVxlan

Internal

WorkerInterna
l

Kube manager
694

WorkerKube manager

IngressServic
es

WorkerIngress
Services

Geneve

MasterGeneve

IpsecIke

MasterIpsecIk
e

IpsecNat

MasterIpsecN
at

IpsecEsp

MasterIpsecEs
p

InternalUDP

MasterInternal
UDP

IngressServic
esUDP
695

MasterIngress
ServicesUDP
Roles and instance profiles

You must grant the machines permissions in AWS. The provided CloudFormation templates grant the
machines Allow permissions for the following AWS::IAM::Role objects and provide a
AWS::IAM::InstanceProfile for each set of roles. If you do not use the templates, you can grant the
machines the following broad permissions or the following individual permissions.
Role Effect Action Resource
Master Allow ec2:* *
Allow elasticloadbalancing *
:*
Allow iam:PassRole *
Allow s3:GetObject *
Worker Allow ec2:Describe* *
Bootstrap Allow ec2:Describe* *
Allow ec2:AttachVolume *
Allow ec2:DetachVolume *
6.13.5.2. Cluster machines
You need AWS::EC2::Instance objects for the following machines:
A bootstrap machine. This machine is required during installation, but you can remove it after
your cluster deploys.
Three control plane machines. The control plane machines are not governed by a control plane
machine set.
Compute machines. You must create at least two compute machines, which are also known as
worker machines, during installation. These machines are not governed by a compute machine
set.
6.13.5.3. Required AWS permissions for the IAM user
NOTE
696
NOTE
ec2:CopyImage
ec2:CreateTags
ec2:CreateVolume
ec2:DeleteSnapshot
ec2:DeleteTags
ec2:DeregisterImage
ec2:DescribeImages
697
ec2:DescribeRegions
ec2:DescribeSubnets
ec2:DescribeTags
ec2:DescribeVolumes
ec2:DescribeVpcs
ec2:RunInstances
ec2:AllocateAddress
698
ec2:CreateRoute
ec2:CreateSubnet
ec2:CreateVpc
NOTE
699
iam:CreateRole
iam:DeleteRole
iam:GetRole
iam:GetRolePolicy
iam:GetUser
700
iam:ListRoles
iam:ListUsers
iam:PassRole
iam:PutRolePolicy
iam:TagRole
NOTE
route53:GetChange
s3:CreateBucket
s3:DeleteBucket
s3:GetBucketAcl
701
s3:GetBucketCors
s3:GetBucketLogging
s3:GetBucketPolicy
s3:GetBucketTagging
s3:GetBucketWebsite
s3:ListBucket
s3:PutBucketAcl
s3:PutBucketTagging
s3:DeleteObject
s3:GetObject
s3:GetObjectAcl
s3:GetObjectTagging
s3:GetObjectVersion
s3:PutObject
s3:PutObjectAcl
s3:PutObjectTagging
702
ec2:DeleteVolume
iam:DeleteAccessKey
iam:DeleteUser
s3:DeleteObject
tag:GetResources
ec2:DeleteRoute
ec2:DeleteSubnet
ec2:DeleteVpc
ec2:ReleaseAddress
NOTE
703
NOTE
iam:UntagRole
iam:DeleteAccessKey
iam:DeleteUser
iam:GetUserPolicy
iam:ListAccessKeys
iam:PutUserPolicy
iam:TagUser
s3:ListBucket
NOTE
704
shared VPC
sts:AssumeRole

authentication.
user.
IMPORTANT
NOTE
Procedure
NOTE
ecdsa algorithm.
705
NOTE

task:
Example output
Agent pid 31874
NOTE

Example output
Next steps
6.13.7. Creating the installation files for AWS

To install OpenShift Container Platform on Amazon Web Services (AWS) using user-provisioned
706
installation.
IMPORTANT
Procedure
Example output

707

Example output
...
variant: openshift
version: 4.15.0
metadata:
labels:
storage:
disks:
partitions:
- label: var
number: 5
filesystems:
path: /var
format: xfs
NOTE
708
NOTE
name.

partition.yaml

your cluster.
Prerequisites
infrastructure and the pull secret for your cluster. For a restricted network installation, these
files are on your mirror host.
Procedure
command:
IMPORTANT
709
IMPORTANT

version.
NOTE

program.
NOTE
stored in the file.
pullSecret: '{"auths":{"<local_registry>": {"auth": "<credentials>","email":

For <local_registry>, specify the registry domain name, and optionally the port, that your
mirror registry uses to serve content. For example registry.example.com or
registry.example.com:5000. For <credentials>, specify the base64-encoded user name
and password for your mirror registry.
710
b. Add the additionalTrustBundle parameter and value. The value must be the contents of
the certificate file that you used for your mirror registry. The certificate file can be an
existing, trusted certificate authority or the self-signed certificate that you generated for
the mirror registry.
c. Add the image content resources:
- mirrors:
- mirrors:
Use the imageContentSources section from the output of the command to mirror the
repository or the values that you used when you mirrored the content from the media that
you brought into your restricted network.
d. Optional: Set the publishing strategy to Internal:
publish: Internal
IMPORTANT

Prerequisites
711
NOTE
(169.254.169.254).
Procedure
apiVersion: v1
proxy:
must be http.

712
NOTE
NOTE
NOTE
created.
machines.
IMPORTANT
Prerequisites
You obtained the OpenShift Container Platform installation program. For a restricted network
713
installation, these files are on your mirror host.
Procedure
machines.
machine-set.yaml
IMPORTANT

these machines.

machines:
714
kind: DNS
metadata:
name: cluster
spec:
privateZone: 1
publicZone: 2
status: {}
.
├── auth
Manually creating long-term credentials
6.13.8. Extracting the infrastructure name

cluster in Amazon Web Services (AWS). The infrastructure name is also used to locate the appropriate
AWS resources during an OpenShift Container Platform installation. The provided CloudFormation
Prerequisites
your cluster.
715
Procedure
following command:
Example output
openshift-vw9j6 1
6.13.9. Creating a VPC in AWS

You must create a Virtual Private Cloud (VPC) in Amazon Web Services (AWS) for your OpenShift
Container Platform cluster to use. You can customize the VPC to meet your requirements, including
VPN and route tables.
NOTE
Prerequisites
Procedure
[
{
716
},
{
},
{
}
]
2. Copy the template from the CloudFormation template for the VPCsection of this topic and
save it as a YAML file on your computer. This template describes the VPC that your cluster
requires.
VPC:
IMPORTANT

that you saved.
file.
Example output
820e-12a48460849f
717

netIds

bnetIds
Parameters:
VpcCidr:
AllowedPattern: ^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-
4][0-9]|25[0-5])(\/(1[6-9]|2[0-4]))$
Default: 10.0.0.0/16
Type: String
MinValue: 1
MaxValue: 3
Default: 1
Type: Number
SubnetBits:
MinValue: 5
MaxValue: 13
Default: 12
/19)"
Type: Number
Metadata:
ParameterGroups:
718
- Label:
Parameters:
- VpcCidr
- SubnetBits
- Label:
Parameters:
ParameterLabels:
VpcCidr:
default: "VPC CIDR"
SubnetBits:
Conditions:
Resources:
VPC:
Properties:
PublicSubnet:
Properties:
VpcId: !Ref VPC
-0
PublicSubnet2:
Condition: DoAz2
Properties:
VpcId: !Ref VPC
-1
PublicSubnet3:
Condition: DoAz3
Properties:
VpcId: !Ref VPC
-2
InternetGateway:
719
GatewayToInternet:
Properties:
VpcId: !Ref VPC
PublicRouteTable:
Properties:
VpcId: !Ref VPC
PublicRoute:
Properties:
Properties:
Condition: DoAz2
Properties:
Condition: DoAz3
Properties:
PrivateSubnet:
Properties:
VpcId: !Ref VPC
-0
PrivateRouteTable:
Properties:
VpcId: !Ref VPC
Properties:
NAT:
DependsOn:
- GatewayToInternet
Properties:
AllocationId:
720
"Fn::GetAtt":
- EIP
- AllocationId
EIP:
Properties:
Domain: vpc
Route:
Properties:
RouteTableId:
NatGatewayId:
Ref: NAT
PrivateSubnet2:
Condition: DoAz2
Properties:
VpcId: !Ref VPC
-1
PrivateRouteTable2:
Condition: DoAz2
Properties:
VpcId: !Ref VPC
Condition: DoAz2
Properties:
NAT2:
DependsOn:
- GatewayToInternet
Condition: DoAz2
Properties:
AllocationId:
"Fn::GetAtt":
- EIP2
- AllocationId
EIP2:
Condition: DoAz2
Properties:
Domain: vpc
Route2:
Condition: DoAz2
Properties:
721
RouteTableId:
NatGatewayId:
Ref: NAT2
PrivateSubnet3:
Condition: DoAz3
Properties:
VpcId: !Ref VPC
-2
PrivateRouteTable3:
Condition: DoAz3
Properties:
VpcId: !Ref VPC
Condition: DoAz3
Properties:
NAT3:
DependsOn:
- GatewayToInternet
Condition: DoAz3
Properties:
AllocationId:
"Fn::GetAtt":
- EIP3
- AllocationId
EIP3:
Condition: DoAz3
Properties:
Domain: vpc
Route3:
Condition: DoAz3
Properties:
RouteTableId:
NatGatewayId:
Ref: NAT3
S3Endpoint:
Properties:
PolicyDocument:
Version: 2012-10-17
Statement:
722
- Effect: Allow
Principal: '*'
Action:
- '*'
Resource:
- '*'
RouteTableIds:
ServiceName: !Join
- ''
- - com.amazonaws.
- .s3
VpcId: !Ref VPC
Outputs:
VpcId:
Value: !Ref VPC
PublicSubnetIds:
Value:
!Join [
",",
]
PrivateSubnetIds:
Value:
!Join [
",",
]
PublicRouteTableId:
Value:
!Join [
",",
[
!Join ["=", [
]],
!If [DoAz2,
!Ref "AWS::NoValue"
],
!If [DoAz3,
723

!Ref "AWS::NoValue"
]
]
]
6.13.10. Creating networking and load balancing components in AWS

You must configure networking and classic or network load balancing in Amazon Web Services (AWS)
that your OpenShift Container Platform cluster can use.
AWS resources. The stack represents the networking and load balancing components that your
OpenShift Container Platform cluster requires. The template also creates a hosted zone and subnet
tags.
You can run the template multiple times within a single Virtual Private Cloud (VPC).
NOTE
Prerequisites
Procedure
1. Obtain the hosted zone ID for the Route 53 base domain that you specified in the install-
config.yaml file for your cluster. You can obtain details about your hosted zone by running the
following command:
$ aws route53 list-hosted-zones-by-name --dns-name <route53_domain> 1
1 For the <route53_domain>, specify the Route 53 base domain that you used when you
generated the install-config.yaml file for the cluster.
Example output
mycluster.example.com. False 100

HOSTEDZONES 65F8F38E-2268-B835-E15C-AB55336FCBFA
/hostedzone/Z21IXYZABCZ2A4 mycluster.example.com. 10
724
In the example output, the hosted zone ID is Z21IXYZABCZ2A4.
[
{
"ParameterKey": "ClusterName", 1
"ParameterValue": "mycluster" 2
},
{
},
{
"ParameterKey": "HostedZoneId", 5
},
{
"ParameterKey": "HostedZoneName", 7
"ParameterValue": "example.com" 8
},
{
"ParameterKey": "PublicSubnets", 9
},
{
},
{
}
]
1 A short, representative cluster name to use for hostnames, etc.
2 Specify the cluster name that you used when you generated the install-config.yaml file
for the cluster.
cluster.
5 The Route 53 public zone ID to register the targets with.
6 Specify the Route 53 public zone ID, which as a format similar to Z21IXYZABCZ2A4. You
can obtain this value from the AWS console.
8 Specify the Route 53 base domain that you used when you generated the install-
725
AWS console.
9 The public subnets that you created for your VPC.
the VPC.
the VPC.
3. Copy the template from the CloudFormation template for the network and load balancers
section of this topic and save it as a YAML file on your computer. This template describes the
networking and load balancing objects that your cluster requires.
IMPORTANT
If you are deploying your cluster to an AWS government or secret region, you
must update the InternalApiServerRecord in the CloudFormation template to
use CNAME records. Records of type ALIAS are not supported for AWS
government regions.
4. Launch the CloudFormation template to create a stack of AWS resources that provide the
networking and load balancing components:
IMPORTANT

1 <name> is the name for the CloudFormation stack, such as cluster-dns. You need the
that you saved.
file.

provided template creates some AWS::IAM::Role resources.
Example output
726
arn:aws:cloudformation:us-east-1:269333783861:stack/cluster-dns/cd3e5de0-2fd4-11eb-
5cf0-12be5c33a183
PrivateHo Hosted zone ID for the private DNS.

stedZoneI
d
ExternalA Full name of the external API load balancer.

piLoadBal
ancerNam
e
InternalAp Full name of the internal API load balancer.

iLoadBala
ncerName
ApiServer Full hostname of the API server.

DnsName
RegisterN Lambda ARN useful to help register/deregister IP targets for these load balancers.
lbIpTarget
sLambda
ExternalA ARN of external API target group.

piTargetG
roupArn
InternalAp ARN of internal API target group.

iTargetGr
oupArn
InternalSe ARN of internal service target group.

rviceTarg
etGroupA
rn
6.13.10.1. CloudFormation template for the network and load balancers
You can use the following CloudFormation template to deploy the networking objects and load
balancers that you need for your OpenShift Container Platform cluster.
Example 6.86. CloudFormation template for the network and load balancers
727
Description: Template for OpenShift Cluster Network Elements (Route53 & LBs)
Parameters:
ClusterName:
MaxLength: 27
MinLength: 1
ConstraintDescription: Cluster name must be alphanumeric, start with a letter, and have a
Description: A short, representative cluster name to use for host names and other identifying
names.
Type: String
InfrastructureName:
MaxLength: 27
MinLength: 1
Type: String
HostedZoneId:
Description: The Route53 public zone ID to register the targets with, such as
Z21IXYZABCZ2A4.
Type: String
HostedZoneName:
Description: The Route53 zone to register the targets with, such as example.com. Omit the
trailing period.
Type: String
Default: "example.com"
PublicSubnets:
Description: The internet-facing subnets.
PrivateSubnets:
VpcId:
Metadata:
ParameterGroups:
- Label:
Parameters:
- ClusterName
- Label:
Parameters:
- VpcId
- PublicSubnets
- PrivateSubnets
- Label:
728
default: "DNS"
Parameters:
- HostedZoneName
- HostedZoneId
ParameterLabels:
ClusterName:
default: "Cluster Name"
InfrastructureName:
VpcId:
default: "VPC ID"
PublicSubnets:
default: "Public Subnets"
PrivateSubnets:
HostedZoneName:
default: "Public Hosted Zone Name"
HostedZoneId:
default: "Public Hosted Zone ID"
Resources:
ExtApiElb:
Properties:
Name: !Join ["-", [!Ref InfrastructureName, "ext"]]
IpAddressType: ipv4
Type: network
IntApiElb:
Properties:
Name: !Join ["-", [!Ref InfrastructureName, "int"]]
Scheme: internal
IpAddressType: ipv4
Type: network
IntDns:
Type: "AWS::Route53::HostedZone"
Properties:
HostedZoneConfig:
Comment: "Managed by CloudFormation"
Name: !Join [".", [!Ref ClusterName, !Ref HostedZoneName]]
HostedZoneTags:
- Key: Name
Value: !Join ["-", [!Ref InfrastructureName, "int"]]
Value: "owned"
VPCs:
- VPCId: !Ref VpcId
VPCRegion: !Ref "AWS::Region"
ExternalApiServerRecord:
Properties:
729

HostedZoneId: !Ref HostedZoneId
RecordSets:
- Name:
!Join [
".",
]
Type: A
AliasTarget:
HostedZoneId: !GetAtt ExtApiElb.CanonicalHostedZoneID
DNSName: !GetAtt ExtApiElb.DNSName
InternalApiServerRecord:
Properties:
HostedZoneId: !Ref IntDns
RecordSets:
- Name:
!Join [
".",
]
Type: A
AliasTarget:
- Name:
!Join [
".",
["api-int", !Ref ClusterName, !Join ["", [!Ref HostedZoneName, "."]]],
]
Type: A
AliasTarget:
ExternalApiListener:
Properties:
DefaultActions:
- Type: forward
TargetGroupArn:
Ref: ExternalApiTargetGroup
LoadBalancerArn:
Ref: ExtApiElb
Port: 6443
Protocol: TCP
ExternalApiTargetGroup:
Properties:
730
Port: 6443
Protocol: TCP
TargetType: ip
VpcId:
Ref: VpcId
Value: 60
InternalApiListener:
Properties:
DefaultActions:
- Type: forward
TargetGroupArn:
Ref: InternalApiTargetGroup
LoadBalancerArn:
Ref: IntApiElb
Port: 6443
Protocol: TCP
InternalApiTargetGroup:
Properties:
Port: 6443
Protocol: TCP
TargetType: ip
VpcId:
Ref: VpcId
Value: 60
InternalServiceInternalListener:
Properties:
DefaultActions:
- Type: forward
TargetGroupArn:
Ref: InternalServiceTargetGroup
LoadBalancerArn:
Ref: IntApiElb
Port: 22623
Protocol: TCP
InternalServiceTargetGroup:
731
Properties:
HealthCheckPath: "/healthz"
Port: 22623
Protocol: TCP
TargetType: ip
VpcId:
Ref: VpcId
Value: 60
RegisterTargetLambdaIamRole:
Properties:
RoleName: !Join ["-", [!Ref InfrastructureName, "nlb", "lambda", "role"]]
Version: "2012-10-17"
Statement:
- Effect: "Allow"
Principal:
Service:
Action:
- "sts:AssumeRole"
Path: "/"
Policies:
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: "Allow"
Action:
[
]
Resource: !Ref InternalApiTargetGroup
- Effect: "Allow"
Action:
[
]
Resource: !Ref InternalServiceTargetGroup
- Effect: "Allow"
Action:
[
]
Resource: !Ref ExternalApiTargetGroup
732
RegisterNlbIpTargets:
Properties:
Role:
Fn::GetAtt:
- "RegisterTargetLambdaIamRole"
- "Arn"
Code:
ZipFile: |
import json
import boto3
import cfnresponse
elb = boto3.client('elbv2')
elb.deregister_targets(TargetGroupArn=event['ResourceProperties']
['TargetArn'],Targets=[{'Id': event['ResourceProperties']['TargetIp']}])
elb.register_targets(TargetGroupArn=event['ResourceProperties']['TargetArn'],Targets=
[{'Id': event['ResourceProperties']['TargetIp']}])
responseData = {}
event['ResourceProperties']['TargetArn']+event['ResourceProperties']['TargetIp'])
Timeout: 120
RegisterSubnetTagsLambdaIamRole:
Properties:
RoleName: !Join ["-", [!Ref InfrastructureName, "subnet-tags-lambda-role"]]
Version: "2012-10-17"
Statement:
- Effect: "Allow"
Principal:
Service:
Action:
- "sts:AssumeRole"
Path: "/"
Policies:
- PolicyName: !Join ["-", [!Ref InfrastructureName, "subnet-tagging-policy"]]
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: "Allow"
Action:
[
"ec2:DeleteTags",
"ec2:CreateTags"
]
Resource: "arn:aws:ec2:*:*:subnet/*"
- Effect: "Allow"
Action:
733
[
"ec2:DescribeSubnets",
"ec2:DescribeTags"
]
Resource: "*"
RegisterSubnetTags:
Properties:
Role:
Fn::GetAtt:
- "RegisterSubnetTagsLambdaIamRole"
- "Arn"
Code:
ZipFile: |
import json
import boto3
import cfnresponse
ec2_client = boto3.client('ec2')
ec2_client.delete_tags(Resources=[subnet_id], Tags=[{'Key': 'kubernetes.io/cluster/' +
event['ResourceProperties']['InfrastructureName']}]);
ec2_client.create_tags(Resources=[subnet_id], Tags=[{'Key': 'kubernetes.io/cluster/' +
event['ResourceProperties']['InfrastructureName'], 'Value': 'shared'}]);
responseData = {}
event['ResourceProperties']['InfrastructureName']+event['ResourceProperties']['Subnets'][0])
Timeout: 120
RegisterPublicSubnetTags:
Properties:
RegisterPrivateSubnetTags:
Properties:
Outputs:
Description: Hosted zone ID for the private DNS, which is required for private records.
Value: !Ref IntDns
ExternalApiLoadBalancerName:
Description: Full name of the external API load balancer.
Value: !GetAtt ExtApiElb.LoadBalancerFullName
734
InternalApiLoadBalancerName:
Description: Full name of the internal API load balancer.
Value: !GetAtt IntApiElb.LoadBalancerFullName
ApiServerDnsName:
Description: Full hostname of the API server, which is required for the Ignition config files.
Value: !Join [".", ["api-int", !Ref ClusterName, !Ref HostedZoneName]]
RegisterNlbIpTargetsLambda:
Description: Lambda ARN useful to help register or deregister IP targets for these load
balancers.
Value: !GetAtt RegisterNlbIpTargets.Arn
Description: ARN of the external API target group.
Value: !Ref ExternalApiTargetGroup
Description: ARN of the internal API target group.
Value: !Ref InternalApiTargetGroup
Description: ARN of the internal service target group.
Value: !Ref InternalServiceTargetGroup
IMPORTANT
If you are deploying your cluster to an AWS government or secret region, you must
update the InternalApiServerRecord to use CNAME records. Records of type ALIAS
are not supported for AWS government regions. For example:
Type: CNAME
TTL: 10
ResourceRecords:
- !GetAtt IntApiElb.DNSName
public hosted zones.
6.13.11. Creating security group and roles in AWS

You must create security groups and roles in Amazon Web Services (AWS) for your OpenShift Container
Platform cluster to use.
AWS resources. The stack represents the security groups and roles that your OpenShift Container
Platform cluster requires.
NOTE
735
Prerequisites
Procedure
[
{
},
{
},
{
},
{
}
]
cluster.
4 Specify the CIDR block parameter that you used for the VPC that you defined in the form
x.x.x.x/16-24.
the VPC.
736
topic and save it as a YAML file on your computer. This template describes the security groups
and roles that your cluster requires.
security groups and roles:
IMPORTANT

1 <name> is the name for the CloudFormation stack, such as cluster-sec. You need the
that you saved.
file.

resources.
Example output
arn:aws:cloudformation:us-east-1:269333783861:stack/cluster-sec/03bd4210-2ed7-11eb-
6d7a-13fc0b61e9db
MasterSec Master Security Group ID

urityGrou
pId
WorkerSe Worker Security Group ID

curityGro
upId
737
MasterIns Master IAM Instance Profile

tanceProfi
le
WorkerIns Worker IAM Instance Profile

tanceProfi
le
6.13.11.1. CloudFormation template for security objects
You can use the following CloudFormation template to deploy the security objects that you need for
Example 6.87. CloudFormation template for security objects
Description: Template for OpenShift Cluster Security Elements (Security Groups & IAM)
Parameters:
InfrastructureName:
MaxLength: 27
MinLength: 1
Type: String
VpcCidr:
AllowedPattern: ^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-
4][0-9]|25[0-5])(\/(1[6-9]|2[0-4]))$
Default: 10.0.0.0/16
Type: String
VpcId:
PrivateSubnets:
Metadata:
ParameterGroups:
- Label:
Parameters:
- Label:
Parameters:
738
- VpcId
- VpcCidr
- PrivateSubnets
ParameterLabels:
InfrastructureName:
VpcId:
default: "VPC ID"
VpcCidr:
default: "VPC CIDR"
PrivateSubnets:
Resources:
MasterSecurityGroup:
Properties:
GroupDescription: Cluster Master Security Group
- IpProtocol: icmp
FromPort: 0
ToPort: 0
- IpProtocol: tcp
FromPort: 22
ToPort: 22
- IpProtocol: tcp
ToPort: 6443
FromPort: 6443
- IpProtocol: tcp
FromPort: 22623
ToPort: 22623
VpcId: !Ref VpcId
WorkerSecurityGroup:
Properties:
GroupDescription: Cluster Worker Security Group
- IpProtocol: icmp
FromPort: 0
ToPort: 0
- IpProtocol: tcp
FromPort: 22
ToPort: 22
VpcId: !Ref VpcId
MasterIngressEtcd:
Properties:
739

Description: etcd
FromPort: 2379
ToPort: 2380
IpProtocol: tcp
MasterIngressVxlan:
Properties:
FromPort: 4789
ToPort: 4789
IpProtocol: udp
MasterIngressWorkerVxlan:
Properties:
FromPort: 4789
ToPort: 4789
IpProtocol: udp
MasterIngressGeneve:
Properties:
FromPort: 6081
ToPort: 6081
IpProtocol: udp
MasterIngressWorkerGeneve:
Properties:
FromPort: 6081
ToPort: 6081
IpProtocol: udp
MasterIngressIpsecIke:
Properties:
FromPort: 500
ToPort: 500
IpProtocol: udp
740
MasterIngressIpsecNat:
Properties:
FromPort: 4500
ToPort: 4500
IpProtocol: udp
MasterIngressIpsecEsp:
Properties:
IpProtocol: 50
MasterIngressWorkerIpsecIke:
Properties:
FromPort: 500
ToPort: 500
IpProtocol: udp
MasterIngressWorkerIpsecNat:
Properties:
FromPort: 4500
ToPort: 4500
IpProtocol: udp
MasterIngressWorkerIpsecEsp:
Properties:
IpProtocol: 50
MasterIngressInternal:
Properties:
FromPort: 9000
ToPort: 9999
IpProtocol: tcp
741
MasterIngressWorkerInternal:
Properties:
FromPort: 9000
ToPort: 9999
IpProtocol: tcp
MasterIngressInternalUDP:
Properties:
FromPort: 9000
ToPort: 9999
IpProtocol: udp
MasterIngressWorkerInternalUDP:
Properties:
FromPort: 9000
ToPort: 9999
IpProtocol: udp
MasterIngressKube:
Properties:
FromPort: 10250
ToPort: 10259
IpProtocol: tcp
MasterIngressWorkerKube:
Properties:
FromPort: 10250
ToPort: 10259
IpProtocol: tcp
MasterIngressIngressServices:
Properties:
742
FromPort: 30000
ToPort: 32767
IpProtocol: tcp
MasterIngressWorkerIngressServices:
Properties:
FromPort: 30000
ToPort: 32767
IpProtocol: tcp
MasterIngressIngressServicesUDP:
Properties:
FromPort: 30000
ToPort: 32767
IpProtocol: udp
MasterIngressWorkerIngressServicesUDP:
Properties:
FromPort: 30000
ToPort: 32767
IpProtocol: udp
WorkerIngressVxlan:
Properties:
FromPort: 4789
ToPort: 4789
IpProtocol: udp
WorkerIngressMasterVxlan:
Properties:
FromPort: 4789
ToPort: 4789
IpProtocol: udp
WorkerIngressGeneve:
743
Properties:
FromPort: 6081
ToPort: 6081
IpProtocol: udp
WorkerIngressMasterGeneve:
Properties:
FromPort: 6081
ToPort: 6081
IpProtocol: udp
WorkerIngressIpsecIke:
Properties:
FromPort: 500
ToPort: 500
IpProtocol: udp
WorkerIngressIpsecNat:
Properties:
FromPort: 4500
ToPort: 4500
IpProtocol: udp
WorkerIngressIpsecEsp:
Properties:
IpProtocol: 50
WorkerIngressMasterIpsecIke:
Properties:
FromPort: 500
ToPort: 500
IpProtocol: udp
744
WorkerIngressMasterIpsecNat:
Properties:
FromPort: 4500
ToPort: 4500
IpProtocol: udp
WorkerIngressMasterIpsecEsp:
Properties:
IpProtocol: 50
WorkerIngressInternal:
Properties:
FromPort: 9000
ToPort: 9999
IpProtocol: tcp
WorkerIngressMasterInternal:
Properties:
FromPort: 9000
ToPort: 9999
IpProtocol: tcp
WorkerIngressInternalUDP:
Properties:
FromPort: 9000
ToPort: 9999
IpProtocol: udp
WorkerIngressMasterInternalUDP:
Properties:
FromPort: 9000
ToPort: 9999
745
IpProtocol: udp
WorkerIngressKube:
Properties:
Description: Kubernetes secure kubelet port
FromPort: 10250
ToPort: 10250
IpProtocol: tcp
WorkerIngressWorkerKube:
Properties:
Description: Internal Kubernetes communication
FromPort: 10250
ToPort: 10250
IpProtocol: tcp
WorkerIngressIngressServices:
Properties:
FromPort: 30000
ToPort: 32767
IpProtocol: tcp
WorkerIngressMasterIngressServices:
Properties:
FromPort: 30000
ToPort: 32767
IpProtocol: tcp
WorkerIngressIngressServicesUDP:
Properties:
FromPort: 30000
ToPort: 32767
IpProtocol: udp
WorkerIngressMasterIngressServicesUDP:
Properties:
746

FromPort: 30000
ToPort: 32767
IpProtocol: udp
MasterIamRole:
Properties:
Version: "2012-10-17"
Statement:
- Effect: "Allow"
Principal:
Service:
Action:
- "sts:AssumeRole"
Policies:
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: "Allow"
Action:
- "ec2:AttachVolume"
- "ec2:AuthorizeSecurityGroupIngress"
- "ec2:CreateSecurityGroup"
- "ec2:CreateTags"
- "ec2:CreateVolume"
- "ec2:DeleteSecurityGroup"
- "ec2:DeleteVolume"
- "ec2:Describe*"
- "ec2:DetachVolume"
- "ec2:ModifyInstanceAttribute"
- "ec2:ModifyVolume"
- "ec2:RevokeSecurityGroupIngress"
- "elasticloadbalancing:AddTags"
- "elasticloadbalancing:AttachLoadBalancerToSubnets"
- "elasticloadbalancing:ApplySecurityGroupsToLoadBalancer"
- "elasticloadbalancing:CreateListener"
- "elasticloadbalancing:CreateLoadBalancer"
- "elasticloadbalancing:CreateLoadBalancerPolicy"
- "elasticloadbalancing:CreateLoadBalancerListeners"
- "elasticloadbalancing:CreateTargetGroup"
- "elasticloadbalancing:ConfigureHealthCheck"
- "elasticloadbalancing:DeleteListener"
- "elasticloadbalancing:DeleteLoadBalancer"
- "elasticloadbalancing:DeleteLoadBalancerListeners"
- "elasticloadbalancing:DeleteTargetGroup"
- "elasticloadbalancing:DeregisterInstancesFromLoadBalancer"
- "elasticloadbalancing:DeregisterTargets"
- "elasticloadbalancing:Describe*"
- "elasticloadbalancing:DetachLoadBalancerFromSubnets"
- "elasticloadbalancing:ModifyListener"
- "elasticloadbalancing:ModifyLoadBalancerAttributes"
747
- "elasticloadbalancing:ModifyTargetGroup"
- "elasticloadbalancing:ModifyTargetGroupAttributes"
- "elasticloadbalancing:RegisterInstancesWithLoadBalancer"
- "elasticloadbalancing:RegisterTargets"
- "elasticloadbalancing:SetLoadBalancerPoliciesForBackendServer"
- "elasticloadbalancing:SetLoadBalancerPoliciesOfListener"
- "kms:DescribeKey"
Resource: "*"
Properties:
Roles:
- Ref: "MasterIamRole"
WorkerIamRole:
Properties:
Version: "2012-10-17"
Statement:
- Effect: "Allow"
Principal:
Service:
Action:
- "sts:AssumeRole"
Policies:
- PolicyName: !Join ["-", [!Ref InfrastructureName, "worker", "policy"]]
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: "Allow"
Action:
- "ec2:DescribeInstances"
- "ec2:DescribeRegions"
Resource: "*"
Properties:
Roles:
- Ref: "WorkerIamRole"
Outputs:
Description: Master Security Group ID
Value: !GetAtt MasterSecurityGroup.GroupId
Description: Worker Security Group ID
Value: !GetAtt WorkerSecurityGroup.GroupId
Description: Master IAM Instance Profile
Value: !Ref MasterInstanceProfile
748
Description: Worker IAM Instance Profile
Value: !Ref WorkerInstanceProfile
6.13.12. Accessing RHCOS AMIs with stream metadata

In OpenShift Container Platform, stream metadata provides standardized metadata about RHCOS in
the JSON format and injects the metadata into the cluster. Stream metadata is a stable format that
supports multiple architectures and is intended to be self-documenting for maintaining automation.
You can use the coreos print-stream-json sub-command of openshift-install to access information
about the boot images in the stream metadata format. This command provides a method for printing
stream metadata in a scriptable, machine-readable format.
For user-provisioned installations, the openshift-install binary contains references to the version of
RHCOS boot images that are tested for use with OpenShift Container Platform, such as the AWS AMI.
Procedure
To parse the stream metadata, use one of the following methods:
From a Go program, use the official stream-metadata-go library at

https://github.com/coreos/stream-metadata-go. You can also view example code in the library.
From another programming language, such as Python or Ruby, use the JSON library of your
preferred programming language.
From a command-line utility that handles JSON data, such as jq:
Print the current x86_64 or aarch64 AMI for an AWS region, such as us-west-1:
For x86_64

'.architectures.x86_64.images.aws.regions["us-west-1"].image'
Example output
ami-0d3e625f84626bbda
For aarch64

'.architectures.aarch64.images.aws.regions["us-west-1"].image'
Example output
ami-0af1d3b7fa5be2131
The output of this command is the AWS AMI ID for your designated architecture and the us-
west-1 region. The AMI must belong to the same region as the cluster.
749
6.13.13. RHCOS AMIs for the AWS infrastructure

Red Hat provides Red Hat Enterprise Linux CoreOS (RHCOS) AMIs that are valid for the various AWS
regions and instance architectures that you can manually specify for your OpenShift Container Platform
nodes.
NOTE
By importing your own AMI, you can also install to regions that do not have a published
RHCOS AMI.
Table 6.21. x86_64 RHCOS AMIs
AWS zone AWS AMI
af-south-1 ami-0493ec0f0a451f83b
ap-east-1 ami-050a6d164705e7f62
ap-northeast-1 ami-00910c337e0f52cff
ap-northeast-2 ami-07e98d33de2b93ac0
ap-northeast-3 ami-09bc0a599f4b3c483
ap-south-1 ami-0ba603a7f9d41228e
ap-south-2 ami-03130aecb5d7459cc
ap-southeast-1 ami-026c056e0a25e5a04
ap-southeast-2 ami-0d471f504ff6d9a0f
ap-southeast-3 ami-0c1b9a0721cbb3291
ap-southeast-4 ami-0ef23bfe787efe11e
ca-central-1 ami-0163965a05b75f976
eu-central-1 ami-01edb54011f870f0c
eu-central-2 ami-0bc500d6056a3b104
eu-north-1 ami-0ab155e935177f16a
eu-south-1 ami-051b4c06b21f5a328
eu-south-2 ami-096644e5555c23b19
750
AWS zone AWS AMI
eu-west-1 ami-0faeeeb3d2b1aa07c
eu-west-2 ami-00bb1522dc71b604f
eu-west-3 ami-01e5397bd2b795bd3
il-central-1 ami-0b32feb5d77c64e61
me-central-1 ami-0a5158a3e68ab7e88
me-south-1 ami-024864ad1b799dbba
sa-east-1 ami-0c402ffb0c4b7edc0
us-east-1 ami-057df4d0cb8cbae0d
us-east-2 ami-07566e5da1fd297f8
us-gov-east-1 ami-0fe03a7e289354670
us-gov-west-1 ami-06b7cc6445c5da732
us-west-1 ami-02d20001c5b9df1e9
us-west-2 ami-0dfba457127fba98c
Table 6.22. aarch64 RHCOS AMIs
AWS zone AWS AMI
af-south-1 ami-06c7b4e42179544df
ap-east-1 ami-07b6a37fa6d2d2e99
ap-northeast-1 ami-056d2eef4a3638246
ap-northeast-2 ami-0bd5a7684f0ff4e02
ap-northeast-3 ami-0fd08063da50de1da
ap-south-1 ami-08f1ae2cef8f9690e
ap-south-2 ami-020ba25cc1ec53b1c
751
AWS zone AWS AMI
ap-southeast-1 ami-0020a1c0964ac8e48
ap-southeast-2 ami-07013a63289350c3c
ap-southeast-3 ami-041d6ca1d57e3190f
ap-southeast-4 ami-06539e9cbefc28702
ca-central-1 ami-0bb3991641f2b40f6
eu-central-1 ami-0908d117c26059e39
eu-central-2 ami-0e48c82ffbde67ed2
eu-north-1 ami-016614599b38d515e
eu-south-1 ami-01b6cc1f0fd7b431f
eu-south-2 ami-0687e1d98e55e402d
eu-west-1 ami-0bf0b7b1cb052d68d
eu-west-2 ami-0ba0bf567caa63731
eu-west-3 ami-0eab6a7956a66deda
il-central-1 ami-03b3cb1f4869bf21d
me-central-1 ami-0a6e1ade3c9e206a1
me-south-1 ami-0aa0775c68eac9f6f
sa-east-1 ami-07235eee0bb930c78
us-east-1 ami-005808ca73e7b36ff
us-east-2 ami-0c5c9420f6b992e9e
us-gov-east-1 ami-08c9b2b8d578caf92
us-gov-west-1 ami-0bdff65422ba7d95d
us-west-1 ami-017ad4dd030a04233
us-west-2 ami-068d0af5e3c08e618
752
6.13.14. Creating the bootstrap node in AWS

You must create the bootstrap node in Amazon Web Services (AWS) to use during OpenShift Container
Platform cluster initialization. You do this by:
Providing a location to serve the bootstrap.ign Ignition config file to your cluster. This file is
located in your installation directory. The provided CloudFormation Template assumes that the
Ignition config files for your cluster are served from an S3 bucket. If you choose to serve the
files from another location, you must modify the templates.
Using the provided CloudFormation template and a custom parameter file to create a stack of
AWS resources. The stack represents the bootstrap node that your OpenShift Container
Platform installation requires.
NOTE
If you do not use the provided CloudFormation template to create your bootstrap node,
installation logs.
Prerequisites
Procedure
1. Create the bucket by running the following command:
$ aws s3 mb s3://<cluster-name>-infra 1
1 <cluster-name>-infra is the bucket name. When creating the install-config.yaml file,

replace <cluster-name> with the name specified for the cluster.
You must use a presigned URL for your S3 bucket, instead of the s3:// schema, if you are:
Deploying to a region that has endpoints that differ from the AWS SDK.
Deploying a proxy.
Providing your own custom endpoints.
2. Upload the bootstrap.ign Ignition config file to the bucket by running the following command:
$ aws s3 cp <installation_directory>/bootstrap.ign s3://<cluster-name>-infra/bootstrap.ign 1
753
3. Verify that the file uploaded by running the following command:
$ aws s3 ls s3://<cluster-name>-infra/
Example output
2019-04-03 16:15:16 314878 bootstrap.ign
NOTE
The bootstrap Ignition config file does contain secrets, like X.509 keys. The
following steps provide basic security for the S3 bucket. To provide additional
security, you can enable an S3 bucket policy to allow only certain users, such as
the OpenShift IAM user, to access objects that the bucket contains. You can
avoid S3 entirely and serve your bootstrap Ignition config file from any address
that the bootstrap machine can reach.
[
{
},
{
},
{
"ParameterKey": "AllowedBootstrapSshCidr", 5
},
{
"ParameterKey": "PublicSubnet", 7
},
{
},
{
},
{
"ParameterKey": "BootstrapIgnitionLocation", 13
"ParameterValue": "s3://<bucket_name>/bootstrap.ign" 14
754
},
{
},
{
},
{
},
{
},
{
}
]
cluster.
3 Current Red Hat Enterprise Linux CoreOS (RHCOS) AMI to use for the bootstrap node
4 Specify a valid AWS::EC2::Image::Id value.
5 CIDR block to allow SSH access to the bootstrap node.
7 The public subnet that is associated with your VPC to launch the bootstrap node into.
the VPC.
9 The master security group ID (for registering temporary rules)

11 The VPC created resources will belong to.
755
13 Location to fetch bootstrap Ignition config file from.
14 Specify the S3 bucket and file name in the form s3://<bucket_name>/bootstrap.ign.
(ARN) value.




5. Copy the template from the CloudFormation template for the bootstrap machinesection of
this topic and save it as a YAML file on your computer. This template describes the bootstrap
machine that your cluster requires.
6. Optional: If you are deploying the cluster with a proxy, you must update the ignition in the
template to add the ignition.config.proxy fields. Additionally, If you have added the Amazon
EC2, Elastic Load Balancing, and S3 VPC endpoints to your VPC, you must add these endpoints
to the noProxy field.
bootstrap node:
IMPORTANT

756
1 <name> is the name for the CloudFormation stack, such as cluster-bootstrap. You need
that you saved.
file.

resources.
Example output
arn:aws:cloudformation:us-east-1:269333783861:stack/cluster-bootstrap/12944486-2add-
11eb-9dee-12dace8e3a83
Bootstrap The bootstrap Instance ID.

InstanceId
Bootstrap The bootstrap node public IP address.

PublicIp
Bootstrap The bootstrap node private IP address.

PrivateIp
6.13.14.1. CloudFormation template for the bootstrap machine
You can use the following CloudFormation template to deploy the bootstrap machine that you need for
Example 6.88. CloudFormation template for the bootstrap machine
Description: Template for OpenShift Cluster Bootstrap (EC2 Instance, Security Groups and IAM)
Parameters:
InfrastructureName:
MaxLength: 27
MinLength: 1
757
Type: String
RhcosAmi:
AllowedPattern: ^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-
4][0-9]|25[0-5])(\/([0-9]|1[0-9]|2[0-9]|3[0-2]))$
Default: 0.0.0.0/0
Description: CIDR block to allow SSH access to the bootstrap node.
Type: String
PublicSubnet:
Description: The public subnet to launch the bootstrap node into.
Description: The master security group ID for registering temporary rules.
VpcId:
Default: s3://my-s3-bucket/bootstrap.ign
Type: String
AutoRegisterELB:
Default: "yes"
AllowedValues:
- "yes"
- "no"
Type: String
Description: ARN for NLB IP target registration lambda.
Type: String
Description: ARN for external API load balancer target group.
Type: String
Description: ARN for internal API load balancer target group.
Type: String
Description: ARN for internal service load balancer target group.
Type: String
BootstrapInstanceType:
Description: Instance type for the bootstrap EC2 instance
Default: "i3.large"
Type: String
Metadata:
ParameterGroups:
- Label:
Parameters:
758
- Label:
Parameters:
- RhcosAmi
- BootstrapIgnitionLocation
- Label:
Parameters:
- VpcId
- PublicSubnet
- Label:
Parameters:
- AutoRegisterELB
ParameterLabels:
InfrastructureName:
VpcId:
default: "VPC ID"
default: "Allowed SSH Source"
PublicSubnet:
default: "Public Subnet"
RhcosAmi:
default: "Bootstrap Ignition Source"
AutoRegisterELB:
Conditions:
Resources:
BootstrapIamRole:
Properties:
Version: "2012-10-17"
Statement:
- Effect: "Allow"
Principal:
Service:
Action:
- "sts:AssumeRole"
Path: "/"
759
Policies:
- PolicyName: !Join ["-", [!Ref InfrastructureName, "bootstrap", "policy"]]
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: "Allow"
Action: "ec2:Describe*"
Resource: "*"
- Effect: "Allow"
Action: "ec2:AttachVolume"
Resource: "*"
- Effect: "Allow"
Action: "ec2:DetachVolume"
Resource: "*"
- Effect: "Allow"
Action: "s3:GetObject"
Resource: "*"
BootstrapInstanceProfile:
Properties:
Path: "/"
Roles:
- Ref: "BootstrapIamRole"
BootstrapSecurityGroup:
Properties:
GroupDescription: Cluster Bootstrap Security Group
- IpProtocol: tcp
FromPort: 22
ToPort: 22
CidrIp: !Ref AllowedBootstrapSshCidr
- IpProtocol: tcp
ToPort: 19531
FromPort: 19531
CidrIp: 0.0.0.0/0
VpcId: !Ref VpcId
BootstrapInstance:
Properties:
IamInstanceProfile: !Ref BootstrapInstanceProfile
InstanceType: !Ref BootstrapInstanceType
NetworkInterfaces:
- AssociatePublicIpAddress: "true"
DeviceIndex: "0"
GroupSet:
- !Ref "BootstrapSecurityGroup"
SubnetId: !Ref "PublicSubnet"
UserData:
Fn::Base64: !Sub
- '{"ignition":{"config":{"replace":{"source":"${S3Loc}"}},"version":"3.1.0"}}'
760
-{
S3Loc: !Ref BootstrapIgnitionLocation
}
RegisterBootstrapApiTarget:
Properties:
RegisterBootstrapInternalApiTarget:
Properties:
RegisterBootstrapInternalServiceTarget:
Properties:
Outputs:
BootstrapInstanceId:
Description: Bootstrap Instance ID.
Value: !Ref BootstrapInstance
BootstrapPublicIp:
Description: The bootstrap node public IP address.
Value: !GetAtt BootstrapInstance.PublicIp
BootstrapPrivateIp:
Description: The bootstrap node private IP address.
Value: !GetAtt BootstrapInstance.PrivateIp
See RHCOS AMIs for the AWS infrastructure for details about the Red Hat Enterprise Linux
CoreOS (RHCOS) AMIs for the AWS zones.
6.13.15. Creating the control plane machines in AWS

You must create the control plane machines in Amazon Web Services (AWS) that your cluster will use.
AWS resources that represent the control plane nodes.
IMPORTANT
761
IMPORTANT
The CloudFormation template creates a stack that represents three control plane nodes.
NOTE
If you do not use the provided CloudFormation template to create your control plane
nodes, you must review the provided information and manually create the infrastructure.
If your cluster does not initialize correctly, you might have to contact Red Hat support
with your installation logs.
Prerequisites
Procedure
[
{
},
{
},
{
"ParameterKey": "AutoRegisterDNS", 5
},
{
"ParameterKey": "PrivateHostedZoneId", 7
},
{
"ParameterKey": "PrivateHostedZoneName", 9
"ParameterValue": "mycluster.example.com" 10
},
{
762
},
{
},
{
},
{
},
{
"ParameterValue": "https://api-int.<cluster_name>.<domain_name>:22623/config/master"
20
},
{
"ParameterValue": "data:text/plain;charset=utf-8;base64,ABC...xYz==" 22
},
{
"ParameterKey": "MasterInstanceProfileName", 23
"ParameterValue": "<roles_stack>-MasterInstanceProfile-<random_string>" 24
},
{
"ParameterKey": "MasterInstanceType", 25
},
{
},
{
},
{
},
{
},
{
763
}
]
cluster.
3 Current Red Hat Enterprise Linux CoreOS (RHCOS) AMI to use for the control plane
machines based on your selected architecture.
5 Whether or not to perform DNS etcd registration.
6 Specify yes or no. If you specify yes, you must provide hosted zone information.
7 The Route 53 private zone ID to register the etcd targets with.
8 Specify the PrivateHostedZoneId value from the output of the CloudFormation template
for DNS and load balancing.
10 Specify <cluster_name>.<domain_name> where <domain_name> is the Route 53 base

domain that you used when you generated install-config.yaml file for the cluster. Do not
include the trailing period (.) that is displayed in the AWS console.
11 13 15 A subnet, preferably private, to launch the control plane machines on.
12 14 16 Specify a subnet from the PrivateSubnets value from the output of the
CloudFormation template for DNS and load balancing.
17 The master security group ID to associate with control plane nodes.

19 The location to fetch control plane Ignition config file from.
20 Specify the generated Ignition config file location, https://api-int.<cluster_name>.

<domain_name>:22623/config/master.
21 The base64 encoded certificate authority string to use.
22 Specify the value from the master.ign file that is in the installation directory. This value is
23 The IAM profile to associate with control plane nodes.
24 Specify the MasterInstanceProfile parameter value from the output of the

25 The type of AWS instance to use for the control plane machines based on your selected
architecture.
764
26 The instance type value corresponds to the minimum resource requirements for control
plane machines. For example m6i.xlarge is a type for AMD64 and m6g.xlarge is a type for
(ARN) value.




2. Copy the template from the CloudFormation template for control plane machinessection of
this topic and save it as a YAML file on your computer. This template describes the control plane
3. If you specified an m5 instance type as the value for MasterInstanceType, add that instance
type to the MasterInstanceType.AllowedValues parameter in the CloudFormation template.
control plane nodes:
IMPORTANT

1 <name> is the name for the CloudFormation stack, such as cluster-control-plane. You
need the name of this stack if you remove the cluster.
765
file.
Example output
arn:aws:cloudformation:us-east-1:269333783861:stack/cluster-control-plane/21c7e2b0-2ee2-
11eb-c6f6-0aa34627df4b
NOTE
The CloudFormation template creates a stack that represents three control plane
nodes.
6.13.15.1. CloudFormation template for control plane machines
You can use the following CloudFormation template to deploy the control plane machines that you need
Example 6.89. CloudFormation template for control plane machines
Description: Template for OpenShift Cluster Node Launch (EC2 master instances)
Parameters:
InfrastructureName:
MaxLength: 27
MinLength: 1
Type: String
RhcosAmi:
AutoRegisterDNS:
Default: ""
Description: unused
Type: String
Default: ""
Description: unused
Type: String
PrivateHostedZoneName:
Default: ""
Description: unused
Type: String
Master0Subnet:
766
Master1Subnet:
Master2Subnet:
Description: The master security group ID to associate with master nodes.
IgnitionLocation:
Default: https://api-int.$CLUSTER_NAME.$DOMAIN:22623/config/master
Type: String
Type: String
Description: IAM profile to associate with master nodes.
Type: String
MasterInstanceType:
Default: m5.xlarge
Type: String
AutoRegisterELB:
Default: "yes"
AllowedValues:
- "yes"
- "no"
Type: String
Description: ARN for NLB IP target registration lambda. Supply the value from the cluster
Type: String
Description: ARN for external API load balancer target group. Supply the value from the cluster
Type: String
Description: ARN for internal API load balancer target group. Supply the value from the cluster
Type: String
Description: ARN for internal service load balancer target group. Supply the value from the
cluster infrastructure or select "no" for AutoRegisterELB.
Type: String
Metadata:
ParameterGroups:
- Label:
Parameters:
767
- Label:
Parameters:
- MasterInstanceType
- RhcosAmi
- IgnitionLocation
- MasterInstanceProfileName
- Label:
Parameters:
- VpcId
- Master0Subnet
- Master1Subnet
- Master2Subnet
- Label:
Parameters:
- AutoRegisterELB
ParameterLabels:
InfrastructureName:
VpcId:
default: "VPC ID"
Master0Subnet:
Master1Subnet:
Master2Subnet:
MasterInstanceType:
default: "Master Instance Type"
default: "Master Instance Profile Name"
RhcosAmi:
default: "Master Ignition Source"
AutoRegisterELB:
Conditions:
Resources:
Master0:
768
Properties:
Ebs:
VolumeSize: "120"
VolumeType: "gp2"
NetworkInterfaces:
DeviceIndex: "0"
GroupSet:
UserData:
Fn::Base64: !Sub
-{
}
Tags:
Value: "shared"
RegisterMaster0:
Properties:
Properties:
Properties:
Master1:
Properties:
769
Ebs:
VolumeSize: "120"
VolumeType: "gp2"
NetworkInterfaces:
DeviceIndex: "0"
GroupSet:
UserData:
Fn::Base64: !Sub
-{
}
Tags:
Value: "shared"
RegisterMaster1:
Properties:
Properties:
Properties:
Master2:
Properties:
Ebs:
770
VolumeSize: "120"
VolumeType: "gp2"
NetworkInterfaces:
DeviceIndex: "0"
GroupSet:
UserData:
Fn::Base64: !Sub
-{
}
Tags:
Value: "shared"
RegisterMaster2:
Properties:
Properties:
Properties:
Outputs:
PrivateIPs:
Description: The control-plane node private IP addresses.
Value:
!Join [
",",
[!GetAtt Master0.PrivateIp, !GetAtt Master1.PrivateIp, !GetAtt Master2.PrivateIp]
]
771
6.13.16. Creating the worker nodes in AWS

You can create worker nodes in Amazon Web Services (AWS) for your cluster to use.
AWS resources that represent a worker node.
IMPORTANT
The CloudFormation template creates a stack that represents one worker node. You
must create a stack for each worker node.
NOTE
If you do not use the provided CloudFormation template to create your worker nodes,
installation logs.
Prerequisites
Procedure
requires:
[
{
},
{
},
{
"ParameterKey": "Subnet", 5
},
772
{
"ParameterKey": "WorkerSecurityGroupId", 7
},
{
"ParameterValue": "https://api-int.<cluster_name>.<domain_name>:22623/config/worker"
10
},
{
},
{
"ParameterKey": "WorkerInstanceProfileName", 13
},
{
"ParameterKey": "WorkerInstanceType", 15
}
]
cluster.
3 Current Red Hat Enterprise Linux CoreOS (RHCOS) AMI to use for the worker nodes
5 A subnet, preferably private, to start the worker nodes on.
6 Specify a subnet from the PrivateSubnets value from the output of the CloudFormation
7 The worker security group ID to associate with worker nodes.
8 Specify the WorkerSecurityGroupId value from the output of the CloudFormation

9 The location to fetch the bootstrap Ignition config file from.
10 Specify the generated Ignition config location, https://api-int.<cluster_name>.

<domain_name>:22623/config/worker.
11 Base64 encoded certificate authority string to use.
12 Specify the value from the worker.ign file that is in the installation directory. This value is
13 The IAM profile to associate with worker nodes.
773
14 Specify the WorkerInstanceProfile parameter value from the output of the

15 The type of AWS instance to use for the compute machines based on your selected
architecture.
16 The instance type value corresponds to the minimum resource requirements for compute
machines. For example m6i.large is a type for AMD64 and m6g.large is a type for ARM64.
2. Copy the template from the CloudFormation template for worker machines section of this
topic and save it as a YAML file on your computer. This template describes the networking
objects and load balancers that your cluster requires.
3. Optional: If you specified an m5 instance type as the value for WorkerInstanceType, add that
instance type to the WorkerInstanceType.AllowedValues parameter in the CloudFormation
template.
4. Optional: If you are deploying with an AWS Marketplace image, update the
Worker0.type.properties.ImageID parameter with the AMI ID that you obtained from your
subscription.
5. Use the CloudFormation template to create a stack of AWS resources that represent a worker
node:
IMPORTANT

1 <name> is the name for the CloudFormation stack, such as cluster-worker-1. You need
that you saved.
file.
Example output
arn:aws:cloudformation:us-east-1:269333783861:stack/cluster-worker-1/729ee301-1c2a-
11eb-348f-sd9888c65b59
NOTE
The CloudFormation template creates a stack that represents one worker node.
774
7. Continue to create worker stacks until you have created enough worker machines for your
cluster. You can create additional worker stacks by referencing the same template and
parameter files and specifying a different stack name.
IMPORTANT
You must create at least two worker machines, so you must create at least two
stacks that use this CloudFormation template.
6.13.16.1. CloudFormation template for worker machines
You can use the following CloudFormation template to deploy the worker machines that you need for
Example 6.90. CloudFormation template for worker machines
Description: Template for OpenShift Cluster Node Launch (EC2 worker instance)
Parameters:
InfrastructureName:
MaxLength: 27
MinLength: 1
Type: String
RhcosAmi:
Subnet:
Description: The subnets, recommend private, to launch the worker nodes into.
Description: The worker security group ID to associate with worker nodes.
IgnitionLocation:
Default: https://api-int.$CLUSTER_NAME.$DOMAIN:22623/config/worker
Type: String
Type: String
Description: IAM profile to associate with worker nodes.
Type: String
WorkerInstanceType:
Default: m5.large
Type: String
775
Metadata:
ParameterGroups:
- Label:
Parameters:
- Label:
Parameters:
- WorkerInstanceType
- RhcosAmi
- IgnitionLocation
- WorkerSecurityGroupId
- WorkerInstanceProfileName
- Label:
Parameters:
- Subnet
ParameterLabels:
Subnet:
default: "Subnet"
InfrastructureName:
WorkerInstanceType:
default: "Worker Instance Type"
default: "Worker Instance Profile Name"
RhcosAmi:
IgnitionLocation:
default: "Worker Ignition Source"
default: "Worker Security Group ID"
Resources:
Worker0:
Properties:
Ebs:
VolumeSize: "120"
VolumeType: "gp2"
IamInstanceProfile: !Ref WorkerInstanceProfileName
InstanceType: !Ref WorkerInstanceType
NetworkInterfaces:
DeviceIndex: "0"
GroupSet:
- !Ref "WorkerSecurityGroupId"
SubnetId: !Ref "Subnet"
776
UserData:
Fn::Base64: !Sub
-{
}
Tags:
Value: "shared"
Outputs:
PrivateIP:
Description: The compute node private IP address.
Value: !GetAtt Worker0.PrivateIp
6.13.17. Initializing the bootstrap sequence on AWS with user-provisioned

infrastructure
After you create all of the required infrastructure in Amazon Web Services (AWS), you can start the
bootstrap sequence that initializes the OpenShift Container Platform control plane.
Prerequisites
You created the worker nodes.
Procedure
1. Change to the directory that contains the installation program and start the bootstrap process
that initializes the OpenShift Container Platform control plane:

--log-level=info 2
777
Example output
INFO Waiting up to 20m0s for the Kubernetes API at

https://api.mycluster.example.com:6443...
INFO API v1.28.5 up
INFO Waiting up to 30m0s for bootstrapping to complete...
INFO It is now safe to remove the bootstrap resources
If the command exits without a FATAL warning, your OpenShift Container Platform control
plane has initialized.
NOTE
After the control plane initializes, it sets up the compute nodes and installs
additional services in the form of Operators.
See Monitoring installation progress for details about monitoring the installation, bootstrap, and
control plane logs as an OpenShift Container Platform installation progresses.
See Gathering bootstrap node diagnostic data for information about troubleshooting issues
related to the bootstrap process.

Prerequisites
Procedure
$ oc whoami
778
Example output
system:admin

Prerequisites
Procedure
$ oc get nodes
Example output

NOTE
$ oc get csr
Example output

...
list.
779
NOTE
NOTE
of the node.

NOTE
$ oc get csr
Example output

Pending
780

Pending
...

$ oc get nodes
Example output

NOTE
6.13.20. Initial Operator configuration

become available.
Prerequisites
Your control plane has initialized.
Procedure
781
1. Watch the cluster components come online:
$ watch -n5 oc get clusteroperators
Example output

SINCE
authentication 4.15.0 True False False 19m
baremetal 4.15.0 True False False 37m
config-operator 4.15.0 True False False 38m
etcd 4.15.0 True False False 36m
machine-approver 4.15.0 True False False 37m
2. Configure the Operators that are not available.
6.13.20.1. Disabling the default OperatorHub catalog sources
Procedure
OperatorHub object:
782

TIP
6.13.20.2. Image registry storage configuration
after installation. However, if the Registry Operator cannot create an S3 bucket and automatically
configure storage, you must manually configure registry storage.
Instructions are shown for configuring a persistent volume, which is required for production clusters.
Where applicable, instructions are shown for configuring an empty directory as the storage location,
which is available for only non-production clusters.
Additional instructions are provided for allowing the image registry to use block storage types by using
the Recreate rollout strategy during upgrades.
6.13.20.2.1. Configuring registry storage for AWS with user-provisioned infrastructure
During installation, your cloud credentials are sufficient to create an Amazon S3 bucket and the Registry
Operator will automatically configure storage.
If the Registry Operator cannot create an S3 bucket and automatically configure storage, you can
create an S3 bucket and configure storage with the following procedure.
Prerequisites
You have a cluster on AWS with user-provisioned infrastructure.
For Amazon S3 storage, the secret is expected to contain two keys:
REGISTRY_STORAGE_S3_ACCESSKEY
REGISTRY_STORAGE_S3_SECRETKEY
Procedure
Use the following procedure if the Registry Operator cannot create an S3 bucket and automatically
configure storage.
1. Set up a Bucket Lifecycle Policy to abort incomplete multipart uploads that are one day old.
2. Fill in the storage configuration in configs.imageregistry.operator.openshift.io/cluster:
$ oc edit configs.imageregistry.operator.openshift.io/cluster
Example configuration
storage:
s3:
783
bucket: <bucket-name>
region: <region-name>

WARNING
To secure your registry images in AWS, block public access to the S3 bucket.
6.13.20.2.2. Configuring storage for the image registry in non-production clusters
You must configure storage for the Image Registry Operator. For non-production clusters, you can set
the image registry to an empty directory. If you do so, all images are lost if you restart the registry.
Procedure
To set the image registry storage to an empty directory:
$ oc patch configs.imageregistry.operator.openshift.io cluster --type merge --patch '{"spec":

{"storage":{"emptyDir":{}}}}'

WARNING
Configure this option for only non-production clusters.
If you run this command before the Image Registry Operator initializes its components, the oc
patch command fails with the following error:
Error from server (NotFound): configs.imageregistry.operator.openshift.io "cluster" not found
Wait a few minutes and run the command again.
6.13.21. Deleting the bootstrap resources

After you complete the initial Operator configuration for the cluster, remove the bootstrap resources
from Amazon Web Services (AWS).
Prerequisites
You completed the initial Operator configuration for your cluster.
Procedure
1. Delete the bootstrap resources. If you used the CloudFormation template, delete its stack :
Delete the stack by using the AWS CLI:
784
$ aws cloudformation delete-stack --stack-name <name> 1
1 <name> is the name of your bootstrap stack.
Delete the stack by using the AWS CloudFormation console.
6.13.22. Creating the Ingress DNS Records

If you removed the DNS Zone configuration, manually create DNS records that point to the Ingress load
balancer. You can create either a wildcard record or specific records. While the following procedure uses
A records, you can use other record types that you require, such as CNAME or alias.
Prerequisites
You deployed an OpenShift Container Platform cluster on Amazon Web Services (AWS) that
uses infrastructure that you provisioned.
You installed the OpenShift CLI (oc).
the Bundled Installer (Linux, macOS, or Unix).
Procedure
1. Determine the routes to create.
To create a wildcard record, use *.apps.<cluster_name>.<domain_name>, where

<cluster_name> is your cluster name, and <domain_name> is the Route 53 base domain
To create specific records, you must create a record for each route that your cluster uses, as
shown in the output of the following command:

Example output
oauth-openshift.apps.<cluster_name>.<domain_name>
console-openshift-console.apps.<cluster_name>.<domain_name>
downloads-openshift-console.apps.<cluster_name>.<domain_name>
alertmanager-main-openshift-monitoring.apps.<cluster_name>.<domain_name>
prometheus-k8s-openshift-monitoring.apps.<cluster_name>.<domain_name>
2. Retrieve the Ingress Operator load balancer status and note the value of the external IP address
that it uses, which is shown in the EXTERNAL-IP column:
Example output
785
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)

AGE
router-default LoadBalancer 172.30.62.215 ab3...28.us-east-2.elb.amazonaws.com
80:31499/TCP,443:30693/TCP 5m
3. Locate the hosted zone ID for the load balancer:
$ aws elb describe-load-balancers | jq -r '.LoadBalancerDescriptions[] | select(.DNSName ==

"<external_ip>").CanonicalHostedZoneNameID' 1
load balancer that you obtained.
Example output
Z3AADJGX6KTTL2
The output of this command is the load balancer hosted zone ID.
4. Obtain the public hosted zone ID for your cluster’s domain:
$ aws route53 list-hosted-zones-by-name \

--dns-name "<domain_name>" \ 1
--query 'HostedZones[? Config.PrivateZone != `true` && Name ==
`<domain_name>.`].Id' 2
--output text
1 2 For <domain_name>, specify the Route 53 base domain for your OpenShift Container
Platform cluster.
Example output
/hostedzone/Z3URY6TWQ91KVV
The public hosted zone ID for your domain is shown in the command output. In this example, it is
Z3URY6TWQ91KVV.
5. Add the alias records to your private zone:
$ aws route53 change-resource-record-sets --hosted-zone-id "<private_hosted_zone_id>" --

change-batch '{ 1
> "Changes": [
> {
> "Type": "A",
> "AliasTarget":{
786
> }
> }
> }
> ]
> }'
1 For <private_hosted_zone_id>, specify the value from the output of the CloudFormation
obtained.
6. Add the records to your public zone:
$ aws route53 change-resource-record-sets --hosted-zone-id "<public_hosted_zone_id>"" --

change-batch '{ 1
> "Changes": [
> {
> "Type": "A",
> "AliasTarget":{
> }
> }
> }
> ]
> }'
1 For <public_hosted_zone_id>, specify the public hosted zone for your domain.
obtained.
6.13.23. Completing an AWS installation on user-provisioned infrastructure

After you start the OpenShift Container Platform installation on Amazon Web Service (AWS) user-
provisioned infrastructure, monitor the deployment to completion.
787
Prerequisites
You removed the bootstrap node for an OpenShift Container Platform cluster on user-
provisioned AWS infrastructure.
Procedure
1. From the directory that contains the installation program, complete the cluster installation:
Example output
INFO Waiting up to 40m0s for the cluster at https://api.mycluster.example.com:6443 to

initialize...
INFO Waiting up to 10m0s for the openshift-console route to be created...
IMPORTANT
installation.
2. Register your cluster on the Cluster registration page.

Prerequisites
788
Prerequisites
Procedure
installation host:
NOTE

NOTE
Example output


789

6.13.27. Next steps

If necessary, see Registering your disconnected cluster
6.14. INSTALLING A CLUSTER ON AWS WITH COMPUTE NODES ON

AWS LOCAL ZONES
You can quickly install an OpenShift Container Platform cluster on Amazon Web Services (AWS) Local
Zones by setting the zone names in the edge compute pool of the install-config.yaml file, or install a
cluster in an existing Amazon Virtual Private Cloud (VPC) with Local Zone subnets.
AWS Local Zones is an infrastructure that place Cloud Resources close to metropolitan regions. For
more information, see the AWS Local Zones Documentation .
6.14.1. Infrastructure prerequisites

You reviewed details about OpenShift Container Platform installation and update processes.
You are familiar with Selecting a cluster installation method and preparing it for users .
790

WARNING
If you have an AWS profile stored on your computer, it must not use a
temporary session token that you generated while using a multi-factor
authentication device. The cluster continues to use your current AWS
credentials to create AWS resources for the entire life of the cluster, so you
must use key-based, long-term credentials. To generate appropriate keys,
see Managing Access Keys for IAM Users in the AWS documentation. You
can supply the keys when you run the installation program.
If you use a firewall, you configured it to allow the sites that your cluster must access.
You noted the region and supported AWS Local Zones locations to create the network
resources in.
You read the AWS Local Zones features in the AWS documentation.
You added permissions for creating network resources that support AWS Local Zones to the
Identity and Access Management (IAM) user or role. The following example enables a zone
group that can provide a user or role access for creating network network resources that
support AWS Local Zones.
Example of an additional IAM policy with the ec2:ModifyAvailabilityZoneGroup

permission attached to an IAM user or role.
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"ec2:ModifyAvailabilityZoneGroup"
],
"Effect": "Allow",
"Resource": "*"
}
]
}
6.14.2. About AWS Local Zones and edge compute pool

Read the following sections to understand infrastructure behaviors and cluster limitations in an AWS
Local Zones environment.
6.14.2.1. Cluster limitations in AWS Local Zones
Some limitations exist when you try to deploy a cluster with a default installation configuration in an
Amazon Web Services (AWS) Local Zone.
IMPORTANT
791
IMPORTANT
The following list details limitations when deploying a cluster in a pre-configured AWS
zone:
The maximum transmission unit (MTU) between an Amazon EC2 instance in a

zone and an Amazon EC2 instance in the Region is 1300. This causes the cluster-
wide network MTU to change according to the network plugin that is used with
the deployment.
Network resources such as Network Load Balancer (NLB), Classic Load

Balancer, and Network Address Translation (NAT) Gateways are not globally
supported.
For an OpenShift Container Platform cluster on AWS, the AWS Elastic Block
Storage (EBS) gp3 type volume is the default for node volumes and the default
for the storage class. This volume type is not globally available on zone locations.
By default, the nodes running in zones are deployed with the gp2 EBS volume.
The gp2-csi StorageClass parameter must be set when creating workloads on
zone nodes.
If you want the installation program to automatically create Local Zone subnets for your OpenShift
Container Platform cluster, specific configuration limitations apply with this method.
IMPORTANT
The following configuration limitation applies when you set the installation program to
automatically create subnets for your OpenShift Container Platform cluster:
When the installation program creates private subnets in AWS Local Zones, the
program associates each subnet with the route table of its parent zone. This
operation ensures that each private subnet can route egress traffic to the
internet by way of NAT Gateways in an AWS Region.
If the parent-zone route table does not exist during cluster installation, the
installation program associates any private subnet with the first available private
route table in the Amazon Virtual Private Cloud (VPC). This approach is valid only
for AWS Local Zones subnets in an OpenShift Container Platform cluster.
6.14.2.2. About edge compute pools
Edge compute nodes are tainted compute nodes that run in AWS Local Zones locations.
When deploying a cluster that uses Local Zones, consider the following points:
Amazon EC2 instances in the Local Zones are more expensive than Amazon EC2 instances in
the Availability Zones.
The latency is lower between the applications running in AWS Local Zones and the end user. A
latency impact exists for some workloads if, for example, ingress traffic is mixed between Local
Zones and Availability Zones.
IMPORTANT
792
IMPORTANT
Generally, the maximum transmission unit (MTU) between an Amazon EC2 instance in a
Local Zones and an Amazon EC2 instance in the Region is 1300. The cluster network MTU
must be always less than the EC2 MTU to account for the overhead. The specific
overhead is determined by the network plugin. For example: OVN-Kubernetes has an
overhead of 100 bytes.
The network plugin can provide additional features, such as IPsec, that also affect the
MTU sizing.
For more information, see How Local Zones work in the AWS documentation.
OpenShift Container Platform 4.12 introduced a new compute pool, edge, that is designed for use in
remote zones. The edge compute pool configuration is common between AWS Local Zones locations.
Because of the type and size limitations of resources like EC2 and EBS on Local Zones resources, the
default instance type can vary from the traditional compute pool.
The default Elastic Block Store (EBS) for Local Zones locations is gp2, which differs from the non-edge
compute pool. The instance type used for each Local Zones on an edge compute pool also might differ
from other compute pools, depending on the instance offerings on the zone.
The edge compute pool creates new labels that developers can use to deploy applications onto AWS
Local Zones nodes. The new labels are:
node-role.kubernetes.io/edge=''
machine.openshift.io/zone-type=local-zone
machine.openshift.io/zone-group=$ZONE_GROUP_NAME
By default, the machine sets for the edge compute pool define the taint of NoSchedule to prevent
other workloads from spreading on Local Zones instances. Users can only run user workloads if they
define tolerations in the pod specification.
MTU value selection
Changing the MTU for the cluster network
Understanding taints and tolerations
Storage classes
Ingress Controller sharding
6.14.3. Installation prerequisites

Before you install a cluster in an AWS Local Zones environment, you must configure your infrastructure
so that it can adopt Local Zone capabilities.
6.14.3.1. Opting in to an AWS Local Zones
If you plan to create subnets in AWS Local Zones, you must opt in to each zone group separately.
Prerequisites
793
Prerequisites
You have installed the AWS CLI.
You have determined an AWS Region for where you want to deploy your OpenShift Container
Platform cluster.
You have attached a permissive IAM policy to a user or role account that opts in to the zone
group.
Procedure
1. List the zones that are available in your AWS Region by running the following command:
Example command for listing available AWS Local Zones in an AWS Region
$ aws --region "<value_of_AWS_Region>" ec2 describe-availability-zones \

--query 'AvailabilityZones[].[{ZoneName: ZoneName, GroupName: GroupName, Status:
OptInStatus}]' \
--filters Name=zone-type,Values=local-zone \
--all-availability-zones
Depending on the AWS Region, the list of available zones might be long. The command returns
the following fields:
ZoneName
The name of the Local Zones.
GroupName
The group that comprises the zone. To opt in to the Region, save the name.
Status
The status of the Local Zones group. If the status is not-opted-in, you must opt in the
GroupName as described in the next step.
2. Opt in to the zone group on your AWS account by running the following command:
$ aws ec2 modify-availability-zone-group \

--group-name "<value_of_GroupName>" \ 1
--opt-in-status opted-in
1 Replace <value_of_GroupName> with the name of the group of the Local Zones where
you want to create subnets. For example, specify us-east-1-nyc-1 to use the zone us-
east-1-nyc-1a (US East New York).
6.14.3.2. Internet access for OpenShift Container Platform
794
IMPORTANT
6.14.3.3. Obtaining an AWS Marketplace image
Prerequisites
Procedure
apiVersion: v1
compute:
name: worker
platform:
aws:
type: m5.4xlarge
replicas: 3
metadata:
name: test-cluster
platform:
aws:
region: us-east-2 2
795
IMPORTANT

Procedure
Portal.
$ tar xvf <file>

$ echo $PATH
Verification
$ oc <command>

Procedure
Portal.
796
C:\> path
Verification
C:\> oc <command>

Procedure
Portal.
NOTE

$ echo $PATH
Verification
$ oc <command>
6.14.3.5. Obtaining the installation program
for installation.
Prerequisites
Procedure
797
IMPORTANT
IMPORTANT
components.
6.14.3.6. Generating a key pair for cluster node SSH access
authentication.
user.
IMPORTANT
NOTE
798
NOTE
Procedure
NOTE
ecdsa algorithm.
NOTE

task:
Example output
Agent pid 31874
NOTE
799
NOTE

Example output
Next steps
program.
6.14.4. Preparing for the installation

Before you extend nodes to Local Zones, you must prepare certain resources for the cluster installation
environment.

System Per Second
(IOPS)[2]

8.6 and later
[3]
800
Platform for use with AWS Local Zones.
NOTE
Example 6.91. Machine types based on 64-bit x86 architecture for AWS Local Zones
c5.*
c5d.*
m6i.*
m5.*
r5.*
t3.*
See AWS Local Zones features in the AWS documentation.
your cluster.
Prerequisites
infrastructure and the pull secret for your cluster.
801
Procedure
command:
IMPORTANT

version.
NOTE

program.
NOTE
stored in the file.
802
IMPORTANT

6.14.4.4. Examples of installation configuration files with edge compute pools
The following examples show install-config.yaml files that contain an edge machine pool configuration.
Configuration that uses an edge pool with a custom instance type
apiVersion: v1
baseDomain: devcluster.openshift.com
metadata:
name: ipi-edgezone
compute:
- name: edge
platform:
aws:
type: r5.2xlarge
platform:
aws:
region: us-west-2
Instance types differ between locations. To verify availability in the Local Zones in which the cluster
runs, see the AWS documentation.
Configuration that uses an edge pool with a custom Amazon Elastic Block Store (EBS) type
apiVersion: v1
metadata:
name: ipi-edgezone
compute:
- name: edge
platform:
aws:
zones:
- us-west-2-lax-1a
- us-west-2-lax-1b
- us-west-2-phx-2a
rootVolume:
type: gp3
size: 120
platform:
aws:
803
region: us-west-2
Elastic Block Storage (EBS) types differ between locations. Check the AWS documentation to verify
availability in the Local Zones in which the cluster runs.
Configuration that uses an edge pool with custom security groups
apiVersion: v1
metadata:
name: ipi-edgezone
compute:
- name: edge
platform:
aws:
- sg-1 1
- sg-2
platform:
aws:
region: us-west-2
1 Specify the name of the security group as it is displayed on the Amazon EC2 console. Ensure that
you include the sg prefix.
6.14.4.5. Customizing the cluster network MTU
Before you deploy a cluster on AWS, you can customize the cluster network maximum transmission unit
(MTU) for your cluster network to meet the needs of your infrastructure.
By default, when you install a cluster with supported Local Zones capabilities, the MTU value for the
cluster network is automatically adjusted to the lowest value that the network plugin accepts.
IMPORTANT
Setting an unsupported MTU value for EC2 instances that operate in the Local Zones
infrastructure can cause issues for your OpenShift Container Platform cluster.
If the Local Zone supports higher MTU values in between EC2 instances in the Local Zone and the AWS
Region, you can manually configure the higher value to increase the network performance of the cluster
network.
You can customize the MTU for a cluster by specifying the networking.clusterNetworkMTU parameter
in the install-config.yaml configuration file.
IMPORTANT
804
IMPORTANT
All subnets in Local Zones must support the higher MTU value, so that each node in that
zone can successfully communicate with services in the AWS Region and deploy your
workloads.
Example of overwriting the default MTU value
apiVersion: v1
metadata:
name: edge-zone
networking:
clusterNetworkMTU: 8901
compute:
- name: edge
platform:
aws:
zones:
- us-west-2-lax-1a
- us-west-2-lax-1b
platform:
aws:
region: us-west-2
For more information about the maximum supported maximum transmission unit (MTU) value,
see AWS resources supported in Local Zones in the AWS documentation.
6.14.5. Cluster installation options for an AWS Local Zones environment

Choose one of the following installation options to install an OpenShift Container Platform cluster on
AWS with edge compute nodes defined in Local Zones:
Fully automated option: Installing a cluster to quickly extend compute nodes to edge compute
pools, where the installation program automatically creates infrastructure resources for the
Existing VPC option: Installing a cluster on AWS into an existing VPC, where you supply Local
Zones subnets to the install-config.yaml file.
Next steps
Choose one of the following options to install an OpenShift Container Platform cluster in an AWS Local
Zones environment:
Installing a cluster quickly in AWS Local Zones
Installing a cluster in an existing VPC with defined Local Zone subnets
6.14.6. Install a cluster quickly in AWS Local Zones
805
For OpenShift Container Platform 4.15, you can quickly install a cluster on Amazon Web Services (AWS)
to extend compute nodes to Local Zones locations. By using this installation route, the installation
program automatically creates network resources and Local Zones subnets for each zone that you
defined in your configuration file. To customize the installation, you must modify parameters in the
install-config.yaml file before you deploy the cluster.
6.14.6.1. Modifying an installation configuration file to use AWS Local Zones
Modify an install-config.yaml file to include AWS Local Zones.
Prerequisites
You have configured an AWS account.
You added your AWS keys and AWS Region to your local AWS profile by running aws
configure.
You are familiar with the configuration limitations that apply when you specify the installation
program to automatically create subnets for your OpenShift Container Platform cluster.
You opted in to the Local Zones group for each zone.
You created an install-config.yaml file by using the procedure "Creating the installation
configuration file".
Procedure
1. Modify the install-config.yaml file by specifying Local Zones names in the platform.aws.zones
property of the edge compute pool.
# ...
platform:
aws:
region: <region_name> 1
compute:
- name: edge
platform:
aws:
zones: 2
- <local_zone_name>
#...
1 The AWS Region name.
2 The list of Local Zones names that you use must exist in the same AWS Region specified in
the platform.aws.region field.
Example of a configuration to install a cluster in the us-west-2 AWS Region that

extends edge nodes to Local Zones in Los Angeles and Las Vegas locations
apiVersion: v1
metadata:
name: cluster-name
806
platform:
aws:
region: us-west-2
compute:
- name: edge
platform:
aws:
zones:
- us-west-2-lax-1a
- us-west-2-lax-1b
- us-west-2-las-1a
sshKey: 'ssh-ed25519 AAAA...'
#...
2. Deploy your cluster.
Creating the installation configuration file
Cluster limitations in AWS Local Zones
Next steps
Deploying the cluster
6.14.7. Installing a cluster in an existing VPC that has Local Zone subnets
You can install a cluster into an existing Amazon Virtual Private Cloud (VPC) on Amazon Web Services
(AWS). The installation program provisions the rest of the required infrastructure, which you can further
customize. To customize the installation, modify parameters in the install-config.yaml file before you
Installing a cluster on AWS into an existing VPC requires extending compute nodes to the edge of the
Cloud Infrastructure by using AWS Local Zones.
Local Zone subnets extend regular compute nodes to edge networks. Each edge compute nodes runs a
user workload. After you create an Amazon Web Service (AWS) Local Zone environment, and you deploy
your cluster, you can use edge compute nodes to create user workloads in Local Zone subnets.
NOTE
If you want to create private subnets, you must either modify the provided
CloudFormation template or create your own template.
You can use a provided CloudFormation template to create network resources. Additionally, you can
modify a template to customize your infrastructure or use the information that they contain to create
AWS resources according to your company’s policies.
IMPORTANT
807
IMPORTANT
The steps for performing an installer-provisioned infrastructure installation are provided

for example purposes only. Installing a cluster in an existing VPC requires that you have
knowledge of the cloud provider and the installation process of OpenShift Container
Platform. You can use a CloudFormation template to assist you with completing these
steps or to help model your own cluster installation. Instead of using the CloudFormation
template to create resources, you can decide to use other methods for generating these
resources.
6.14.7.1. Creating a VPC in AWS
You can create a Virtual Private Cloud (VPC), and subnets for all Local Zones locations, in Amazon Web
Services (AWS) for your OpenShift Container Platform cluster to extend compute nodes to edge
locations. You can further customize your VPC to meet your requirements, including a VPN and route
tables. You can also add new Local Zones subnets not included at initial deployment.
NOTE
Prerequisites
configure.
You opted in to the AWS Local Zones on your AWS account.
Procedure
requires:
[
{
},
{
},
{
808
}
]
2. Go to the section of the documentation named "CloudFormation template for the VPC", and
then copy the syntax from the provided template. Save the copied template syntax as a YAML
file on your local system. This template describes the VPC that your cluster requires.
VPC by running the following command:
IMPORTANT
$ aws cloudformation create-stack --stack-name <name> \ 1

that you saved.
3 <parameters> is the relative path and the name of the CloudFormation parameters JSON
file.
Example output
820e-12a48460849f
4. Confirm that the template components exist by running the following command:
templates that you run to create your cluster.
809

netIds

bnetIds
PublicRou The ID of the new public route table ID.

teTableId
Parameters:
VpcCidr:
AllowedPattern: ^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-
4][0-9]|25[0-5])(\/(1[6-9]|2[0-4]))$
Default: 10.0.0.0/16
Type: String
MinValue: 1
MaxValue: 3
Default: 1
Type: Number
SubnetBits:
MinValue: 5
MaxValue: 13
Default: 12
/19)"
Type: Number
Metadata:
ParameterGroups:
- Label:
Parameters:
- VpcCidr
810
- SubnetBits
- Label:
Parameters:
ParameterLabels:
VpcCidr:
default: "VPC CIDR"
SubnetBits:
Conditions:
Resources:
VPC:
Properties:
PublicSubnet:
Properties:
VpcId: !Ref VPC
-0
PublicSubnet2:
Condition: DoAz2
Properties:
VpcId: !Ref VPC
-1
PublicSubnet3:
Condition: DoAz3
Properties:
VpcId: !Ref VPC
-2
InternetGateway:
GatewayToInternet:
Properties:
VpcId: !Ref VPC
811

PublicRouteTable:
Properties:
VpcId: !Ref VPC
PublicRoute:
Properties:
Properties:
Condition: DoAz2
Properties:
Condition: DoAz3
Properties:
PrivateSubnet:
Properties:
VpcId: !Ref VPC
-0
PrivateRouteTable:
Properties:
VpcId: !Ref VPC
Properties:
NAT:
DependsOn:
- GatewayToInternet
Properties:
AllocationId:
"Fn::GetAtt":
- EIP
- AllocationId
812
EIP:
Properties:
Domain: vpc
Route:
Properties:
RouteTableId:
NatGatewayId:
Ref: NAT
PrivateSubnet2:
Condition: DoAz2
Properties:
VpcId: !Ref VPC
-1
PrivateRouteTable2:
Condition: DoAz2
Properties:
VpcId: !Ref VPC
Condition: DoAz2
Properties:
NAT2:
DependsOn:
- GatewayToInternet
Condition: DoAz2
Properties:
AllocationId:
"Fn::GetAtt":
- EIP2
- AllocationId
EIP2:
Condition: DoAz2
Properties:
Domain: vpc
Route2:
Condition: DoAz2
Properties:
RouteTableId:
NatGatewayId:
813
Ref: NAT2
PrivateSubnet3:
Condition: DoAz3
Properties:
VpcId: !Ref VPC
-2
PrivateRouteTable3:
Condition: DoAz3
Properties:
VpcId: !Ref VPC
Condition: DoAz3
Properties:
NAT3:
DependsOn:
- GatewayToInternet
Condition: DoAz3
Properties:
AllocationId:
"Fn::GetAtt":
- EIP3
- AllocationId
EIP3:
Condition: DoAz3
Properties:
Domain: vpc
Route3:
Condition: DoAz3
Properties:
RouteTableId:
NatGatewayId:
Ref: NAT3
S3Endpoint:
Properties:
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal: '*'
Action:
- '*'
814
Resource:
- '*'
RouteTableIds:
ServiceName: !Join
- ''
- - com.amazonaws.
- .s3
VpcId: !Ref VPC
Outputs:
VpcId:
Value: !Ref VPC
PublicSubnetIds:
Value:
!Join [
",",
]
PrivateSubnetIds:
Value:
!Join [
",",
]
PublicRouteTableId:
Value:
!Join [
",",
[
!Join ["=", [
]],
!If [DoAz2,
!Ref "AWS::NoValue"
],
!If [DoAz3,
!Ref "AWS::NoValue"
815
]
]
]
6.14.7.3. Creating subnets in Local Zones
Before you configure a machine set for edge compute nodes in your OpenShift Container Platform
cluster, you must create the subnets in Local Zones. Complete the following procedure for each Local
Zone that you want to deploy compute nodes to.
You can use the provided CloudFormation template and create a CloudFormation stack. You can then
use this stack to custom provision a subnet.
NOTE
Prerequisites
You opted in to the Local Zones group.
Procedure
1. Go to the section of the documentation named "CloudFormation template for the VPC subnet",
and copy the syntax from the template. Save the copied template syntax as a YAML file on your
local system. This template describes the VPC that your cluster requires.
2. Run the following command to deploy the CloudFormation template, which creates a stack of
AWS resources that represent the VPC:
$ aws cloudformation create-stack --stack-name <stack_name> \ 1

--region ${CLUSTER_REGION} \
--parameters \
ParameterKey=VpcId,ParameterValue="${VPC_ID}" \ 3
ParameterKey=ClusterName,ParameterValue="${CLUSTER_NAME}" \ 4
ParameterKey=ZoneName,ParameterValue="${ZONE_NAME}" \ 5
ParameterKey=PublicRouteTableId,ParameterValue="${ROUTE_TABLE_PUB}" \ 6
ParameterKey=PublicSubnetCidr,ParameterValue="${SUBNET_CIDR_PUB}" \ 7
ParameterKey=PrivateRouteTableId,ParameterValue="${ROUTE_TABLE_PVT}" \ 8
ParameterKey=PrivateSubnetCidr,ParameterValue="${SUBNET_CIDR_PVT}" 9
1 <stack_name> is the name for the CloudFormation stack, such as cluster-wl-

<local_zone_shortname>. You need the name of this stack if you remove the cluster.
<template> is the relative path and the name of the CloudFormation template YAML file
816
2 <template> is the relative path and the name of the CloudFormation template YAML file
that you saved.
3 ${VPC_ID} is the VPC ID, which is the value VpcID in the output of the CloudFormation
template for the VPC.
4 ${ZONE_NAME} is the value of Local Zones name to create the subnets.
5 ${CLUSTER_NAME} is the value of ClusterName to be used as a prefix of the new AWS

resource names.
6 ${SUBNET_CIDR_PUB} is a valid CIDR block that is used to create the public subnet. This
block must be part of the VPC CIDR block VpcCidr.
7 ${ROUTE_TABLE_PVT} is the PrivateRouteTableId extracted from the output of the

VPC’s CloudFormation stack.
8 ${SUBNET_CIDR_PVT} is a valid CIDR block that is used to create the private subnet. This
Example output
arn:aws:cloudformation:us-east-1:123456789012:stack/<stack_name>/dbedae40-820e-11eb-2fd3-
12a48460849f
Verification
Confirm that the template components exist by running the following command:
$ aws cloudformation describe-stacks --stack-name <stack_name>
following parameters. Ensure that you provide these parameter values to the other
CloudFormation templates that you run to create for your cluster.
PublicSub The IDs of the public subnet created by the CloudFormation stack.
netId
PrivateSu The IDs of the private subnet created by the CloudFormation stack.
bnetId
6.14.7.4. CloudFormation template for the VPC subnet
You can use the following CloudFormation template to deploy the private and public subnets in a zone
on Local Zones infrastructure.
Example 6.93. CloudFormation template for VPC subnets
Description: Template for Best Practice Subnets (Public and Private)
Parameters:
817
VpcId:
Description: VPC ID that comprises all the target subnets.
Type: String
AllowedPattern: ^(?:(?:vpc)(?:-[a-zA-Z0-9]+)?\b|(?:[0-9]{1,3}\.){3}[0-9]{1,3})$
ConstraintDescription: VPC ID must be with valid name, starting with vpc-.*.
ClusterName:
Description: Cluster name or prefix name to prepend the Name tag for each subnet.
Type: String
AllowedPattern: ".+"
ConstraintDescription: ClusterName parameter must be specified.
ZoneName:
Description: Zone Name to create the subnets, such as us-west-2-lax-1a.
Type: String
ConstraintDescription: ZoneName parameter must be specified.
PublicRouteTableId:
Description: Public Route Table ID to associate the public subnet.
Type: String
ConstraintDescription: PublicRouteTableId parameter must be specified.
PublicSubnetCidr:
AllowedPattern: ^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-
4][0-9]|25[0-5])(\/(1[6-9]|2[0-4]))$
Default: 10.0.128.0/20
Description: CIDR block for public subnet.
Type: String
PrivateRouteTableId:
Description: Private Route Table ID to associate the private subnet.
Type: String
ConstraintDescription: PrivateRouteTableId parameter must be specified.
PrivateSubnetCidr:
AllowedPattern: ^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-
4][0-9]|25[0-5])(\/(1[6-9]|2[0-4]))$
Default: 10.0.128.0/20
Description: CIDR block for private subnet.
Type: String
Resources:
PublicSubnet:
Properties:
VpcId: !Ref VpcId
CidrBlock: !Ref PublicSubnetCidr
AvailabilityZone: !Ref ZoneName
Tags:
- Key: Name
Value: !Join ['-', [!Ref ClusterName, "public", !Ref ZoneName]]
Properties:
818
RouteTableId: !Ref PublicRouteTableId
PrivateSubnet:
Properties:
VpcId: !Ref VpcId
CidrBlock: !Ref PrivateSubnetCidr
Tags:
- Key: Name
Value: !Join ['-', [!Ref ClusterName, "private", !Ref ZoneName]]
Properties:
RouteTableId: !Ref PrivateRouteTableId
Outputs:
PublicSubnetId:
Description: Subnet ID of the public subnets.
Value:
!Join ["", [!Ref PublicSubnet]]
PrivateSubnetId:
Description: Subnet ID of the private subnets.
Value:
!Join ["", [!Ref PrivateSubnet]]
6.14.7.5. Modifying an installation configuration file to use AWS Local Zones subnets
Modify your install-config.yaml file to include Local Zones subnets.
Prerequisites
You created subnets by using the procedure "Creating subnets in Local Zones".
Procedure
Modify the install-config.yaml configuration file by specifying Local Zones subnets in the
platform.aws.subnets parameter.
Example installation configuration file with Local Zones subnets
# ...
819
platform:
aws:
region: us-west-2
subnets: 1
- publicSubnetId-1
- publicSubnetId-2
- publicSubnetId-3
- privateSubnetId-1
- privateSubnetId-2
- privateSubnetId-3
- publicSubnetId-LocalZone-1
# ...
1 List of subnet IDs created in the zones: Availability and Local Zones.
For more information about viewing the CloudFormation stacks that you created, see AWS
For more information about AWS profile and credential configuration, see Configuration and
credential file settings in the AWS documentation.
Next steps
6.14.8. Optional: AWS security groups

For more information, see "Edge compute pools and AWS Local Zones".
6.14.9. Optional: Assign public IP addresses to edge compute nodes

If your workload requires deploying the edge compute nodes in public subnets on Local Zones
infrastructure, you can configure the machine set manifests when installing a cluster.
AWS Local Zones infrastructure accesses the network traffic in a specified zone, so applications can
take advantage of lower latency when serving end users that are closer to that zone.
The default setting that deploys compute nodes in private subnets might not meet your needs, so
consider creating edge compute nodes in public subnets when you want to apply more customization to
your infrastructure.
IMPORTANT
820
IMPORTANT
By default, OpenShift Container Platform deploy the compute nodes in private subnets.
For best performance, consider placing compute nodes in subnets that have their Public
IP addresses attached to the subnets.
You must create additional security groups, but ensure that you only open the groups'
rules over the internet when you really need to.
Procedure
1. Change to the directory that contains the installation program and generate the manifest files.
Ensure that the installation manifests get created at the openshift and manifests directory
level.
2. Edit the machine set manifest that the installation program generates for the Local Zones, so
that the manifest gets deployed in public subnets. Specify true for the
spec.template.spec.providerSpec.value.publicIP parameter.
Example machine set manifest configuration for installing a cluster quickly in Local
Zones
spec:
template:
spec:
providerSpec:
value:
publicIp: true
subnet:
filters:
- name: tag:Name
values:
- ${INFRA_ID}-public-${ZONE_NAME}
Example machine set manifest configuration for installing a cluster in an existing

VPC that has Local Zones subnets
apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
name: <infrastructure_id>-edge-<zone>
namespace: openshift-machine-api
spec:
template:
spec:
providerSpec:
value:
publicIp: true

821
IMPORTANT
Prerequisites
cluster.
Procedure
deployment:

--log-level=info 2

config.yaml file.
NOTE

Verification
IMPORTANT
Example output
822
...
IMPORTANT
6.14.11. Verifying the status of the deployed cluster

Verify that your OpenShift Container Platform successfully deployed on AWS Local Zones.
6.14.11.1. Logging in to the cluster by using the CLI
Prerequisites
Procedure
823
$ oc whoami
Example output
system:admin
6.14.11.2. Logging in to the cluster by using the web console
Prerequisites
Procedure
installation host:
NOTE

NOTE
Example output

For more information about accessing and understanding the OpenShift Container Platform
web console, see Accessing the web console for more details about accessing and
understanding the OpenShift Container Platform web console.
824
6.14.11.3. Verifying nodes that were created with edge compute pool
After you install a cluster that uses AWS Local Zones infrastructure, check the status of the machine
that was created by the machine set manifests created during installation.
1. To check the machine sets created from the subnet you added to the install-config.yaml file,
$ oc get machineset -n openshift-machine-api
Example output
NAME DESIRED CURRENT READY AVAILABLE AGE

cluster-7xw5g-edge-us-east-1-nyc-1a 1 1 1 1 3h4m
cluster-7xw5g-worker-us-east-1a 1 1 1 1 3h4m
cluster-7xw5g-worker-us-east-1b 1 1 1 1 3h4m
cluster-7xw5g-worker-us-east-1c 1 1 1 1 3h4m
2. To check the machines that were created from the machine sets, run the following command:
$ oc get machines -n openshift-machine-api
Example output
NAME PHASE TYPE REGION ZONE AGE

cluster-7xw5g-edge-us-east-1-nyc-1a-wbclh Running c5d.2xlarge us-east-1 us-east-1-
nyc-1a 3h
cluster-7xw5g-master-0 Running m6i.xlarge us-east-1 us-east-1a 3h4m
cluster-7xw5g-master-1 Running m6i.xlarge us-east-1 us-east-1b 3h4m
cluster-7xw5g-master-2 Running m6i.xlarge us-east-1 us-east-1c 3h4m
cluster-7xw5g-worker-us-east-1a-rtp45 Running m6i.xlarge us-east-1 us-east-1a
3h
cluster-7xw5g-worker-us-east-1b-glm7c Running m6i.xlarge us-east-1 us-east-1b
3h
cluster-7xw5g-worker-us-east-1c-qfvz4 Running m6i.xlarge us-east-1 us-east-1c
3h
3. To check nodes with edge roles, run the following command:
$ oc get nodes -l node-role.kubernetes.io/edge
Example output

ip-10-0-207-188.ec2.internal Ready edge,worker 172m v1.25.2+d2e245f

825
For more information about the Telemetry service, see About remote health monitoring.
Next steps
If necessary, you can opt out of remote health .

AWS WAVELENGTH ZONES
You can quickly install an OpenShift Container Platform cluster on Amazon Web Services (AWS)
Wavelength Zones by setting the zone names in the edge compute pool of the install-config.yaml file,
or install a cluster in an existing Amazon Virtual Private Cloud (VPC) with Wavelength Zone subnets.
AWS Wavelength Zones is an infrastructure that AWS configured for mobile edge computing (MEC)
applications.
A Wavelength Zone embeds AWS compute and storage services within the 5G network of a
communication service provider (CSP). By placing application servers in a Wavelength Zone, the
application traffic from your 5G devices can stay in the 5G network. The application traffic of the device
reaches the target server directly, making latency a non-issue.
See Wavelength Zones in the AWS documentation.
6.15.1. Infrastructure prerequisites

You reviewed details about OpenShift Container Platform installation and update processes.
You are familiar with Selecting a cluster installation method and preparing it for users .

WARNING
If you have an AWS profile stored on your computer, it must not use a
temporary session token that you generated while using a multi-factor
authentication device. The cluster continues to use your current AWS
credentials to create AWS resources for the entire life of the cluster, so you
must use key-based, long-term credentials. To generate appropriate keys,
see Managing Access Keys for IAM Users in the AWS documentation. You
can supply the keys when you run the installation program.
826
If you use a firewall, you configured it to allow the sites that your cluster must access.
You noted the region and supported AWS Wavelength Zone locations to create the network
resources in.
You read AWS Wavelength features in the AWS documentation.
You read the Quotas and considerations for Wavelength Zones in the AWS documentation.
You added permissions for creating network resources that support AWS Wavelength Zones to
the Identity and Access Management (IAM) user or role. For example:
Example of an additional IAM policy that attached ec2:ModifyAvailabilityZoneGroup ,

ec2:CreateCarrierGateway, and ec2:DeleteCarrierGateway permissions to a user or role
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ec2:DeleteCarrierGateway",
"ec2:CreateCarrierGateway"
],
"Resource": "*"
},
{
"Action": [
"ec2:ModifyAvailabilityZoneGroup"
],
"Effect": "Allow",
"Resource": "*"
}
]
}
6.15.2. About AWS Wavelength Zones and edge compute pool

Read the following sections to understand infrastructure behaviors and cluster limitations in an AWS
Wavelength Zones environment.
6.15.2.1. Cluster limitations in AWS Wavelength Zones
Some limitations exist when you try to deploy a cluster with a default installation configuration in an
Amazon Web Services (AWS) Wavelength Zone.
IMPORTANT
827
IMPORTANT
The following list details limitations when deploying a cluster in a pre-configured AWS
zone:
The maximum transmission unit (MTU) between an Amazon EC2 instance in a

zone and an Amazon EC2 instance in the Region is 1300. This causes the cluster-
wide network MTU to change according to the network plugin that is used with
the deployment.
Network resources such as Network Load Balancer (NLB), Classic Load

Balancer, and Network Address Translation (NAT) Gateways are not globally
supported.
For an OpenShift Container Platform cluster on AWS, the AWS Elastic Block
Storage (EBS) gp3 type volume is the default for node volumes and the default
for the storage class. This volume type is not globally available on zone locations.
By default, the nodes running in zones are deployed with the gp2 EBS volume.
The gp2-csi StorageClass parameter must be set when creating workloads on
zone nodes.
If you want the installation program to automatically create Wavelength Zone subnets for your
OpenShift Container Platform cluster, specific configuration limitations apply with this method. The
following note details some of these limitations. For other limitations, ensure that you read the "Quotas
and considerations for Wavelength Zones" document that Red Hat provides in the "Infrastructure
prerequisites" section.
IMPORTANT
The following configuration limitation applies when you set the installation program to
automatically create subnets for your OpenShift Container Platform cluster:
When the installation program creates private subnets in AWS Wavelength

Zones, the program associates each subnet with the route table of its parent
zone. This operation ensures that each private subnet can route egress traffic to
the internet by way of NAT Gateways in an AWS Region.
If the parent-zone route table does not exist during cluster installation, the
installation program associates any private subnet with the first available private
route table in the Amazon Virtual Private Cloud (VPC). This approach is valid only
for AWS Wavelength Zones subnets in an OpenShift Container Platform cluster.
6.15.2.2. About edge compute pools
Edge compute nodes are tainted compute nodes that run in AWS Wavelength Zones locations.
When deploying a cluster that uses Wavelength Zones, consider the following points:
Amazon EC2 instances in the Wavelength Zones are more expensive than Amazon EC2
instances in the Availability Zones.
The latency is lower between the applications running in AWS Wavelength Zones and the end
user. A latency impact exists for some workloads if, for example, ingress traffic is mixed between
Wavelength Zones and Availability Zones.
IMPORTANT
828
IMPORTANT
Generally, the maximum transmission unit (MTU) between an Amazon EC2 instance in a
Wavelength Zones and an Amazon EC2 instance in the Region is 1300. The cluster
network MTU must be always less than the EC2 MTU to account for the overhead. The
specific overhead is determined by the network plugin. For example: OVN-Kubernetes
has an overhead of 100 bytes.
The network plugin can provide additional features, such as IPsec, that also affect the
MTU sizing.
For more information, see How AWS Wavelength work in the AWS documentation.
OpenShift Container Platform 4.12 introduced a new compute pool, edge, that is designed for use in
remote zones. The edge compute pool configuration is common between AWS Wavelength Zones
locations. Because of the type and size limitations of resources like EC2 and EBS on Wavelength Zones
resources, the default instance type can vary from the traditional compute pool.
The default Elastic Block Store (EBS) for Wavelength Zones locations is gp2, which differs from the
non-edge compute pool. The instance type used for each Wavelength Zones on an edge compute pool
also might differ from other compute pools, depending on the instance offerings on the zone.
The edge compute pool creates new labels that developers can use to deploy applications onto AWS
Wavelength Zones nodes. The new labels are:
node-role.kubernetes.io/edge=''
machine.openshift.io/zone-type=wavelength-zone
machine.openshift.io/zone-group=$ZONE_GROUP_NAME
By default, the machine sets for the edge compute pool define the taint of NoSchedule to prevent
other workloads from spreading on Wavelength Zones instances. Users can only run user workloads if
they define tolerations in the pod specification.
MTU value selection
Changing the MTU for the cluster network
Understanding taints and tolerations
Storage classes
Ingress Controller sharding
6.15.3. Installation prerequisites

Before you install a cluster in an AWS Wavelength Zones environment, you must configure your
infrastructure so that it can adopt Wavelength Zone capabilities.
6.15.3.1. Opting in to an AWS Wavelength Zones
If you plan to create subnets in AWS Wavelength Zones, you must opt in to each zone group separately.
Prerequisites
829
Prerequisites
You have installed the AWS CLI.
You have determined an AWS Region for where you want to deploy your OpenShift Container
Platform cluster.
You have attached a permissive IAM policy to a user or role account that opts in to the zone
group.
Procedure
1. List the zones that are available in your AWS Region by running the following command:
Example command for listing available AWS Wavelength Zones in an AWS Region
$ aws --region "<value_of_AWS_Region>" ec2 describe-availability-zones \

--query 'AvailabilityZones[].[{ZoneName: ZoneName, GroupName: GroupName, Status:
OptInStatus}]' \
--filters Name=zone-type,Values=wavelength-zone \
--all-availability-zones
Depending on the AWS Region, the list of available zones might be long. The command returns
the following fields:
ZoneName
The name of the Wavelength Zones.
GroupName
The group that comprises the zone. To opt in to the Region, save the name.
Status
The status of the Wavelength Zones group. If the status is not-opted-in, you must opt in the
GroupName as described in the next step.
2. Opt in to the zone group on your AWS account by running the following command:
$ aws ec2 modify-availability-zone-group \

--group-name "<value_of_GroupName>" \ 1
--opt-in-status opted-in
1 Replace <value_of_GroupName> with the name of the group of the Wavelength Zones
where you want to create subnets. As an example for Wavelength Zones, specify us-east-
1-wl1 to use the zone us-east-1-wl1-nyc-wlz-1 (US East New York).
830
IMPORTANT
6.15.3.3. Obtaining an AWS Marketplace image
Prerequisites
Procedure
apiVersion: v1
compute:
name: worker
platform:
aws:
type: m5.4xlarge
replicas: 3
metadata:
name: test-cluster
platform:
aws:
region: us-east-2 2
831
IMPORTANT

Procedure
Portal.
$ tar xvf <file>

$ echo $PATH
Verification
$ oc <command>

Procedure
Portal.
832
C:\> path
Verification
C:\> oc <command>

Procedure
Portal.
NOTE

$ echo $PATH
Verification
$ oc <command>
for installation.
Prerequisites
Procedure
833
IMPORTANT
IMPORTANT
components.
authentication.
user.
IMPORTANT
NOTE
834
NOTE
Procedure
NOTE
ecdsa algorithm.
NOTE

task:
Example output
Agent pid 31874
NOTE
835
NOTE

Example output
Next steps
program.
6.15.4. Preparing for the installation

Before you extend nodes to Wavelength Zones, you must prepare certain resources for the cluster
installation environment.

System Per Second
(IOPS)[2]

8.6 and later
[3]
836
Platform for use with AWS Wavelength Zones.
NOTE
Example 6.94. Machine types based on 64-bit x86 architecture for AWS Wavelength Zones
r5.*
t3.*
See AWS Wavelength features in the AWS documentation.
your cluster.
Prerequisites
your cluster.
Procedure
837
command:
IMPORTANT

version.
NOTE

program.
NOTE
stored in the file.
IMPORTANT
838
IMPORTANT

6.15.4.4. Examples of installation configuration files with edge compute pools
The following examples show install-config.yaml files that contain an edge machine pool configuration.
Configuration that uses an edge pool with a custom instance type
apiVersion: v1
metadata:
name: ipi-edgezone
compute:
- name: edge
platform:
aws:
type: r5.2xlarge
platform:
aws:
region: us-west-2
Instance types differ between locations. To verify availability in the Wavelength Zones in which the
cluster runs, see the AWS documentation.
Configuration that uses an edge pool with custom security groups
apiVersion: v1
metadata:
name: ipi-edgezone
compute:
- name: edge
platform:
aws:
- sg-1 1
- sg-2
platform:
aws:
region: us-west-2
1 Specify the name of the security group as it is displayed on the Amazon EC2 console. Ensure that
you include the sg prefix.
6.15.5. Cluster installation options for an AWS Wavelength Zones environment
839
AWS with edge compute nodes defined in Wavelength Zones:
Fully automated option: Installing a cluster to quickly extend compute nodes to edge compute
pools, where the installation program automatically creates infrastructure resources for the
Existing VPC option: Installing a cluster on AWS into an existing VPC, where you supply
Wavelength Zones subnets to the install-config.yaml file.
Next steps
Choose one of the following options to install an OpenShift Container Platform cluster in an AWS
Wavelength Zones environment:
Installing a cluster quickly in AWS Wavelength Zones
Modifying an installation configuration file to use AWS Wavelength Zones
6.15.6. Install a cluster quickly in AWS Wavelength Zones

For OpenShift Container Platform 4.15, you can quickly install a cluster on Amazon Web Services (AWS)
to extend compute nodes to Wavelength Zones locations. By using this installation route, the
installation program automatically creates network resources and Wavelength Zones subnets for each
zone that you defined in your configuration file. To customize the installation, you must modify
parameters in the install-config.yaml file before you deploy the cluster.
6.15.6.1. Modifying an installation configuration file to use AWS Wavelength Zones
Modify an install-config.yaml file to include AWS Wavelength Zones.
Prerequisites
You have configured an AWS account.
configure.
You are familiar with the configuration limitations that apply when you specify the installation
program to automatically create subnets for your OpenShift Container Platform cluster.
You opted in to the Wavelength Zones group for each zone.
Procedure
1. Modify the install-config.yaml file by specifying Wavelength Zones names in the

platform.aws.zones property of the edge compute pool.
# ...
platform:
aws:
region: <region_name> 1
compute:
840
- name: edge
platform:
aws:
zones: 2
- <wavelength_zone_name>
#...
1 The AWS Region name.
2 The list of Wavelength Zones names that you use must exist in the same AWS Region
specified in the platform.aws.region field.
Example of a configuration to install a cluster in the us-west-2 AWS Region that

extends edge nodes to Wavelength Zones in Los Angeles and Las Vegas locations
apiVersion: v1
metadata:
name: cluster-name
platform:
aws:
region: us-west-2
compute:
- name: edge
platform:
aws:
zones:
- us-west-2-wl1-lax-wlz-1
- us-west-2-wl1-las-wlz-1
sshKey: 'ssh-ed25519 AAAA...'
#...
2. Deploy your cluster.
Creating the installation configuration file
Cluster limitations in AWS Wavelength Zones
Next steps
6.15.7. Installing a cluster in an existing VPC that has Wavelength Zone subnets
You can install a cluster into an existing Amazon Virtual Private Cloud (VPC) on Amazon Web Services
(AWS). The installation program provisions the rest of the required infrastructure, which you can further
customize. To customize the installation, modify parameters in the install-config.yaml file before you
Installing a cluster on AWS into an existing VPC requires extending compute nodes to the edge of the
Cloud Infrastructure by using AWS Wavelength Zones.
841
You can use a provided CloudFormation template to create network resources. Additionally, you can
modify a template to customize your infrastructure or use the information that they contain to create
AWS resources according to your company’s policies.
IMPORTANT
The steps for performing an installer-provisioned infrastructure installation are provided

for example purposes only. Installing a cluster in an existing VPC requires that you have
knowledge of the cloud provider and the installation process of OpenShift Container
Platform. You can use a CloudFormation template to assist you with completing these
steps or to help model your own cluster installation. Instead of using the CloudFormation
template to create resources, you can decide to use other methods for generating these
resources.
6.15.7.1. Creating a VPC in AWS
You can create a Virtual Private Cloud (VPC), and subnets for all Wavelength Zones locations, in
Amazon Web Services (AWS) for your OpenShift Container Platform cluster to extend compute nodes
to edge locations. You can further customize your VPC to meet your requirements, including a VPN and
route tables. You can also add new Wavelength Zones subnets not included at initial deployment.
NOTE
Prerequisites
configure.
You opted in to the AWS Wavelength Zones on your AWS account.
Procedure
requires:
[
{
},
{
},
842
{
}
]
2. Go to the section of the documentation named "CloudFormation template for the VPC", and
then copy the syntax from the provided template. Save the copied template syntax as a YAML
file on your local system. This template describes the VPC that your cluster requires.
VPC by running the following command:
IMPORTANT
$ aws cloudformation create-stack --stack-name <name> \ 1

that you saved.
3 <parameters> is the relative path and the name of the CloudFormation parameters JSON
file.
Example output
820e-12a48460849f
4. Confirm that the template components exist by running the following command:
843
templates that you run to create your cluster.

netIds

bnetIds
PublicRou The ID of the new public route table ID.

teTableId
Parameters:
VpcCidr:
AllowedPattern: ^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-
4][0-9]|25[0-5])(\/(1[6-9]|2[0-4]))$
Default: 10.0.0.0/16
Type: String
MinValue: 1
MaxValue: 3
Default: 1
Type: Number
SubnetBits:
MinValue: 5
MaxValue: 13
Default: 12
/19)"
Type: Number
Metadata:
ParameterGroups:
- Label:
844

Parameters:
- VpcCidr
- SubnetBits
- Label:
Parameters:
ParameterLabels:
VpcCidr:
default: "VPC CIDR"
SubnetBits:
Conditions:
Resources:
VPC:
Properties:
PublicSubnet:
Properties:
VpcId: !Ref VPC
-0
PublicSubnet2:
Condition: DoAz2
Properties:
VpcId: !Ref VPC
-1
PublicSubnet3:
Condition: DoAz3
Properties:
VpcId: !Ref VPC
-2
InternetGateway:
GatewayToInternet:
845
Properties:
VpcId: !Ref VPC
PublicRouteTable:
Properties:
VpcId: !Ref VPC
PublicRoute:
Properties:
Properties:
Condition: DoAz2
Properties:
Condition: DoAz3
Properties:
PrivateSubnet:
Properties:
VpcId: !Ref VPC
-0
PrivateRouteTable:
Properties:
VpcId: !Ref VPC
Properties:
NAT:
DependsOn:
- GatewayToInternet
Properties:
AllocationId:
"Fn::GetAtt":
846
- EIP
- AllocationId
EIP:
Properties:
Domain: vpc
Route:
Properties:
RouteTableId:
NatGatewayId:
Ref: NAT
PrivateSubnet2:
Condition: DoAz2
Properties:
VpcId: !Ref VPC
-1
PrivateRouteTable2:
Condition: DoAz2
Properties:
VpcId: !Ref VPC
Condition: DoAz2
Properties:
NAT2:
DependsOn:
- GatewayToInternet
Condition: DoAz2
Properties:
AllocationId:
"Fn::GetAtt":
- EIP2
- AllocationId
EIP2:
Condition: DoAz2
Properties:
Domain: vpc
Route2:
Condition: DoAz2
Properties:
RouteTableId:
847
NatGatewayId:
Ref: NAT2
PrivateSubnet3:
Condition: DoAz3
Properties:
VpcId: !Ref VPC
-2
PrivateRouteTable3:
Condition: DoAz3
Properties:
VpcId: !Ref VPC
Condition: DoAz3
Properties:
NAT3:
DependsOn:
- GatewayToInternet
Condition: DoAz3
Properties:
AllocationId:
"Fn::GetAtt":
- EIP3
- AllocationId
EIP3:
Condition: DoAz3
Properties:
Domain: vpc
Route3:
Condition: DoAz3
Properties:
RouteTableId:
NatGatewayId:
Ref: NAT3
S3Endpoint:
Properties:
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
848
Principal: '*'
Action:
- '*'
Resource:
- '*'
RouteTableIds:
ServiceName: !Join
- ''
- - com.amazonaws.
- .s3
VpcId: !Ref VPC
Outputs:
VpcId:
Value: !Ref VPC
PublicSubnetIds:
Value:
!Join [
",",
]
PrivateSubnetIds:
Value:
!Join [
",",
]
PublicRouteTableId:
Value:
!Join [
",",
[
!Join ["=", [
]],
!If [DoAz2,
!Ref "AWS::NoValue"
],
!If [DoAz3,
849
!Ref "AWS::NoValue"
]
]
]
6.15.7.3. Creating a VPC carrier gateway
To use public subnets in your OpenShift Container Platform cluster that runs on Wavelength Zones, you
must create the carrier gateway and associate the carrier gateway to the VPC. Subnets are useful for
deploying load balancers or edge compute nodes.
To create edge nodes or internet-facing load balancers in Wavelength Zones locations for your
OpenShift Container Platform cluster, you must create the following required network components:
A carrier gateway that associates to the existing VPC.
A carrier route table that lists route entries.
A subnet that associates to the carrier route table.
Carrier gateways exist for VPCs that only contain subnets in a Wavelength Zone.
The following list explains the functions of a carrier gateway in the context of an AWS Wavelength Zones
location:
Provides connectivity between your Wavelength Zone and the carrier network, which includes
any available devices from the carrier network.
Performs Network Address Translation (NAT) functions, such as translating IP addresses that
are public IP addresses stored in a network border group, from Wavelength Zones to carrier IP
addresses. These translation functions apply to inbound and outbound traffic.
Authorizes inbound traffic from a carrier network that is located in a specific location.
Authorizes outbound traffic to a carrier network and the internet.
NOTE
No inbound connection configuration exists from the internet to a Wavelength Zone

through the carrier gateway.
You can use the provided CloudFormation template to create a stack of the following AWS resources:
One carrier gateway that associates to the VPC ID in the template.
One public route table for the Wavelength Zone named as <ClusterName>-public-carrier.
Default IPv4 route entry in the new route table that targets the carrier gateway.
VPC gateway endpoint for an AWS Simple Storage Service (S3).
NOTE
850
NOTE
Prerequisites
Procedure
1. Go to the next section of the documentation named "CloudFormation template for the VPC
Carrier Gateway", and then copy the syntax from the CloudFormation template for VPC
Carrier Gateway template. Save the copied template syntax as a YAML file on your local
system. This template describes the VPC that your cluster requires.

--parameters \//
ParameterKey=VpcId,ParameterValue="${VpcId}" \ 3
ParameterKey=ClusterName,ParameterValue="${ClusterName}" 4
1 <stack_name> is the name for the CloudFormation stack, such as clusterName-vpc-

carrier-gw. You need the name of this stack if you remove the cluster.
that you saved.
3 <VpcId> is the VPC ID extracted from the CloudFormation stack output created in the
section named "Creating a VPC in AWS".
4 <ClusterName> is a custom value that prefixes to resources that the CloudFormation

stack creates. You can use the same name that is defined in the metadata.name section
of the install-config.yaml configuration file.
Example output
arn:aws:cloudformation:us-east-1:123456789012:stack/<stack_name>/dbedae40-2fd3-11eb-
820e-12a48460849f
Verification
Confirm that the CloudFormation template components exist by running the following
command:
851
following parameter. Ensure that you provide the parameter value to the other CloudFormation
templates that you run to create for your cluster.
PublicRou The ID of the Route Table in the Carrier infrastructure.

teTableId
See Amazon S3 in the AWS documentation.
6.15.7.4. CloudFormation template for the VPC Carrier Gateway
You can use the following CloudFormation template to deploy the Carrier Gateway on AWS Wavelength
infrastructure.
Example 6.96. CloudFormation template for VPC Carrier Gateway
Description: Template for Creating Wavelength Zone Gateway (Carrier Gateway).
Parameters:
VpcId:
Description: VPC ID to associate the Carrier Gateway.
Type: String
ClusterName:
Description: Cluster Name or Prefix name to prepend the tag Name for each subnet.
Type: String
Resources:
CarrierGateway:
Type: "AWS::EC2::CarrierGateway"
Properties:
VpcId: !Ref VpcId
Tags:
- Key: Name
Value: !Join ['-', [!Ref ClusterName, "cagw"]]
PublicRouteTable:
Properties:
VpcId: !Ref VpcId
Tags:
- Key: Name
Value: !Join ['-', [!Ref ClusterName, "public-carrier"]]
PublicRoute:
852
DependsOn: CarrierGateway
Properties:
CarrierGatewayId: !Ref CarrierGateway
S3Endpoint:
Properties:
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal: '*'
Action:
- '*'
Resource:
- '*'
RouteTableIds:
ServiceName: !Join
- ''
- - com.amazonaws.
- .s3
VpcId: !Ref VpcId
Outputs:
PublicRouteTableId:
6.15.7.5. Creating subnets in Wavelength Zones
Before you configure a machine set for edge compute nodes in your OpenShift Container Platform
cluster, you must create the subnets in Wavelength Zones. Complete the following procedure for each
Wavelength Zone that you want to deploy compute nodes to.
You can use the provided CloudFormation template and create a CloudFormation stack. You can then
use this stack to custom provision a subnet.
NOTE
Prerequisites
853
You opted in to the Wavelength Zones group.
Procedure
1. Go to the section of the documentation named "CloudFormation template for the VPC subnet",
and copy the syntax from the template. Save the copied template syntax as a YAML file on your
local system. This template describes the VPC that your cluster requires.

--parameters \
ParameterKey=VpcId,ParameterValue="${VPC_ID}" \ 3
ParameterKey=ClusterName,ParameterValue="${CLUSTER_NAME}" \ 4
ParameterKey=ZoneName,ParameterValue="${ZONE_NAME}" \ 5
ParameterKey=PublicRouteTableId,ParameterValue="${ROUTE_TABLE_PUB}" \ 6
ParameterKey=PublicSubnetCidr,ParameterValue="${SUBNET_CIDR_PUB}" \ 7
ParameterKey=PrivateRouteTableId,ParameterValue="${ROUTE_TABLE_PVT}" \ 8
ParameterKey=PrivateSubnetCidr,ParameterValue="${SUBNET_CIDR_PVT}" 9
1 <stack_name> is the name for the CloudFormation stack, such as cluster-wl-

<wavelength_zone_shortname>. You need the name of this stack if you remove the
cluster.
that you saved.
3 ${VPC_ID} is the VPC ID, which is the value VpcID in the output of the CloudFormation
template for the VPC.
4 ${ZONE_NAME} is the value of Wavelength Zones name to create the subnets.
5 ${CLUSTER_NAME} is the value of ClusterName to be used as a prefix of the new AWS

resource names.
6 ${ROUTE_TABLE_PUB} is the PublicRouteTableId extracted from the output of the

VPC’s carrier gateway CloudFormation stack.
7 ${SUBNET_CIDR_PUB} is a valid CIDR block that is used to create the public subnet. This
8 ${ROUTE_TABLE_PVT} is the PrivateRouteTableId extracted from the output of the

VPC’s CloudFormation stack.
9 ${SUBNET_CIDR_PVT} is a valid CIDR block that is used to create the private subnet. This
Example output
arn:aws:cloudformation:us-east-1:123456789012:stack/<stack_name>/dbedae40-820e-11eb-2fd3-
12a48460849f
854
Verification
Confirm that the template components exist by running the following command:
following parameters. Ensure that you provide these parameter values to the other
CloudFormation templates that you run to create for your cluster.
PublicSub The IDs of the public subnet created by the CloudFormation stack.
netId
PrivateSu The IDs of the private subnet created by the CloudFormation stack.
bnetId
6.15.7.6. CloudFormation template for the VPC subnet
You can use the following CloudFormation template to deploy the private and public subnets in a zone
on Wavelength Zones infrastructure.
Example 6.97. CloudFormation template for VPC subnets
Description: Template for Best Practice Subnets (Public and Private)
Parameters:
VpcId:
Description: VPC ID that comprises all the target subnets.
Type: String
ClusterName:
Description: Cluster name or prefix name to prepend the Name tag for each subnet.
Type: String
ZoneName:
Description: Zone Name to create the subnets, such as us-west-2-lax-1a.
Type: String
ConstraintDescription: ZoneName parameter must be specified.
PublicRouteTableId:
Description: Public Route Table ID to associate the public subnet.
Type: String
ConstraintDescription: PublicRouteTableId parameter must be specified.
PublicSubnetCidr:
AllowedPattern: ^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-
4][0-9]|25[0-5])(\/(1[6-9]|2[0-4]))$
Default: 10.0.128.0/20
855
Description: CIDR block for public subnet.

Type: String
PrivateRouteTableId:
Description: Private Route Table ID to associate the private subnet.
Type: String
ConstraintDescription: PrivateRouteTableId parameter must be specified.
PrivateSubnetCidr:
AllowedPattern: ^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-
4][0-9]|25[0-5])(\/(1[6-9]|2[0-4]))$
Default: 10.0.128.0/20
Description: CIDR block for private subnet.
Type: String
Resources:
PublicSubnet:
Properties:
VpcId: !Ref VpcId
CidrBlock: !Ref PublicSubnetCidr
Tags:
- Key: Name
Value: !Join ['-', [!Ref ClusterName, "public", !Ref ZoneName]]
Properties:
RouteTableId: !Ref PublicRouteTableId
PrivateSubnet:
Properties:
VpcId: !Ref VpcId
CidrBlock: !Ref PrivateSubnetCidr
Tags:
- Key: Name
Value: !Join ['-', [!Ref ClusterName, "private", !Ref ZoneName]]
Properties:
RouteTableId: !Ref PrivateRouteTableId
Outputs:
PublicSubnetId:
Description: Subnet ID of the public subnets.
Value:
!Join ["", [!Ref PublicSubnet]]
PrivateSubnetId:
856
Description: Subnet ID of the private subnets.

Value:
!Join ["", [!Ref PrivateSubnet]]
6.15.7.7. Modifying an installation configuration file to use AWS Wavelength Zones subnets
Modify your install-config.yaml file to include Wavelength Zones subnets.
Prerequisites
You created subnets by using the procedure "Creating subnets in Wavelength Zones".
Procedure
Modify the install-config.yaml configuration file by specifying Wavelength Zones subnets in

the platform.aws.subnets parameter.
Example installation configuration file with Wavelength Zones subnets
# ...
platform:
aws:
region: us-west-2
subnets: 1
- publicSubnetId-1
- publicSubnetId-2
- publicSubnetId-3
- privateSubnetId-1
- privateSubnetId-2
- privateSubnetId-3
- publicOrPrivateSubnetID-Wavelength-1
# ...
1 List of subnet IDs created in the zones: Availability and Wavelength Zones.
For more information about viewing the CloudFormation stacks that you created, see AWS
For more information about AWS profile and credential configuration, see Configuration and
credential file settings in the AWS documentation.
Next steps
6.15.8. Optional: Assign public IP addresses to edge compute nodes
857
If your workload requires deploying the edge compute nodes in public subnets on Wavelength Zones
infrastructure, you can configure the machine set manifests when installing a cluster.
AWS Wavelength Zones infrastructure accesses the network traffic in a specified zone, so applications
can take advantage of lower latency when serving end users that are closer to that zone.
The default setting that deploys compute nodes in private subnets might not meet your needs, so
consider creating edge compute nodes in public subnets when you want to apply more customization to
your infrastructure.
IMPORTANT
By default, OpenShift Container Platform deploy the compute nodes in private subnets.
For best performance, consider placing compute nodes in subnets that have their Public
IP addresses attached to the subnets.
You must create additional security groups, but ensure that you only open the groups'
rules over the internet when you really need to.
Procedure
1. Change to the directory that contains the installation program and generate the manifest files.
Ensure that the installation manifests get created at the openshift and manifests directory
level.
2. Edit the machine set manifest that the installation program generates for the Wavelength
Zones, so that the manifest gets deployed in public subnets. Specify true for the
spec.template.spec.providerSpec.value.publicIP parameter.
Example machine set manifest configuration for installing a cluster quickly in

Wavelength Zones
spec:
template:
spec:
providerSpec:
value:
publicIp: true
subnet:
filters:
- name: tag:Name
values:
- ${INFRA_ID}-public-${ZONE_NAME}
Example machine set manifest configuration for installing a cluster in an existing

VPC that has Wavelength Zones subnets
apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
name: <infrastructure_id>-edge-<zone>
858
spec:
template:
spec:
providerSpec:
value:
publicIp: true

IMPORTANT
Prerequisites
cluster.
Procedure
deployment:

--log-level=info 2

config.yaml file.
NOTE

Verification
859
IMPORTANT
Example output
...
IMPORTANT
6.15.10. Verifying the status of the deployed cluster

Verify that your OpenShift Container Platform successfully deployed on AWS Wavelength Zones.
6.15.10.1. Logging in to the cluster by using the CLI
Prerequisites
860
Procedure
$ oc whoami
Example output
system:admin
6.15.10.2. Logging in to the cluster by using the web console
Prerequisites
Procedure
installation host:
NOTE

NOTE
Example output
861

For more information about accessing and understanding the OpenShift Container Platform
web console, see Accessing the web console for more details about accessing and
understanding the OpenShift Container Platform web console.
6.15.10.3. Verifying nodes that were created with edge compute pool
After you install a cluster that uses AWS Wavelength Zones infrastructure, check the status of the
machine that was created by the machine set manifests created during installation.
1. To check the machine sets created from the subnet you added to the install-config.yaml file,
$ oc get machineset -n openshift-machine-api
Example output
NAME DESIRED CURRENT READY AVAILABLE AGE

cluster-7xw5g-edge-us-east-1-wl1-nyc-wlz-1 1 1 1 1 3h4m
cluster-7xw5g-worker-us-east-1a 1 1 1 1 3h4m
cluster-7xw5g-worker-us-east-1b 1 1 1 1 3h4m
cluster-7xw5g-worker-us-east-1c 1 1 1 1 3h4m
2. To check the machines that were created from the machine sets, run the following command:
$ oc get machines -n openshift-machine-api
Example output
NAME PHASE TYPE REGION ZONE AGE

cluster-7xw5g-edge-us-east-1-wl1-nyc-wlz-1-wbclh Running c5d.2xlarge us-east-1 us-
east-1-wl1-nyc-wlz-1 3h
cluster-7xw5g-master-0 Running m6i.xlarge us-east-1 us-east-1a
3h4m
cluster-7xw5g-master-1 Running m6i.xlarge us-east-1 us-east-1b
3h4m
cluster-7xw5g-master-2 Running m6i.xlarge us-east-1 us-east-1c
3h4m
cluster-7xw5g-worker-us-east-1a-rtp45 Running m6i.xlarge us-east-1 us-east-1a
3h
cluster-7xw5g-worker-us-east-1b-glm7c Running m6i.xlarge us-east-1 us-east-1b
3h
cluster-7xw5g-worker-us-east-1c-qfvz4 Running m6i.xlarge us-east-1 us-east-1c
3h
3. To check nodes with edge roles, run the following command:
862
$ oc get nodes -l node-role.kubernetes.io/edge
Example output

ip-10-0-207-188.ec2.internal Ready edge,worker 172m v1.25.2+d2e245f

For more information about the Telemetry service, see About remote health monitoring.
Next steps
If necessary, you can opt out of remote health .

AWS OUTPOSTS
In OpenShift Container Platform version 4.14, you could install a cluster on Amazon Web Services (AWS)
with compute nodes running in AWS Outposts as a Technology Preview. As of OpenShift Container
Platform version 4.15, this installation method is no longer supported.
Instead, you can install a cluster on AWS into an existing VPC and provision compute nodes on AWS
Outposts as a postinstallation configuration task.
For more information, see Extending an AWS VPC cluster into an AWS Outpost
6.17. INSTALLING A THREE-NODE CLUSTER ON AWS

In OpenShift Container Platform version 4.15, you can install a three-node cluster on Amazon Web
Services (AWS). A three-node cluster consists of three control plane machines, which also act as
compute machines. This type of cluster provides a smaller, more resource efficient cluster, for cluster
administrators and developers to use for testing, development, and production.
You can install a three-node cluster using either installer-provisioned or user-provisioned infrastructure.
NOTE
Deploying a three-node cluster using an AWS Marketplace image is not supported.
863
6.17.1. Configuring a three-node cluster

You configure a three-node cluster by setting the number of worker nodes to 0 in the install-
config.yaml file before deploying the cluster. Setting the number of worker nodes to 0 ensures that the
control plane machines are schedulable. This allows application workloads to be scheduled to run from
the control plane nodes.
NOTE
Because application workloads run from control plane nodes, additional subscriptions are
required, as the control plane nodes are considered to be compute nodes.
Prerequisites
Procedure
1. Set the number of compute replicas to 0 in your install-config.yaml file, as shown in the
following compute stanza:
Example install-config.yaml file for a three-node cluster
apiVersion: v1
compute:
- name: worker
platform: {}
replicas: 0
# ...
2. If you are deploying a cluster with user-provisioned infrastructure:
After you create the Kubernetes manifest files, make sure that the
spec.mastersSchedulable parameter is set to true in cluster-scheduler-02-config.yml
file. You can locate this file in <installation_directory>/manifests. For more information,
see "Creating the Kubernetes manifest and Ignition config files" in "Installing a cluster on
user-provisioned infrastructure in AWS by using CloudFormation templates".
Do not create additional worker nodes.
Example cluster-scheduler-02-config.yml file for a three-node cluster
kind: Scheduler
metadata:
name: cluster
spec:
mastersSchedulable: true
policy:
name: ""
status: {}
864
6.17.2. Next steps

Installing a cluster on AWS with customizations
Installing a cluster on user-provisioned infrastructure in AWS by using CloudFormation

templates
6.18. UNINSTALLING A CLUSTER ON AWS

You can remove a cluster that you deployed to Amazon Web Services (AWS).

NOTE
Prerequisites
Procedure

NOTE
6.18.2. Deleting Amazon Web Services resources with the Cloud Credential Operator
utility
After uninstalling an OpenShift Container Platform cluster that uses short-term credentials managed
865
outside the cluster, you can use the CCO utility (ccoctl) to remove the Amazon Web Services (AWS)
resources that ccoctl created during installation.
Prerequisites
Uninstall an OpenShift Container Platform cluster on AWS that uses short-term credentials.
Procedure
Delete the AWS resources that ccoctl created by running the following command:
$ ccoctl aws delete \

--name=<name> \ 1
--region=<aws_region> 2
1 <name> matches the name that was originally used to create and tag the cloud resources.
2 <aws_region> is the AWS region in which to delete cloud resources.
Example output
2021/04/08 17:50:41 Identity Provider object .well-known/openid-configuration deleted from

the bucket <name>-oidc
2021/04/08 17:50:42 Identity Provider object keys.json deleted from the bucket <name>-oidc
2021/04/08 17:50:43 Identity Provider bucket <name>-oidc deleted
2021/04/08 17:51:05 Policy <name>-openshift-cloud-credential-operator-cloud-credential-o
associated with IAM Role <name>-openshift-cloud-credential-operator-cloud-credential-o
deleted
2021/04/08 17:51:05 IAM Role <name>-openshift-cloud-credential-operator-cloud-credential-
o deleted
2021/04/08 17:51:07 Policy <name>-openshift-cluster-csi-drivers-ebs-cloud-credentials
associated with IAM Role <name>-openshift-cluster-csi-drivers-ebs-cloud-credentials deleted
2021/04/08 17:51:07 IAM Role <name>-openshift-cluster-csi-drivers-ebs-cloud-credentials
deleted
2021/04/08 17:51:08 Policy <name>-openshift-image-registry-installer-cloud-credentials
associated with IAM Role <name>-openshift-image-registry-installer-cloud-credentials
deleted
2021/04/08 17:51:08 IAM Role <name>-openshift-image-registry-installer-cloud-credentials
deleted
2021/04/08 17:51:09 Policy <name>-openshift-ingress-operator-cloud-credentials associated
with IAM Role <name>-openshift-ingress-operator-cloud-credentials deleted
2021/04/08 17:51:10 IAM Role <name>-openshift-ingress-operator-cloud-credentials deleted
2021/04/08 17:51:11 Policy <name>-openshift-machine-api-aws-cloud-credentials associated
with IAM Role <name>-openshift-machine-api-aws-cloud-credentials deleted
2021/04/08 17:51:11 IAM Role <name>-openshift-machine-api-aws-cloud-credentials deleted
2021/04/08 17:51:39 Identity Provider with ARN arn:aws:iam::<aws_account_id>:oidc-
provider/<name>-oidc.s3.<aws_region>.amazonaws.com deleted
Verification
To verify that the resources are deleted, query AWS. For more information, refer to AWS
866
To verify that the resources are deleted, query AWS. For more information, refer to AWS
documentation.
6.18.3. Deleting a cluster with a configured AWS Local Zone infrastructure

After you install a cluster on Amazon Web Services (AWS) into an existing Virtual Private Cloud (VPC),
and you set subnets for each Local Zone location, you can delete the cluster and any AWS resources
associated with it.
The example in the procedure assumes that you created a VPC and its subnets by using a
CloudFormation template.
Prerequisites
You know the name of the CloudFormation stacks, <local_zone_stack_name> and

<vpc_stack_name>, that were used during the creation of the network. You need the name of
the stack to delete the cluster.
You have access rights to the directory that contains the installation files that were created by
Your account includes a policy that provides you with permissions to delete the CloudFormation
stack.
Procedure
1. Change to the directory that contains the stored installation program, and delete the cluster by
using the destroy cluster command:
$ ./openshift-install destroy cluster --dir <installation_directory> \ 1

--log-level=debug 2
1 For <installation_directory>, specify the directory that stored any files created by the
2 To view different log details, specify error, info, or warn instead of debug.
2. Delete the CloudFormation stack for the Local Zone subnet:
$ aws cloudformation delete-stack --stack-name <local_zone_stack_name>
3. Delete the stack of resources that represent the VPC:
$ aws cloudformation delete-stack --stack-name <vpc_stack_name>
Verification
Check that you removed the stack resources by issuing the following commands in the AWS CLI.
The AWS CLI outputs that no template component exists.
$ aws cloudformation describe-stacks --stack-name <local_zone_stack_name>
$ aws cloudformation describe-stacks --stack-name <vpc_stack_name>
867
Opt into AWS Local Zones
AWS Local Zones available locations
AWS Local Zones features
6.19. INSTALLATION CONFIGURATION PARAMETERS FOR AWS

Before you deploy an OpenShift Container Platform cluster on AWS, you provide parameters to
customize your cluster and the platform that hosts it. When you create the install-config.yaml file, you
provide values for the required parameters through the command line. You can then modify the install-
config.yaml file to customize your cluster further.
6.19.1. Available installation configuration parameters for AWS

The following tables specify the required, optional, and AWS-specific installation configuration
NOTE

versions.
868

combination of the
baseDomain and
<metadata.name>.

consumed.
name: subdomains of
{{.metadata.name}}.
{{.baseDomain}}.

alibabacloud, aws,
ibmcloud, nutanix,
869

"auth":"b3Blb=",
}
}
}
NOTE

NOTE
You cannot modify

by the networking
object after
installation.

870

networking:
with a host prefix of /23.
clusterNetwork:
- cidr: 10.128.0.0/14
hostPrefix: 23

An IPv4 network.

23 provides 510 (2^(32 - 23) - 2) pod
IP addresses.
serviceNetwork:
networking:
serviceNetwork:
- 172.30.0.0/16

networking:
machineNetwork: If you specify multiple IP address
networking:
machineNetwork:
- cidr: 10.0.0.0/16

NOTE
the CIDR that the
in.
871



et:


section.

872

and arm64. Not all installation options
support the 64-bit ARM architecture.
To verify if your installation option is
supported on your platform, see
Supported installation methods for
different platforms in Selecting a
cluster installation method and
preparing it for users.

cores.
IMPORTANT
If you disable
simultaneous
that your capacity
the dramatically
decreased machine
performance.

name:
value.

873



cores.
IMPORTANT
If you disable
simultaneous
that your capacity
the dramatically
decreased machine
performance.
874

name:

parameter value.

replicas:

875

IMPORTANT
To enable FIPS mode

must run the
from a Red Hat
Enterprise Linux
(RHEL) computer
in FIPS mode. For
more information
about configuring
FIPS mode on RHEL,
see Installing the
Enterprise Linux
(RHEL) or Red Hat
Enterprise Linux
CoreOS (RHCOS)
OpenShift Container
Platform core
components use the
RHEL cryptographic
libraries that have
been submitted to
NIST for FIPS 140-
only the x86_64,
ppc64le, and s390x
architectures.
NOTE

File storage, you
cannot enable FIPS
mode.

876


ces:
mirrors:
Required to set the NLB load balancer Classic or NLB . The default value is
platform: type in AWS. Valid values are Classic Classic.
aws: or NLB . If no value is specified, the
lbType: installation program defaults to
Classic. The installation program sets
the value provided here in the ingress
cluster configuration object. If you do
not specify a load balancer type for
other Ingress Controllers, they use the
type set in this parameter.
How to publish or expose the user- Internal or External. To deploy a

publish: facing endpoints of your cluster, such private cluster, which cannot be
as the Kubernetes API, OpenShift accessed from the internet, set
routes. publish to Internal . The default
value is External.

NOTE
For production
OpenShift Container
which you want to
SSH key that your
ssh-agent process
uses.
NOTE
877
NOTE
If your AWS account has service control policies (SCP) enabled, you must
configure the credentialsMode parameter to Mint, Passthrough, or Manual.
IMPORTANT
Setting this parameter to Manual enables alternatives to storing administrator-

level secrets in the kube-system project, which require additional configuration
steps. For more information, see "Alternatives to storing administrator-level
secrets in the kube-system project".
6.19.1.4. Optional AWS configuration parameters
Optional AWS configuration parameters are described in the following table:
Table 6.28. Optional AWS parameters
The AWS AMI used to boot Any published or custom RHCOS AMI that belongs
compute: compute machines for the to the set AWS region. See RHCOS AMIs for AWS
platform: cluster. This is required for infrastructure for available AMI IDs.
aws: regions that require a custom
amiID: RHCOS AMI.
A pre-existing AWS IAM role The name of a valid AWS IAM role.
compute: applied to the compute
platform: machine pool instance
aws: profiles. You can use these
iamRole: fields to match naming
schemes and include
predefined permissions
boundaries for your IAM roles.
If undefined, the installation
program creates a new IAM
role.
The Input/Output Operations Integer, for example 4000.

compute: Per Second (IOPS) that is
platform: reserved for the root volume.
aws:
rootVolume:
iops:
878
The size in GiB of the root Integer, for example 500.

compute: volume.
platform:
aws:
rootVolume:
size:
The type of the root volume. Valid AWS EBS volume type, such as io1.
compute:
platform:
aws:
rootVolume:
type:
The Amazon Resource Name Valid key ID or the key ARN.

compute: (key ARN) of a KMS key. This
platform: is required to encrypt
aws: operating system volumes of
worker nodes with a specific
rootVolume: KMS key.
kmsKeyARN:
The EC2 instance type for the Valid AWS instance type, such as m4.2xlarge. See
compute: compute machines. the Supported AWS machine types table that
platform: follows.
aws:
type:
The availability zones where A list of valid AWS availability zones, such as us-
compute: the installation program east-1c, in a YAML sequence.
platform: creates machines for the
aws: compute machine pool. If you
zones: provide your own VPC, you
must provide a subnet in that
availability zone.
879
The AWS region that the Any valid AWS region, such as us-east-1. You can
compute: installation program creates use the AWS CLI to access the regions available
aws: compute resources in. based on your selected instance type. For example:
region:
aws ec2 describe-instance-type-offerings --
filters Name=instance-
type,Values=c7g.xlarge
IMPORTANT
When running on ARM based AWS

instances, ensure that you enter a
region where AWS Graviton
processors are available. See Global
availability map in the AWS
documentation. Currently, AWS
Graviton3 processors are only
available in some regions.
The AWS AMI used to boot Any published or custom RHCOS AMI that belongs
controlPlane: control plane machines for the to the set AWS region. See RHCOS AMIs for AWS
platform: cluster. This is required for infrastructure for available AMI IDs.
aws: regions that require a custom
amiID: RHCOS AMI.
A pre-existing AWS IAM role The name of a valid AWS IAM role.
controlPlane: applied to the control plane
platform: machine pool instance
aws: profiles. You can use these
iamRole: fields to match naming
schemes and include
predefined permissions
boundaries for your IAM roles.
If undefined, the installation
program creates a new IAM
role.
The Input/Output Operations Integer, for example 4000.

controlPlane: Per Second (IOPS) that is
platform: reserved for the root volume
aws: on control plane machines.
rootVolume:
iops:
880
The size in GiB of the root Integer, for example 500.

controlPlane: volume for control plane
platform: machines.
aws:
rootVolume:
size:
The type of the root volume Valid AWS EBS volume type, such as io1.
controlPlane: for control plane machines.
platform:
aws:
rootVolume:
type:
The Amazon Resource Name Valid key ID and the key ARN.
controlPlane: (key ARN) of a KMS key. This
platform: is required to encrypt
aws: operating system volumes of
control plane nodes with a
rootVolume: specific KMS key.
kmsKeyARN:
The EC2 instance type for the Valid AWS instance type, such as m6i.xlarge. See
controlPlane: control plane machines. the Supported AWS machine types table that
platform: follows.
aws:
type:
The availability zones where A list of valid AWS availability zones, such as us-
controlPlane: the installation program east-1c, in a YAML sequence.
platform: creates machines for the
aws: control plane machine pool.
zones:
The AWS region that the Valid AWS region, such as us-east-1.
controlPlane: installation program creates
aws: control plane resources in.
region:
881
The AWS AMI used to boot all Any published or custom RHCOS AMI that belongs
platform: machines for the cluster. If to the set AWS region. See RHCOS AMIs for AWS
aws: set, the AMI must belong to infrastructure for available AMI IDs.
amiID: the same region as the
cluster. This is required for
regions that require a custom
RHCOS AMI.
An existing Route 53 private String, for example Z3URY6TWQ91KVV .

platform: hosted zone for the cluster.
aws: You can only use a pre-
existing hosted zone when
hostedZone: also supplying your own VPC.
The hosted zone must already
be associated with the user-
provided VPC before
installation. Also, the domain
of the hosted zone must be
the cluster domain or a parent
of the cluster domain. If
undefined, the installation
program creates a new hosted
zone.
An Amazon Resource Name String, for example

platform: (ARN) for an existing IAM role arn:aws:iam::1234567890:role/shared-vpc-
aws: in the account containing the role.
specified hosted zone. The
hostedZoneR installation program and
ole: cluster operators will assume
this role when performing
operations on the hosted
zone. This parameter should
only be used if you are
installing a cluster into a
shared VPC.
The AWS service endpoint Valid AWS service endpoint name and valid AWS
platform: name and URL. Custom service endpoint URL.
aws: endpoints are only required
for cases where alternative
serviceEndpoi AWS endpoints, like FIPS,
nts: must be used. Custom API
- name: endpoints can be specified for
url: EC2, S3, IAM, Elastic Load
Balancing, Tagging, Route 53,
and STS AWS services.
882
A map of keys and values that Any valid YAML map, such as key value pairs in the
platform: the installation program adds <key>: <value> format. For more information
aws: as tags to all resources that it about AWS tags, see Tagging Your Amazon EC2
userTags: creates. Resources in the AWS documentation.
NOTE
You can add up to 25 user defined

tags during installation. The
remaining 25 tags are reserved for
A flag that directs in-cluster Boolean values, for example true or false.
platform: Operators to include the
aws: specified user tags in the tags
of the AWS resources that the
propagateUse Operators create.
rTags:
If you provide the VPC Valid subnet IDs.

platform: instead of allowing the
aws: installation program to create
subnets: the VPC for you, specify the
subnet for the cluster to use.
The subnet must be part of
the same
machineNetwork[].cidr
ranges that you specify.
For a standard cluster, specify

a public and a private subnet
for each availability zone.
For a private cluster, specify a

private subnet for each
availability zone.
For clusters that use AWS

Local Zones, you must add
AWS Local Zone subnets to
this list to ensure edge
machine pool creation.
Prevents the S3 bucket from true or false. The default value is false, which
platform: being deleted after results in the S3 bucket being deleted.
aws: completion of bootstrapping.
preserveBoots
trapIgnition:
883
CHAPTER 7. INSTALLING ON AZURE
7.1. PREPARING TO INSTALL ON AZURE
processes.
users.
7.1.2. Requirements for installing OpenShift Container Platform on Azure

Before installing OpenShift Container Platform on Microsoft Azure, you must configure an Azure
account. See Configuring an Azure account for details about account configuration, account limits,
public DNS zone configuration, required roles, creating service principals, and supported Azure regions.
Alternatives to storing administrator-level secrets in the kube-system project for other options.
7.1.3. Choosing a method to install OpenShift Container Platform on Azure

You can install a cluster on Azure infrastructure that is provisioned by the OpenShift Container Platform
Installing a cluster quickly on Azure: You can install OpenShift Container Platform on Azure
Installing a customized cluster on Azure: You can install a customized cluster on Azure
Installing a cluster on Azure with network customizations: You can customize your OpenShift
Installing a cluster on Azure into an existing VNet: You can install OpenShift Container
Platform on an existing Azure Virtual Network (VNet) on Azure. You can use this installation
method if you have constraints set by the guidelines of your company, such as limits when
884
creating new accounts or infrastructure.
Installing a private cluster on Azure: You can install a private cluster into an existing Azure
Virtual Network (VNet) on Azure. You can use this method to deploy OpenShift Container
Platform on an internal network that is not visible to the internet.
Installing a cluster on Azure into a government region: OpenShift Container Platform can be
deployed into Microsoft Azure Government (MAG) regions that are specifically designed for US
government agencies at the federal, state, and local level, as well as contractors, educational
institutions, and other US customers that must run sensitive workloads on Azure.
You can install a cluster on Azure infrastructure that you provision, by using the following method:
Installing a cluster on Azure using ARM templates: You can install OpenShift Container
Platform on Azure by using infrastructure that you provide. You can use the provided Azure
Resource Manager (ARM) templates to assist with an installation.
7.1.4. Next steps

Configuring an Azure account
7.2. CONFIGURING AN AZURE ACCOUNT

Before you can install OpenShift Container Platform, you must configure a Microsoft Azure account to
meet installation requirements.
IMPORTANT
All Azure resources that are available through public endpoints are subject to resource
name restrictions, and you cannot create resources that use certain terms. For a list of
terms that Azure restricts, see Resolve reserved resource name errors in the Azure
documentation.
7.2.1. Azure account limits

The OpenShift Container Platform cluster uses a number of Microsoft Azure components, and the
default Azure subscription and service limits, quotas, and constraints affect your ability to install
OpenShift Container Platform clusters.
IMPORTANT
Default limits vary by offer category types, such as Free Trial and Pay-As-You-Go, and by
series, such as Dv2, F, and G. For example, the default for Enterprise Agreement
subscriptions is 350 cores.
Check the limits for your subscription type and if necessary, increase quota limits for your
account before you install a default cluster on Azure.
The following table summarizes the Azure components whose limits can impact your ability to install and
885
Compone Number of Default Azure Description

nt components limit
required by
default
vCPU 44 20 per region A default cluster requires 44 vCPUs, so you must

increase the account limit.
By default, each cluster creates the following

instances:

after installation
Three control plane machines
Three compute machines
Because the bootstrap and control plane machines

use Standard_D8s_v3 virtual machines, which use
8 vCPUs, and the compute machines use
Standard_D4s_v3 virtual machines, which use 4
vCPUs, a default cluster requires 44 vCPUs. The
bootstrap node VM, which uses 8 vCPUs, is used
only during installation.
To deploy more worker nodes, enable autoscaling,

deploy large workloads, or use a different instance
type, you must further increase the vCPU limit for
your account to ensure that your cluster can deploy
the machines that you require.
OS Disk 7 Each cluster machine must have a minimum of 100

GB of storage and 300 IOPS. While these are the
minimum supported values, faster storage is
recommended for production clusters and clusters
with intensive workloads. For more information
about optimizing storage for performance, see the
page titled "Optimizing storage" in the "Scalability
and performance" section.
VNet 1 1000 per region Each default cluster requires one Virtual Network
(VNet), which contains two subnets.
Network 7 65,536 per Each default cluster requires seven network

interfaces region interfaces. If you create more machines or your
deployed workloads create load balancers, your
cluster uses more network interfaces.
886

nt components limit
required by
default
Network 2 5000 Each cluster creates network security groups for

security each subnet in the VNet. The default cluster creates
groups network security groups for the control plane and for
the compute node subnets:
co Allows the control plane machines to be

ntr reached on port 6443 from anywhere
olp
lan
e
no Allows worker nodes to be reached from the

de internet on ports 80 and 443
Network 3 1000 per region Each cluster creates the following load balancers:
load
balancers
def Public IP address that load balances requests
aul to ports 80 and 443 across worker machines
t
int Private IP address that load balances

ern requests to ports 6443 and 22623 across
al control plane machines
ext Public IP address that load balances requests

ern to port 6443 across control plane machines
al
If your applications create more Kubernetes

LoadBalancer service objects, your cluster uses
more load balancers.
Public IP 3 Each of the two public load balancers uses a public

addresses IP address. The bootstrap machine also uses a public
IP address so that you can SSH into the machine to
troubleshoot issues during installation. The IP
address for the bootstrap node is used only during
installation.
Private IP 7 The internal load balancer, each of the three control

addresses plane machines, and each of the three worker
machines each use a private IP address.
887

nt components limit
required by
default
Spot VM 0 20 per region This is an optional component. To use spot VMs, you
vCPUs must increase the Azure default limit to at least
(optional) If you configure twice the number of compute nodes in your cluster.
spot VMs, your
cluster must have
two spot VM
NOTE
vCPUs for every
Using spot VMs for control plane
compute node. nodes is not recommended.
Optimizing storage .
7.2.2. Configuring a public DNS zone in Azure

To install OpenShift Container Platform, the Microsoft Azure account you use must have a dedicated
public hosted DNS zone in your account. This zone must be authoritative for the domain. This service
Procedure
registrar or obtain a new one through Azure or another source.
NOTE
For more information about purchasing domains through Azure, see Buy a
custom domain name for Azure App Service in the Azure documentation.
2. If you are using an existing domain and registrar, migrate its DNS to Azure. See Migrate an active
DNS name to Azure App Service in the Azure documentation.
3. Configure DNS for your domain. Follow the steps in the Tutorial: Host your domain in Azure
DNS in the Azure documentation to create a public hosted zone for your domain or subdomain,
extract the new authoritative name servers, and update the registrar records for the name
servers that your domain uses.
4. If you use a subdomain, follow your company’s procedures to add its delegation records to the
parent domain.
7.2.3. Increasing Azure account limits

To increase an account limit, file a support request on the Azure portal.
NOTE
888
NOTE
You can increase only one type of quota per support request.
Procedure
1. From the Azure portal, click Help + support in the lower left corner.
2. Click New support request and then select the required values:
a. From the Issue type list, select Service and subscription limits (quotas).
b. From the Subscription list, select the subscription to modify.
c. From the Quota type list, select the quota to increase. For example, select Compute-VM
(cores-vCPUs) subscription limit increases to increase the number of vCPUs, which is
required to install a cluster.
d. Click Next: Solutions.
3. On the Problem Details page, provide the required information for your quota increase:
a. Click Provide details and provide the required details in the Quota details window.
b. In the SUPPORT METHOD and CONTACT INFO sections, provide the issue severity and
your contact details.
4. Click Next: Review + create and then click Create.
7.2.4. Recording the subscription and tenant IDs

The installation program requires the subscription and tenant IDs that are associated with your Azure
account. You can use the Azure CLI to gather this information.
Prerequisites
You have installed or updated the Azure CLI.
Procedure
1. Log in to the Azure CLI by running the following command:
$ az login
2. Ensure that you are using the right subscription:
a. View a list of available subscriptions by running the following command:
$ az account list --refresh
Example output
[
{
"cloudName": "AzureCloud",
889
"id": "8xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
"isDefault": true,
"name": "Subscription Name 1",
"state": "Enabled",
"tenantId": "6xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
"user": {
"name": "you@example.com",
"type": "user"
}
},
{
"isDefault": false,
"state": "Enabled",
"user": {
"name": "you2@example.com",
"type": "user"
}
}
]
b. View the details of the active account, and confirm that this is the subscription you want to
use, by running the following command:
$ az account show
Example output
{
"environmentName": "AzureCloud",
"isDefault": true,
"state": "Enabled",
"user": {
"type": "user"
}
}
3. If you are not using the right subscription:
a. Change the active subscription by running the following command:
$ az account set -s <subscription_id>
b. Verify that you are using the subscription you need by running the following command:
$ az account show
890
Example output
{
"isDefault": true,
"state": "Enabled",
"user": {
"type": "user"
}
}
4. Record the id and tenantId parameter values from the output. You require these values to
install an OpenShift Container Platform cluster.
7.2.5. Supported identities to access Azure resources

An OpenShift Container Platform cluster requires an Azure identity to create and manage Azure
resources. As such, you need one of the following types of identities to complete the installation:
A service principal
A system-assigned managed identity
A user-assigned managed identity
7.2.5.1. Required Azure roles
resources. Before you create the identity, verify that your environment meets the following
requirements:
The Azure account that you use to create the identity is assigned the User Access
Administrator and Contributor roles. These roles are required when:
Creating a service principal or user-assigned managed identity.
Enabling a system-assigned managed identity on a virtual machine.
If you are going to use a service principal to complete the installation, verify that the Azure
account that you use to create the identity is assigned the
microsoft.directory/servicePrincipals/createAsOwner permission in Azure Active Directory.
To set roles on the Azure portal, see the Manage access to Azure resources using RBAC and the Azure
portal in the Azure documentation.
7.2.5.2. Required Azure permissions for installer-provisioned infrastructure
The installation program requires access to an Azure service principal or managed identity with the
necessary permissions to deploy the cluster and to maintain its daily operation. These permissions must
be granted to the Azure subscription that is associated with the identity.
891
The following options are available to you:
You can assign the identity the Contributor and User Access Administrator roles. Assigning
these roles is the quickest way to grant all of the required permissions.
For more information about assigning roles, see the Azure documentation for managing access
to Azure resources using the Azure portal.
If your organization’s security policies require a more restrictive set of permissions, you can
create a custom role with the necessary permissions.
The following permissions are required for creating an OpenShift Container Platform cluster on
Microsoft Azure.
Example 7.1. Required permissions for creating authorization resources
Microsoft.Authorization/policies/audit/action
Microsoft.Authorization/policies/auditIfNotExists/action
Microsoft.Authorization/roleAssignments/read
Microsoft.Authorization/roleAssignments/write
Example 7.2. Required permissions for creating compute resources
Microsoft.Compute/availabilitySets/write
Microsoft.Compute/availabilitySets/read
Microsoft.Compute/disks/beginGetAccess/action
Microsoft.Compute/disks/delete
Microsoft.Compute/disks/read
Microsoft.Compute/disks/write
Microsoft.Compute/galleries/images/read
Microsoft.Compute/galleries/images/versions/read
Microsoft.Compute/galleries/images/versions/write
Microsoft.Compute/galleries/images/write
Microsoft.Compute/galleries/read
Microsoft.Compute/galleries/write
Microsoft.Compute/snapshots/read
Microsoft.Compute/snapshots/write
Microsoft.Compute/snapshots/delete
Microsoft.Compute/virtualMachines/delete
892
Microsoft.Compute/virtualMachines/powerOff/action
Microsoft.Compute/virtualMachines/read
Microsoft.Compute/virtualMachines/write
Example 7.3. Required permissions for creating identity management resources
Microsoft.ManagedIdentity/userAssignedIdentities/assign/action
Microsoft.ManagedIdentity/userAssignedIdentities/read
Microsoft.ManagedIdentity/userAssignedIdentities/write
Example 7.4. Required permissions for creating network resources
Microsoft.Network/dnsZones/A/write
Microsoft.Network/dnsZones/CNAME/write
Microsoft.Network/dnszones/CNAME/read
Microsoft.Network/dnszones/read
Microsoft.Network/loadBalancers/backendAddressPools/join/action
Microsoft.Network/loadBalancers/backendAddressPools/read
Microsoft.Network/loadBalancers/backendAddressPools/write
Microsoft.Network/loadBalancers/read
Microsoft.Network/loadBalancers/write
Microsoft.Network/networkInterfaces/delete
Microsoft.Network/networkInterfaces/join/action
Microsoft.Network/networkInterfaces/read
Microsoft.Network/networkInterfaces/write
Microsoft.Network/networkSecurityGroups/join/action
Microsoft.Network/networkSecurityGroups/read
Microsoft.Network/networkSecurityGroups/securityRules/delete
Microsoft.Network/networkSecurityGroups/securityRules/read
Microsoft.Network/networkSecurityGroups/securityRules/write
Microsoft.Network/networkSecurityGroups/write
Microsoft.Network/privateDnsZones/A/read
893
Microsoft.Network/privateDnsZones/A/write
Microsoft.Network/privateDnsZones/A/delete
Microsoft.Network/privateDnsZones/SOA/read
Microsoft.Network/privateDnsZones/read
Microsoft.Network/privateDnsZones/virtualNetworkLinks/read
Microsoft.Network/privateDnsZones/virtualNetworkLinks/write
Microsoft.Network/privateDnsZones/write
Microsoft.Network/publicIPAddresses/delete
Microsoft.Network/publicIPAddresses/join/action
Microsoft.Network/publicIPAddresses/read
Microsoft.Network/publicIPAddresses/write
Microsoft.Network/virtualNetworks/join/action
Microsoft.Network/virtualNetworks/read
Microsoft.Network/virtualNetworks/subnets/join/action
Microsoft.Network/virtualNetworks/subnets/read
Microsoft.Network/virtualNetworks/subnets/write
Microsoft.Network/virtualNetworks/write
NOTE
The following permissions are not required to create the private OpenShift Container
Platform cluster on Azure.
Example 7.5. Required permissions for checking the health of resources
Microsoft.Resourcehealth/healthevent/Activated/action
Microsoft.Resourcehealth/healthevent/InProgress/action
Microsoft.Resourcehealth/healthevent/Pending/action
Microsoft.Resourcehealth/healthevent/Resolved/action
894
Microsoft.Resourcehealth/healthevent/Updated/action
Example 7.6. Required permissions for creating a resource group
Microsoft.Resources/subscriptions/resourceGroups/read
Microsoft.Resources/subscriptions/resourcegroups/write
Example 7.7. Required permissions for creating resource tags
Microsoft.Resources/tags/write
Example 7.8. Required permissions for creating storage resources
Microsoft.Storage/storageAccounts/blobServices/read
Microsoft.Storage/storageAccounts/blobServices/containers/write
Microsoft.Storage/storageAccounts/fileServices/read
Microsoft.Storage/storageAccounts/fileServices/shares/read
Microsoft.Storage/storageAccounts/fileServices/shares/write
Microsoft.Storage/storageAccounts/fileServices/shares/delete
Microsoft.Storage/storageAccounts/listKeys/action
Microsoft.Storage/storageAccounts/read
Microsoft.Storage/storageAccounts/write
Example 7.9. Optional permissions for creating a private storage endpoint for the image
registry
Microsoft.Network/privateEndpoints/write
Microsoft.Network/privateEndpoints/read
Microsoft.Network/privateEndpoints/privateDnsZoneGroups/write
Microsoft.Network/privateEndpoints/privateDnsZoneGroups/read
Microsoft.Network/privateDnsZones/join/action
Microsoft.Storage/storageAccounts/PrivateEndpointConnectionsApproval/action
Example 7.10. Optional permissions for creating marketplace virtual machine resources
Microsoft.MarketplaceOrdering/offertypes/publishers/offers/plans/agreements/read
895
Microsoft.MarketplaceOrdering/offertypes/publishers/offers/plans/agreements/write
Example 7.11. Optional permissions for creating compute resources
Microsoft.Compute/images/read
Microsoft.Compute/images/write
Microsoft.Compute/images/delete
Example 7.12. Optional permissions for enabling user-managed encryption
Microsoft.Compute/diskEncryptionSets/read
Microsoft.Compute/diskEncryptionSets/write
Microsoft.Compute/diskEncryptionSets/delete
Microsoft.KeyVault/vaults/read
Microsoft.KeyVault/vaults/write
Microsoft.KeyVault/vaults/delete
Microsoft.KeyVault/vaults/deploy/action
Microsoft.KeyVault/vaults/keys/read
Microsoft.KeyVault/vaults/keys/write
Microsoft.Features/providers/features/register/action
Example 7.13. Optional permissions for installing a cluster using theNatGateway outbound type
Microsoft.Network/natGateways/read
Microsoft.Network/natGateways/write
Example 7.14. Optional permissions for installing a private cluster with Azure Network Address
Translation (NAT)
Microsoft.Network/natGateways/join/action
Microsoft.Network/natGateways/read
Microsoft.Network/natGateways/write
Example 7.15. Optional permissions for installing a private cluster with Azure firewall
Microsoft.Network/azureFirewalls/applicationRuleCollections/write
896
Microsoft.Network/azureFirewalls/read
Microsoft.Network/azureFirewalls/write
Microsoft.Network/routeTables/join/action
Microsoft.Network/routeTables/read
Microsoft.Network/routeTables/routes/read
Microsoft.Network/routeTables/routes/write
Microsoft.Network/routeTables/write
Microsoft.Network/virtualNetworks/peer/action
Microsoft.Network/virtualNetworks/virtualNetworkPeerings/read
Microsoft.Network/virtualNetworks/virtualNetworkPeerings/write
Example 7.16. Optional permission for running gather bootstrap
Microsoft.Compute/virtualMachines/instanceView/read
The following permissions are required for deleting an OpenShift Container Platform cluster on
Microsoft Azure. You can use the same permissions to delete a private OpenShift Container Platform
cluster on Azure.
Example 7.17. Required permissions for deleting authorization resources
Microsoft.Authorization/roleAssignments/delete
Example 7.18. Required permissions for deleting compute resources
Microsoft.Compute/galleries/delete
Microsoft.Compute/galleries/images/delete
Microsoft.Compute/galleries/images/versions/delete
Example 7.19. Required permissions for deleting identity management resources
Microsoft.ManagedIdentity/userAssignedIdentities/delete
Example 7.20. Required permissions for deleting network resources
897
Microsoft.Network/dnsZones/A/read
Microsoft.Network/dnsZones/A/delete
Microsoft.Network/dnsZones/CNAME/read
Microsoft.Network/dnsZones/CNAME/delete
Microsoft.Network/loadBalancers/delete
Microsoft.Network/networkSecurityGroups/delete
Microsoft.Network/privateDnsZones/delete
Microsoft.Network/privateDnsZones/virtualNetworkLinks/delete
Microsoft.Network/virtualNetworks/delete
NOTE
The following permissions are not required to delete a private OpenShift Container
Platform cluster on Azure.
Example 7.22. Required permissions for deleting a resource group
Microsoft.Resources/subscriptions/resourcegroups/delete
898
Example 7.23. Required permissions for deleting storage resources
Microsoft.Storage/storageAccounts/delete
NOTE
To install OpenShift Container Platform on Azure, you must scope the permissions to
your subscription. Later, you can re-scope these permissions to the installer created
resource group. If the public DNS zone is present in a different resource group, then the
network DNS zone related permissions must always be applied to your subscription. By
default, the OpenShift Container Platform installation program assigns the Azure identity
the Contributor role.
You can scope all the permissions to your subscription when deleting an OpenShift
7.2.5.3. Using Azure managed identities
The installation program requires an Azure identity to complete the installation. You can use either a
system-assigned or user-assigned managed identity.
If you are unable to use a managed identity, you can use a service principal.
Procedure
1. If you are using a system-assigned managed identity, enable it on the virtual machine that you
will run the installation program from.
2. If you are using a user-assigned managed identity:
a. Assign it to the virtual machine that you will run the installation program from.
b. Record its client ID. You require this value when installing the cluster.
For more information about viewing the details of a user-assigned managed identity, see
the Microsoft Azure documentation for listing user-assigned managed identities .
3. Verify that the required permissions are assigned to the managed identity.
7.2.5.4. Creating a service principal
The installation program requires an Azure identity to complete the installation. You can use a service
principal.
If you are unable to use a service principal, you can use a managed identity.
Prerequisites
You have an Azure subscription ID.
If you are not going to assign the Contributor and User Administrator Access roles to the
899
service principal, you have created a custom role with the required Azure permissions.
Procedure
1. Create the service principal for your account by running the following command:
$ az ad sp create-for-rbac --role <role_name> \ 1

--name <service_principal> \ 2
--scopes /subscriptions/<subscription_id> 3
1 Defines the role name. You can use the Contributor role, or you can specify a custom role
which contains the necessary permissions.
2 Defines the service principal name.
3 Specifies the subscription ID.
Example output
Creating 'Contributor' role assignment under scope '/subscriptions/<subscription_id>'

The output includes credentials that you must protect. Be sure that you do not
include these credentials in your code or check the credentials into your source
control. For more information, see https://aka.ms/azadsp-cli
{
"appId": "axxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
"displayName": <service_principal>",
"password": "00000000-0000-0000-0000-000000000000",
"tenantId": "8xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
}
2. Record the values of the appId and password parameters from the output. You require these
values when installing the cluster.
3. If you applied the Contributor role to your service principal, assign the User Administrator
Access role by running the following command:
$ az role assignment create --role "User Access Administrator" \

--assignee-object-id $(az ad sp show --id <appId> --query id -o tsv) 1
--scope /subscriptions/<subscription_id> 2
1 Specify the appId parameter value for your service principal.
7.2.6. Supported Azure Marketplace regions
Installing a cluster using the Azure Marketplace image is available to customers who purchase the offer
900
Installing a cluster using the Azure Marketplace image is available to customers who purchase the offer
in North America and EMEA.
While the offer must be purchased in North America or EMEA, you can deploy the cluster to any of the
Azure public partitions that OpenShift Container Platform supports.
NOTE
Deploying a cluster using the Azure Marketplace image is not supported for the Azure
Government regions.
7.2.7. Supported Azure regions

The installation program dynamically generates the list of available Microsoft Azure regions based on
your subscription.
Supported Azure public regions
australiacentral (Australia Central)
australiaeast (Australia East)
australiasoutheast (Australia South East)
brazilsouth (Brazil South)
canadacentral (Canada Central)
canadaeast (Canada East)
centralindia (Central India)
centralus (Central US)
eastasia (East Asia)
eastus (East US)
eastus2 (East US 2)
francecentral (France Central)
germanywestcentral (Germany West Central)
israelcentral (Israel Central)
italynorth (Italy North)
japaneast (Japan East)
japanwest (Japan West)
koreacentral (Korea Central)
koreasouth (Korea South)
northcentralus (North Central US)
901
northeurope (North Europe)
norwayeast (Norway East)
polandcentral (Poland Central)
qatarcentral (Qatar Central)
southafricanorth (South Africa North)
southcentralus (South Central US)
southeastasia (Southeast Asia)
southindia (South India)
swedencentral (Sweden Central)
switzerlandnorth (Switzerland North)
uaenorth (UAE North)
uksouth (UK South)
ukwest (UK West)
westcentralus (West Central US)
westeurope (West Europe)
westindia (West India)
westus (West US)
westus2 (West US 2)
westus3 (West US 3)
Supported Azure Government regions

Support for the following Microsoft Azure Government (MAG) regions was added in OpenShift
Container Platform version 4.6:
usgovtexas (US Gov Texas)
usgovvirginia (US Gov Virginia)
You can reference all available MAG regions in the Azure documentation. Other provided MAG regions
are expected to work with OpenShift Container Platform, but have not been tested.
7.2.8. Next steps

Install an OpenShift Container Platform cluster on Azure. You can install a customized cluster or
quickly install a cluster with default options.
7.3. ENABLING USER-MANAGED ENCRYPTION FOR AZURE
In OpenShift Container Platform version 4.15, you can install a cluster with a user-managed encryption
902
In OpenShift Container Platform version 4.15, you can install a cluster with a user-managed encryption
key in Azure. To enable this feature, you can prepare an Azure DiskEncryptionSet before installation,
modify the install-config.yaml file, and then complete the installation.
7.3.1. Preparing an Azure Disk Encryption Set

The OpenShift Container Platform installer can use an existing Disk Encryption Set with a user-
managed key. To enable this feature, you can create a Disk Encryption Set in Azure and provide the key
to the installer.
Procedure
1. Set the following environment variables for the Azure resource group by running the following
command:
$ export RESOURCEGROUP="<resource_group>" \ 1
LOCATION="<location>" 2
1 Specifies the name of the Azure resource group where you will create the Disk Encryption
Set and encryption key. To avoid losing access to your keys after destroying the cluster,
you should create the Disk Encryption Set in a different resource group than the resource
group where you install the cluster.
2 Specifies the Azure location where you will create the resource group.
2. Set the following environment variables for the Azure Key Vault and Disk Encryption Set by
$ export KEYVAULT_NAME="<keyvault_name>" \ 1
KEYVAULT_KEY_NAME="<keyvault_key_name>" \ 2
DISK_ENCRYPTION_SET_NAME="<disk_encryption_set_name>" 3
1 Specifies the name of the Azure Key Vault you will create.
2 Specifies the name of the encryption key you will create.
3 Specifies the name of the disk encryption set you will create.
3. Set the environment variable for the ID of your Azure Service Principal by running the following
command:
$ export CLUSTER_SP_ID="<service_principal_id>" 1
1 Specifies the ID of the service principal you will use for this installation.
4. Enable host-level encryption in Azure by running the following commands:
$ az feature register --namespace "Microsoft.Compute" --name "EncryptionAtHost"
$ az feature show --namespace Microsoft.Compute --name EncryptionAtHost
903
$ az provider register -n Microsoft.Compute
5. Create an Azure Resource Group to hold the disk encryption set and associated resources by
$ az group create --name $RESOURCEGROUP --location $LOCATION
6. Create an Azure key vault by running the following command:
$ az keyvault create -n $KEYVAULT_NAME -g $RESOURCEGROUP -l $LOCATION \

--enable-purge-protection true
7. Create an encryption key in the key vault by running the following command:
$ az keyvault key create --vault-name $KEYVAULT_NAME -n $KEYVAULT_KEY_NAME \

--protection software
8. Capture the ID of the key vault by running the following command:
$ KEYVAULT_ID=$(az keyvault show --name $KEYVAULT_NAME --query "[id]" -o tsv)
9. Capture the key URL in the key vault by running the following command:
$ KEYVAULT_KEY_URL=$(az keyvault key show --vault-name $KEYVAULT_NAME --name

\
$KEYVAULT_KEY_NAME --query "[key.kid]" -o tsv)
10. Create a disk encryption set by running the following command:
$ az disk-encryption-set create -n $DISK_ENCRYPTION_SET_NAME -l $LOCATION -g \

$RESOURCEGROUP --source-vault $KEYVAULT_ID --key-url $KEYVAULT_KEY_URL
11. Grant the DiskEncryptionSet resource access to the key vault by running the following
commands:
$ DES_IDENTITY=$(az disk-encryption-set show -n $DISK_ENCRYPTION_SET_NAME -g \

$RESOURCEGROUP --query "[identity.principalId]" -o tsv)
$ az keyvault set-policy -n $KEYVAULT_NAME -g $RESOURCEGROUP --object-id \

$DES_IDENTITY --key-permissions wrapkey unwrapkey get
12. Grant the Azure Service Principal permission to read the DiskEncryptionSet by running the
following commands:
$ DES_RESOURCE_ID=$(az disk-encryption-set show -n

$DISK_ENCRYPTION_SET_NAME -g \
$RESOURCEGROUP --query "[id]" -o tsv)
$ az role assignment create --assignee $CLUSTER_SP_ID --role "<reader_role>" \ 1

--scope $DES_RESOURCE_ID -o jsonc
904
1 Specifies an Azure role with read permissions to the disk encryption set. You can use the
Owner role or a custom role with the necessary permissions.
7.3.2. Next steps

Install a cluster with customizations on installer-provisioned infrastructure
Install a cluster with network customizations on installer-provisioned infrastructure
Install a cluster into an existing VNet on installer-provisioned infrastructure
Install a private cluster on installer-provisioned infrastructure
Install a cluster into an government region on installer-provisioned infrastructure
7.4. INSTALLING A CLUSTER QUICKLY ON AZURE

In OpenShift Container Platform version 4.15, you can install a cluster on Microsoft Azure that uses the
processes.
users.
You configured an Azure account to host the cluster and determined the tested and validated
region to deploy the cluster to.

IMPORTANT
905
IMPORTANT

authentication.
user.
IMPORTANT
NOTE
Procedure
NOTE
ecdsa algorithm.
906
NOTE

task:
Example output
Agent pid 31874
NOTE

Example output
Next steps
program.

for installation.
907
Prerequisites
Procedure
IMPORTANT
IMPORTANT
components.

IMPORTANT
Prerequisites
cluster.
908
You have an Azure subscription ID and tenant ID.
You have the application ID and password of a service principal.
Procedure
1. Optional: If you have run the installation program on this computer before, and want to use an
alternative service principal, go to the ~/.azure/ directory and delete the
osServicePrincipal.json configuration file.
Deleting this file prevents the installation program from automatically reusing subscription and
authentication values from a previous installation.
deployment:

--log-level=info 2
Platform version.
NOTE

b. Select azure as the platform to target.

If the installation program cannot locate the osServicePrincipal.json configuration file
from a previous installation, you are prompted for Azure subscription and authentication
values.
c. Specify the following Azure parameter values for your subscription and service principal:
azure subscription id: Enter the subscription ID to use for the cluster.
azure tenant id: Enter the tenant ID.
909
azure service principal client id: Enter its application ID.
azure service principal client secret: Enter its password.
d. Select the region to deploy the cluster to.
e. Select the base domain to deploy the cluster to. The base domain corresponds to the Azure
DNS Zone that you created for your cluster.
f. Enter a descriptive name for your cluster.
IMPORTANT
All Azure resources that are available through public endpoints are subject to
resource name restrictions, and you cannot create resources that use certain
terms. For a list of terms that Azure restricts, see Resolve reserved resource
name errors in the Azure documentation.
g. Paste the pull secret from Red Hat OpenShift Cluster Manager .
If previously not detected, the installation program creates an osServicePrincipal.json configuration

file and stores this file in the ~/.azure/ directory on your computer. This ensures that the installation
program can load the profile when it is creating an OpenShift Container Platform cluster on the target
platform.
Verification
IMPORTANT
Example output
...
IMPORTANT
910
IMPORTANT

IMPORTANT

Procedure
Portal.
$ tar xvf <file>

$ echo $PATH
Verification
$ oc <command>
911

Procedure
Portal.

C:\> path
Verification
C:\> oc <command>

Procedure
Portal.
NOTE

$ echo $PATH
Verification
912
$ oc <command>

Prerequisites
Procedure
$ oc whoami
Example output
system:admin

913
7.4.9. Next steps

7.5. INSTALLING A CLUSTER ON AZURE WITH CUSTOMIZATIONS

the installation program provisions on Microsoft Azure. To customize the installation, you modify
processes.
users.
If you use customer-managed encryption keys, you prepared your Azure environment for
encryption.

IMPORTANT

914
authentication.
user.
IMPORTANT
NOTE
Procedure
NOTE
ecdsa algorithm.
915
NOTE

task:
Example output
Agent pid 31874
NOTE

Example output
Next steps
program.
7.5.4. Using the Azure Marketplace offering

Using the Azure Marketplace offering lets you deploy an OpenShift Container Platform cluster, which is
billed on pay-per-use basis (hourly, per core) through Azure, while still being supported directly by
Red Hat.
To deploy an OpenShift Container Platform cluster using the Azure Marketplace offering, you must first
obtain the Azure Marketplace image. The installation program uses this image to deploy worker nodes.
When obtaining your image, consider the following:
While the images are the same, the Azure Marketplace publisher is different depending on your
region. If you are located in North America, specify redhat as the publisher. If you are located in
EMEA, specify redhat-limited as the publisher.
The offer includes a rh-ocp-worker SKU and a rh-ocp-worker-gen1 SKU. The rh-ocp-worker
SKU represents a Hyper-V generation version 2 VM image. The default instance types used in
OpenShift Container Platform are version 2 compatible. If you plan to use an instance type that
916
is only version 1 compatible, use the image associated with the rh-ocp-worker-gen1 SKU. The
rh-ocp-worker-gen1 SKU represents a Hyper-V version 1 VM image.
IMPORTANT
Installing images with the Azure marketplace is not supported on clusters with 64-bit
ARM instances.
Prerequisites
You have installed the Azure CLI client (az).
Your Azure account is entitled for the offer and you have logged into this account with the
Azure CLI client.
Procedure
1. Display all of the available OpenShift Container Platform images by running one of the following
commands:
North America:
$ az vm image list --all --offer rh-ocp-worker --publisher redhat -o table
Example output
Offer Publisher Sku Urn Version

------------- -------------- ------------------ ----------------------------------------------------------
---- -----------------
rh-ocp-worker RedHat rh-ocp-worker RedHat:rh-ocp-worker:rh-ocp-
worker:413.92.2023101700 413.92.2023101700
rh-ocp-worker RedHat rh-ocp-worker-gen1 RedHat:rh-ocp-worker:rh-ocp-worker-
gen1:413.92.2023101700 413.92.2023101700
EMEA:
$ az vm image list --all --offer rh-ocp-worker --publisher redhat-limited -o table
Example output
Offer Publisher Sku Urn

Version
------------- -------------- ------------------ ----------------------------------------------------------
---- -----------------
rh-ocp-worker redhat-limited rh-ocp-worker redhat-limited:rh-ocp-worker:rh-ocp-
worker:413.92.2023101700 413.92.2023101700
rh-ocp-worker redhat-limited rh-ocp-worker-gen1 redhat-limited:rh-ocp-worker:rh-ocp-
worker-gen1:413.92.2023101700 413.92.2023101700
NOTE
917
NOTE
Regardless of the version of OpenShift Container Platform that you install, the
correct version of the Azure Marketplace image to use is 4.13. If required, your
VMs are automatically upgraded as part of the installation process.
2. Inspect the image for your offer by running one of the following commands:
North America:
$ az vm image show --urn redhat:rh-ocp-worker:rh-ocp-worker:<version>
EMEA:
$ az vm image show --urn redhat-limited:rh-ocp-worker:rh-ocp-worker:<version>
3. Review the terms of the offer by running one of the following commands:
North America:
$ az vm image terms show --urn redhat:rh-ocp-worker:rh-ocp-worker:<version>
EMEA:
$ az vm image terms show --urn redhat-limited:rh-ocp-worker:rh-ocp-worker:<version>
4. Accept the terms of the offering by running one of the following commands:
North America:
$ az vm image terms accept --urn redhat:rh-ocp-worker:rh-ocp-worker:<version>
EMEA:
$ az vm image terms accept --urn redhat-limited:rh-ocp-worker:rh-ocp-worker:<version>
5. Record the image details of your offer. You must update the compute section in the install-
config.yaml file with values for publisher, offer, sku, and version before deploying the cluster.
Sample install-config.yaml file with the Azure Marketplace worker nodes
apiVersion: v1
compute:
name: worker
platform:
azure:
type: Standard_D4s_v5
osImage:
publisher: redhat
offer: rh-ocp-worker
918
sku: rh-ocp-worker
version: 413.92.2023101700
replicas: 3

for installation.
Prerequisites
Procedure
IMPORTANT
IMPORTANT
components.

You can customize the OpenShift Container Platform cluster you install on Microsoft Azure.
Prerequisites
919
cluster.
If you are installing the cluster using a service principal, you have its application ID and password.
If you are installing the cluster using a system-assigned managed identity, you have enabled it
on the virtual machine that you will run the installation program from.
If you are installing the cluster using a user-assigned managed identity, you have met these
prerequisites:
You have its client ID.
You have assigned it to the virtual machine that you will run the installation program from.
Procedure
alternative service principal or managed identity, go to the ~/.azure/ directory and delete the
command:
NOTE

920
ii. Select azure as the platform to target.

values.
iii. Enter the following Azure parameter values for your subscription:
iv. Depending on the Azure identity you are using to deploy the cluster, do one of the
following when prompted for the azure service principal client id:
If you are using a service principal, enter its application ID.
If you are using a system-assigned managed identity, leave this value blank.
If you are using a user-assigned managed identity, specify its client ID.
v. Depending on the Azure identity you are using to deploy the cluster, do one of the
following when prompted for the azure service principal client secret:
If you are using a service principal, enter its password.
If you are using a user-assigned managed identity, leave this value blank.
vi. Select the region to deploy the cluster to.
vii. Select the base domain to deploy the cluster to. The base domain corresponds to the
Azure DNS Zone that you created for your cluster.
viii. Enter a descriptive name for your cluster.
IMPORTANT
All Azure resources that are available through public endpoints are
subject to resource name restrictions, and you cannot create resources
that use certain terms. For a list of terms that Azure restricts, see
Resolve reserved resource name errors in the Azure documentation.
NOTE

more information, see "Installing a three-node cluster on Azure".
IMPORTANT
921
IMPORTANT


platform.
Installation configuration parameters for Azure

System Per Second
(IOPS)[2]

8.6 and later
[3]
IMPORTANT
You are required to use Azure virtual machines that have the premiumIO parameter set
to true.
922
Optimizing storage
7.5.6.2. Tested instance types for Azure
The following Microsoft Azure instance types have been tested with OpenShift Container Platform.
c4.*
c5.*
c5a.*
i3.*
m4.*
m5.*
m5a.*
m6a.*
m6i.*
r4.*
r5.*
r5a.*
r6i.*
t3.*
t3a.*
7.5.6.3. Tested instance types for Azure on 64-bit ARM infrastructures
The following Microsoft Azure ARM64 instance types have been tested with OpenShift Container
Platform.
c6g.*
m6g.*
923
7.5.6.4. Enabling trusted launch for Azure VMs
You can enable two trusted launch features when installing your cluster on Azure: secure boot and
virtualized Trusted Platform Modules.
See the Azure documentation about virtual machine sizes to learn what sizes of virtual machines support
these features.
IMPORTANT
Trusted launch is a Technology Preview feature only. Technology Preview features are
not supported with Red Hat production service level agreements (SLAs) and might not
be functionally complete. Red Hat does not recommend using them in production. These
features provide early access to upcoming product features, enabling customers to test
functionality and provide feedback during the development process.
Prerequisites
You have created an install-config.yaml file.
Procedure
Use a text editor to edit the install-config.yaml file prior to deploying your cluster and add the
following stanza:
controlPlane: 1
platform:
azure:
settings:
securityType: TrustedLaunch 2
trustedLaunch:
uefiSettings:
secureBoot: Enabled 3
virtualizedTrustedPlatformModule: Enabled 4
1 Specify controlPlane.platform.azure or compute.platform.azure to enable trusted

launch on only control plane or compute nodes respectively. Specify
platform.azure.defaultMachinePlatform to enable trusted launch on all nodes.
2 Enable trusted launch features.
3 Enable secure boot. For more information, see the Azure documentation about secure
boot.
4 Enable the virtualized Trusted Platform Module. For more information, see the Azure
documentation about virtualized Trusted Platform Modules.
7.5.6.5. Enabling confidential VMs
You can enable confidential VMs when installing your cluster. You can enable confidential VMs for
compute nodes, control plane nodes, or all nodes.
924
IMPORTANT
Using confidential VMs is a Technology Preview feature only. Technology Preview

features are not supported with Red Hat production service level agreements (SLAs) and
might not be functionally complete. Red Hat does not recommend using them in
production. These features provide early access to upcoming product features, enabling
customers to test functionality and provide feedback during the development process.
You can use confidential VMs with the following VM sizes:
DCasv5-series
DCadsv5-series
ECasv5-series
ECadsv5-series
IMPORTANT
Confidential VMs are currently not supported on 64-bit ARM architectures.
Prerequisites
Procedure
following stanza:
controlPlane: 1
platform:
azure:
settings:
securityType: ConfidentialVM 2
confidentialVM:
uefiSettings:
osDisk:
securityProfile:
securityEncryptionType: VMGuestStateOnly 5
1 Specify controlPlane.platform.azure or compute.platform.azure to deploy confidential

VMs on only control plane or compute nodes respectively. Specify
platform.azure.defaultMachinePlatform to deploy confidential VMs on all nodes.
2 Enable confidential VMs.
925
5 Specify VMGuestStateOnly to encrypt the VM guest state.
7.5.6.6. Sample customized install-config.yaml file for Azure
You can customize the install-config.yaml file to specify more details about your OpenShift Container
Platform cluster’s platform or modify the values of the required parameters.
IMPORTANT
apiVersion: v1
controlPlane: 2
hyperthreading: Enabled 3 4
name: master
platform:
azure:
encryptionAtHost: true
ultraSSDCapability: Enabled
osDisk:
diskSizeGB: 1024 5
diskType: Premium_LRS
diskEncryptionSet:
resourceGroup: disk_encryption_set_resource_group
name: disk_encryption_set_name
subscriptionId: secondary_subscription_id
osImage:
publisher: example_publisher_name
offer: example_image_offer
sku: example_offer_sku
version: example_image_version
replicas: 3
compute: 6
name: worker
platform:
azure:
osDisk:
diskSizeGB: 512 8
diskType: Standard_LRS
diskEncryptionSet:
926
osImage:
zones: 9
- "1"
- "2"
- "3"
replicas: 5
metadata:
networking:
clusterNetwork:
- cidr: 10.128.0.0/14
hostPrefix: 23
machineNetwork:
- cidr: 10.0.0.0/16
serviceNetwork:
- 172.30.0.0/16
platform:
azure:
defaultMachinePlatform:
osImage: 12
baseDomainResourceGroupName: resource_group 13
region: centralus 14
resourceGroupName: existing_resource_group 15
outboundType: Loadbalancer
cloudName: AzurePublicCloud
fips: false 17
2 6 If you do not provide these parameters and values, the installation program provides the default
value.
3 7 The controlPlane section is a single mapping, but the compute section is a sequence of
mappings. To meet the requirements of the different data structures, the first line of the compute
section must begin with a hyphen, -, and the first line of the controlPlane section must not. Only
one control plane pool is used.
4 Whether to enable or disable simultaneous multithreading, or hyperthreading. By default,

IMPORTANT
927
IMPORTANT

accounts for the dramatically decreased machine performance. Use larger virtual
machine types, such as Standard_D8s_v3, for your machines if you disable
5 8 You can specify the size of the disk to use in GB. Minimum recommendation for control plane
nodes is 1024 GB.
9 Specify a list of zones to deploy your machines to. For high availability, specify at least two zones.
value.
12 Optional: A custom Red Hat Enterprise Linux CoreOS (RHCOS) image that should be used to boot
control plane and compute machines. The publisher, offer, sku, and version parameters under
platform.azure.defaultMachinePlatform.osImage apply to both control plane and compute
machines. If the parameters under controlPlane.platform.azure.osImage or
compute.platform.azure.osImage are set, they override the
platform.azure.defaultMachinePlatform.osImage parameters.
13 Specify the name of the resource group that contains the DNS zone for your base domain.
15 Specify the name of an already existing resource group to install your cluster to. If undefined, a new
resource group is created for the cluster.
IMPORTANT
NOTE
process uses.
928
Prerequisites
NOTE
(169.254.169.254).
Procedure
apiVersion: v1
proxy:
must be http.

destinations.
929
NOTE
NOTE
NOTE
created.
For more details about Accelerated Networking, see Accelerated Networking for Microsoft
Azure VMs.
7.5.7. Configuring user-defined tags for Azure

In OpenShift Container Platform, you can use the tags for grouping resources and for managing
resource access and cost. You can define the tags on the Azure resources in the install-config.yaml file
only during OpenShift Container Platform cluster creation. You cannot modify the user-defined tags
after cluster creation.
Support for user-defined tags is available only for the resources created in the Azure Public Cloud.
User-defined tags are not supported for the OpenShift Container Platform clusters upgraded to
OpenShift Container Platform 4.15.
User-defined and OpenShift Container Platform specific tags are applied only to the resources created
by the OpenShift Container Platform installer and its core operators such as Machine api provider azure
Operator, Cluster Ingress Operator, Cluster Image Registry Operator.
By default, OpenShift Container Platform installer attaches the OpenShift Container Platform tags to
the Azure resources. These OpenShift Container Platform tags are not accessible for the users.
930
You can use the .platform.azure.userTags field in the install-config.yaml file to define the list of user-
defined tags as shown in the following install-config.yaml file.
Sample install-config.yaml file
additionalTrustBundlePolicy: Proxyonly 1
apiVersion: v1
baseDomain: catchall.azure.devcluster.openshift.com 2
compute: 3
name: worker
platform: {}
replicas: 3
controlPlane: 5
architecture: amd64
name: master
platform: {}
replicas: 3
metadata:
name: user 7
networking:
clusterNetwork:
- cidr: 10.128.0.0/14
hostPrefix: 23
machineNetwork:
- cidr: 10.0.0.0/16
serviceNetwork:
- 172.30.0.0/16
platform:
azure:
baseDomainResourceGroupName: os4-common 9
cloudName: AzurePublicCloud 10
region: southindia 11
userTags: 12
createdBy: user
environment: dev
1 Defines the trust bundle policy.
2 Required. The baseDomain parameter specifies the base domain of your cloud provider. The
installation program prompts you for this value.
3 The configuration for the machines that comprise compute. The compute section is a sequence of
section must begin with a hyphen, -. If you do not provide these parameters and values, the
installation program provides the default value.
4 To enable or disable simultaneous multithreading, or hyperthreading. By default, simultaneous

multithreading is enabled to increase the performance of your machines' cores. You can disable it
by setting the parameter value to Disabled. If you disable simultaneous multithreading in some
931
cluster machines, you must disable it in all cluster machines.
5 The configuration for the machines that comprise the control plane. The controlPlane section is a
single mapping. The first line of the controlPlane section must not begin with a hyphen, -. You can
use only one control plane pool. If you do not provide these parameters and values, the installation
program provides the default value.
6 To enable or disable simultaneous multithreading, or hyperthreading. By default, simultaneous

multithreading is enabled to increase the performance of your machines' cores. You can disable it
cluster machines, you must disable it in all cluster machines.
7 The installation program prompts you for this value.
value.
9 Specifies the resource group for the base domain of the Azure DNS zone.
10 Specifies the name of the Azure cloud environment. You can use the cloudName field to configure
the Azure SDK with the Azure API endpoints. If you do not provide value, the default value is Azure
Public Cloud.
11 Required. Specifies the name of the Azure region that hosts your cluster. The installation program
prompts you for this value.
12 Defines the additional keys and values that the installation program adds as tags to all Azure
resources that it creates.
The user-defined tags have the following limitations:
A tag key can have a maximum of 128 characters.
A tag key must begin with a letter, end with a letter, number or underscore, and can contain only
letters, numbers, underscores, periods, and hyphens.
Tag keys are case-insensitive.
Tag keys cannot be name. It cannot have prefixes such as kubernetes.io, openshift.io,
microsoft, azure, and windows.
A tag value can have a maximum of 256 characters.
You can configure a maximum of 10 tags for resource group and resources.
For more information about Azure tags, see Azure user-defined tags
7.5.8. Querying user-defined tags for Azure

After creating the OpenShift Container Platform cluster, you can access the list of defined tags for the
Azure resources. The format of the OpenShift Container Platform tags is kubernetes.io_cluster.
<cluster_id>:owned. The cluster_id parameter is the value of .status.infrastructureName present in
config.openshift.io/Infrastructure.
Query the tags defined for Azure resources by running the following command:
932
$ oc get infrastructures.config.openshift.io cluster -o=jsonpath-as-

json='{.status.platformStatus.azure.resourceTags}'
Example output
[
[
{
"key": "createdBy",
"value": "user"
},
{
"key": "environment",
"value": "dev"
}
]
]

IMPORTANT

Procedure
Portal.
$ tar xvf <file>

$ echo $PATH
Verification
933
$ oc <command>

Procedure
Portal.

C:\> path
Verification
C:\> oc <command>

Procedure
Portal.
NOTE

$ echo $PATH
Verification
934
Verification
$ oc <command>

project
components, follow the procedures in Configuring an Azure cluster to use short-term
credentials.
namespace.
Procedure
apiVersion: v1
# ...
command:
935
--included \ 1
metadata:
...
spec:
providerSpec:
kind: AzureProviderSpec
roleBindings:
- role: Contributor
...
metadata:
...
spec:
providerSpec:
roleBindings:
- role: Contributor
...
secretRef:
936
...
apiVersion: v1
kind: Secret
metadata:
data:
azure_subscription_id: <base64_encoded_azure_subscription_id>
azure_client_id: <base64_encoded_azure_client_id>
azure_client_secret: <base64_encoded_azure_client_secret>
azure_tenant_id: <base64_encoded_azure_tenant_id>
azure_resource_prefix: <base64_encoded_azure_resource_prefix>
azure_resourcegroup: <base64_encoded_azure_resourcegroup>
azure_region: <base64_encoded_azure_region>
IMPORTANT
7.5.10.2. Configuring an Azure cluster to use short-term credentials
To install a cluster that uses Azure AD Workload Identity, you must configure the Cloud Credential
Operator utility and create the required Azure resources for your cluster.
NOTE
Prerequisites
You have created a global Microsoft Azure account for the ccoctl utility to use with the
following permissions:
Example 7.26. Required Azure permissions
Microsoft.Resources/subscriptions/resourceGroups/write
Microsoft.Resources/subscriptions/resourceGroups/delete
937
Microsoft.Authorization/roleDefinitions/read
Microsoft.Authorization/roleDefinitions/write
Microsoft.Authorization/roleDefinitions/delete
Microsoft.Storage/storageAccounts/listkeys/action
Microsoft.Storage/storageAccounts/blobServices/containers/delete
Microsoft.Storage/storageAccounts/blobServices/containers/read
Microsoft.ManagedIdentity/userAssignedIdentities/federatedIdentityCredentials/read
Microsoft.ManagedIdentity/userAssignedIdentities/federatedIdentityCredentials/write
Microsoft.ManagedIdentity/userAssignedIdentities/federatedIdentityCredentials/delete
Microsoft.Storage/register/action
Microsoft.ManagedIdentity/register/action
Procedure

NOTE
938
NOTE

$ chmod 775 ccoctl
Verification
$ ccoctl --help
Usage:
ccoctl [command]
Available Commands:
Flags:
7.5.10.2.2. Creating Azure resources with the Cloud Credential Operator utility
You can use the ccoctl azure create-all command to automate the creation of Azure resources.
NOTE
Prerequisites
939
You must have:
Access to your Microsoft Azure account by using the Azure CLI.
Procedure

--included \ 1
NOTE
3. To enable the ccoctl utility to detect your Azure credentials automatically, log in to the Azure
CLI by running the following command:
$ az login
command:
$ ccoctl azure create-all \

--name=<azure_infra_name> \ 1
--output-dir=<ccoctl_output_dir> \ 2
--region=<azure_region> \ 3
--subscription-id=<azure_subscription_id> \ 4
--dnszone-resource-group-name=<azure_dns_zone_resource_group_name> \ 6
--tenant-id=<azure_tenant_id> 7
940
1 Specify the user-defined name for all created Azure resources used for tracking.
3 Specify the Azure region in which cloud resources will be created.
4 Specify the Azure subscription ID to use.
6 Specify the name of the resource group containing the cluster’s base domain Azure DNS
zone.
7 Specify the Azure tenant ID to use.
NOTE
preview parameter.
To see additional optional parameters and explanations of how to use them, run
the azure create-all --help command.
Verification
Example output
azure-ad-pod-identity-webhook-config.yaml
openshift-cloud-controller-manager-azure-cloud-credentials-credentials.yaml
openshift-cluster-api-capz-manager-bootstrap-credentials-credentials.yaml
openshift-cluster-csi-drivers-azure-disk-credentials-credentials.yaml
openshift-cluster-csi-drivers-azure-file-credentials-credentials.yaml
openshift-machine-api-azure-cloud-credentials-credentials.yaml
You can verify that the Azure AD service accounts are created by querying Azure. For more
information, refer to Azure documentation on listing AD service accounts.
941
Prerequisites
utility.
Procedure
apiVersion: v1
# ...
2. If you used the ccoctl utility to create a new Azure resource group instead of using an existing
resource group, modify the resourceGroupName parameter in the install-config.yaml as
shown:
apiVersion: v1
# ...
platform:
azure:
resourceGroupName: <azure_infra_name> 1
# ...
1 This value must match the user-defined name for Azure resources that was specified with
the --name argument of the ccoctl azure create-all command.
command:
942

IMPORTANT
Prerequisites
cluster.
Procedure
deployment:

--log-level=info 2

config.yaml file.
Verification
IMPORTANT
Example output
...
943
IMPORTANT

Prerequisites
Procedure
$ oc whoami
Example output
system:admin
944

7.5.14. Next steps

7.6. INSTALLING A CLUSTER ON AZURE WITH NETWORK

CUSTOMIZATIONS
In OpenShift Container Platform version 4.15, you can install a cluster with a customized network
configuration on infrastructure that the installation program provisions on Microsoft Azure. By
customizing your network configuration, your cluster can coexist with existing IP address allocations in
your environment and integrate with existing MTU and VXLAN configurations.
processes.
users.
encryption.
945
IMPORTANT

authentication.
user.
IMPORTANT
NOTE
Procedure
Specify the path and file name, such as ~/.ssh/id_ed25519, of the new SSH key. If you have
946
NOTE
ecdsa algorithm.
NOTE

task:
Example output
Agent pid 31874
NOTE

Example output
947
Next steps
program.

for installation.
Prerequisites
Procedure
IMPORTANT
IMPORTANT
components.
948
Prerequisites
cluster.
prerequisites:
Procedure
command:
NOTE
949
NOTE


values.
IMPORTANT
IMPORTANT
950
IMPORTANT


platform.

System Per Second
(IOPS)[2]

8.6 and later
[3]
IMPORTANT
to true.
951
Optimizing storage
c4.*
c5.*
c5a.*
i3.*
m4.*
m5.*
m5a.*
m6a.*
m6i.*
r4.*
r5.*
r5a.*
r6i.*
t3.*
t3a.*
Platform.
c6g.*
m6g.*
952
these features.
IMPORTANT
Prerequisites
Procedure
following stanza:
controlPlane: 1
platform:
azure:
settings:
trustedLaunch:
uefiSettings:

boot.
953
IMPORTANT

DCasv5-series
DCadsv5-series
ECasv5-series
ECadsv5-series
IMPORTANT
Prerequisites
Procedure
following stanza:
controlPlane: 1
platform:
azure:
settings:
confidentialVM:
uefiSettings:
osDisk:
securityProfile:

954
IMPORTANT
apiVersion: v1
controlPlane: 2
name: master
platform:
azure:
osDisk:
diskSizeGB: 1024 5
diskEncryptionSet:
osImage:
replicas: 3
compute: 6
name: worker
platform:
azure:
osDisk:
diskSizeGB: 512 8
diskEncryptionSet:
955
osImage:
zones: 9
- "1"
- "2"
- "3"
replicas: 5
metadata:
networking: 11
clusterNetwork:
- cidr: 10.128.0.0/14
hostPrefix: 23
machineNetwork:
- cidr: 10.0.0.0/16
serviceNetwork:
- 172.30.0.0/16
platform:
azure:
osImage: 13
fips: false 18
default value.

IMPORTANT
956
IMPORTANT

nodes is 1024 GB.
value.
IMPORTANT
NOTE
process uses.
957
Prerequisites
NOTE
(169.254.169.254).
Procedure
apiVersion: v1
proxy:
must be http.

destinations.
958
NOTE
NOTE
NOTE
created.

Phase 1
NOTE

NIC resides in.
959
IMPORTANT
cluster.
Phase 2

the cluster.
IMPORTANT

Prerequisites
Procedure
kind: Network
metadata:
name: cluster
spec:
960
kind: Network
metadata:
name: cluster
spec:
defaultNetwork:
ipsecConfig:
mode: Full


clusterNetwork
serviceNetwork
defaultNetwork.type
961
spec:
clusterNetwork:
- cidr: 10.128.0.0/19
hostPrefix: 23
- cidr: 10.128.32.0/19
hostPrefix: 23
spec:
serviceNetwork:
- 172.30.0.0/14

manifest file.
work


NOTE

962

network plugin.

detected MTU.


value to 1400.

configuration.

963

NOTE


t network
infrastructure
overlaps with the
100.64.0.0/16
IPv4 subnet, you
can specify a
different IP
address range for
internal use by
OVN-Kubernetes.
You must ensure
that the IP address
range does not
overlap with any
other subnet used
by your OpenShift
Container
Platform
address range
must be larger
than the maximum
number of nodes
that can be added
to the cluster. For
example, if the
clusterNetwork.
cidr value is
10.128.0.0/14
and the
clusterNetwork.
hostPrefix value
is /23, then the
maximum number
of nodes is 2^(23-
14)=512 .
This field cannot

be changed after
installation.
964

t network
infrastructure
overlaps with the
fd98::/48 IPv6
subnet, you can
specify a different
IP address range
for internal use by
OVN-Kubernetes.
You must ensure
that the IP address
range does not
overlap with any
other subnet used
by your OpenShift
Container
Platform
address range
must be larger
than the maximum
number of nodes
that can be added
to the cluster.
This field cannot

be changed after
installation.

50000000 or 50 MB.
965
libc
host.
udp:<host>:<port>
unix:<file>
null

966


external hosts.

defaultNetwork:
type: OVNKubernetes
mtu: 1400
genevePort: 6081
ipsecConfig:
mode: Full
IMPORTANT

documentation.
NOTE
967

kubeProxyConfig:
proxyArguments:
- 0s

NOTE
Prerequisites

Procedure
where:
cluster.

kind: Network
metadata:
name: cluster
spec:
EOF
968
where:

kind: Network
metadata:
name: cluster
spec:
defaultNetwork:
- cidr: 10.132.0.0/14
hostPrefix: 23
NOTE

port.

NOTE
Azure VMs.
969

IMPORTANT

Procedure
Portal.
$ tar xvf <file>

$ echo $PATH
Verification
$ oc <command>

Procedure
Portal.
970
C:\> path
Verification
C:\> oc <command>

Procedure
Portal.
NOTE

$ echo $PATH
Verification
$ oc <command>

credentials.
971
namespace.
Procedure
apiVersion: v1
# ...
command:

--included \ 1
972
metadata:
...
spec:
providerSpec:
roleBindings:
- role: Contributor
...
metadata:
...
spec:
providerSpec:
roleBindings:
- role: Contributor
...
secretRef:
...
apiVersion: v1
kind: Secret
metadata:
data:
IMPORTANT
973
IMPORTANT
NOTE
Prerequisites
974
Procedure

NOTE

$ chmod 775 ccoctl
Verification
$ ccoctl --help
975
Usage:
ccoctl [command]
Available Commands:
Flags:
NOTE
Prerequisites
You must have:
Procedure

976
--included \ 1
NOTE
$ az login
command:

zone.
NOTE
977
NOTE
preview parameter.
Verification
Example output
Prerequisites
utility.
Procedure
978
apiVersion: v1
# ...
shown:
apiVersion: v1
# ...
platform:
azure:
# ...
command:

IMPORTANT
Prerequisites
979
cluster.
Procedure
deployment:

--log-level=info 2

config.yaml file.
Verification
IMPORTANT
Example output
...
IMPORTANT
980
IMPORTANT

Prerequisites
Procedure
$ oc whoami
Example output
system:admin
981
7.6.15. Next steps

7.7. INSTALLING A CLUSTER ON AZURE INTO AN EXISTING VNET

In OpenShift Container Platform version 4.15, you can install a cluster into an existing Azure Virtual
Network (VNet) on Microsoft Azure. The installation program provisions the rest of the required
processes.
users.
encryption.
7.7.2. About reusing a VNet for your OpenShift Container Platform cluster
In OpenShift Container Platform 4.15, you can deploy a cluster into an existing Azure Virtual Network
(VNet) in Microsoft Azure. If you do, you must also use existing subnets within the VNet and routing
rules.
By deploying OpenShift Container Platform into an existing Azure VNet, you might be able to avoid
service limit constraints in new accounts or more easily abide by the operational constraints that your
company’s guidelines set. This is a good option to use if you cannot obtain the infrastructure creation
permissions that are required to create the VNet.
7.7.2.1. Requirements for using your VNet
When you deploy a cluster by using an existing VNet, you must perform additional network configuration
982
before you install the cluster. In installer-provisioned infrastructure clusters, the installer usually creates
the following components, but it does not create them when you install into an existing VNet:
Subnets
Route tables
VNets
Network Security Groups
NOTE
If you use a custom VNet, you must correctly configure it and its subnets for the installation program
and the cluster to use. The installation program cannot subdivide network ranges for the cluster to use,
set route tables for the subnets, or set VNet options like DHCP, so you must do so before you install the
cluster.
The cluster must be able to access the resource group that contains the existing VNet and subnets.
While all of the resources that the cluster creates are placed in a separate resource group that it
creates, some network resources are used from a separate group. Some cluster Operators must be able
to access resources in both resource groups. For example, the Machine API controller attaches NICS for
the virtual machines that it creates to subnets from the networking resource group.
Your VNet must meet the following characteristics:
The VNet’s CIDR block must contain the Networking.MachineCIDR range, which is the IP
address pool for cluster machines.
The VNet and its subnets must belong to the same resource group, and the subnets must be
configured to use Azure-assigned DHCP IP addresses instead of static IP addresses.
You must provide two subnets within your VNet, one for the control plane machines and one for the
compute machines. Because Azure distributes machines in different availability zones within the region
that you specify, your cluster will have high availability by default.
NOTE
By default, if you specify availability zones in the install-config.yaml file, the installation
program distributes the control plane machines and the compute machines across these
availability zones within a region. To ensure high availability for your cluster, select a
region with at least three availability zones. If your region contains fewer than three
availability zones, the installation program places more than one control plane machine in
the available zones.
data:
All the specified subnets exist.
There are two private subnets, one for the control plane machines and one for the compute
983
machines.
The subnet CIDRs belong to the machine CIDR that you specified. Machines are not provisioned
in availability zones that you do not provide private subnets for. If required, the installation
program creates public load balancers that manage the control plane and worker nodes, and
Azure allocates a public IP address to them.
NOTE
If you destroy a cluster that uses an existing VNet, the VNet is not deleted.
7.7.2.1.1. Network security group requirements
The network security groups for the subnets that host the compute and control plane machines require
specific access to ensure that the cluster communication is correct. You must create rules to allow
access to the required cluster communication ports.
IMPORTANT
The network security group rules must be in place before you install the cluster. If you
attempt to install a cluster without the required access, the installation program cannot
reach the Azure APIs, and installation fails.
Table 7.10. Required ports
Port Description Control plane Compute
80 Allows HTTP traffic x
443 Allows HTTPS traffic x
6443 Allows communication to the control plane machines x
22623 Allows internal communication to the machine config x

server for provisioning machines
1. If you are using Azure Firewall to restrict the internet access, then you can configure Azure
Firewall to allow the Azure APIs. A network security group rule is not needed.
IMPORTANT
Currently, there is no supported way to block or restrict the machine config server
endpoint. The machine config server must be exposed to the network so that newly-
provisioned machines, which have no existing configuration or state, are able to fetch
their configuration. In this model, the root of trust is the certificate signing requests
(CSR) endpoint, which is where the kubelet sends its certificate signing request for
approval to join the cluster. Because of this, machine configs should not be used to
distribute sensitive information, such as secrets and certificates.
To ensure that the machine config server endpoints, ports 22623 and 22624, are secured
in bare metal scenarios, customers must configure proper network policies.
984
Because cluster components do not modify the user-provided network security groups, which the
Kubernetes controllers update, a pseudo-network security group is created for the Kubernetes
controller to modify without impacting the rest of the environment.
About the OpenShift SDN network plugin
Configuring your firewall
resources in your clouds than others. For example, you might be able to create application-specific
items, like instances, storage, and load balancers, but not networking-related components such as
VNets, subnet, or ingress rules.
The Azure credentials that you use when you create your cluster do not need the networking permissions
that are required to make VNets and core networking components within the VNet, such as subnets,
resources that the machines within the cluster require, such as load balancers, security groups, storage
accounts, and nodes.
Because the cluster is unable to modify network security groups in an existing subnet, there is no way to
isolate clusters from each other on the VNet.

IMPORTANT
985
authentication.
user.
IMPORTANT
NOTE
Procedure
NOTE
ecdsa algorithm.
986
NOTE

task:
Example output
Agent pid 31874
NOTE

Example output
Next steps
program.

for installation.
Prerequisites
Procedure
987
IMPORTANT
IMPORTANT
components.

Prerequisites
cluster.
prerequisites:
988
Procedure
command:
NOTE


values.
989
IMPORTANT
IMPORTANT


platform.
990

System Per Second
(IOPS)[2]

8.6 and later
[3]
IMPORTANT
to true.
Optimizing storage
c4.*
c5.*
c5a.*
i3.*
991
m4.*
m5.*
m5a.*
m6a.*
m6i.*
r4.*
r5.*
r5a.*
r6i.*
t3.*
t3a.*
Platform.
c6g.*
m6g.*
these features.
IMPORTANT
Prerequisites
992
Procedure
following stanza:
controlPlane: 1
platform:
azure:
settings:
trustedLaunch:
uefiSettings:

boot.
IMPORTANT

DCasv5-series
DCadsv5-series
ECasv5-series
ECadsv5-series
993
IMPORTANT
Prerequisites
Procedure
following stanza:
controlPlane: 1
platform:
azure:
settings:
confidentialVM:
uefiSettings:
osDisk:
securityProfile:

boot.
IMPORTANT
apiVersion: v1
994
controlPlane: 2
name: master
platform:
azure:
osDisk:
diskSizeGB: 1024 5
diskEncryptionSet:
osImage:
replicas: 3
compute: 6
name: worker
platform:
azure:
osDisk:
diskSizeGB: 512 8
diskEncryptionSet:
osImage:
zones: 9
- "1"
- "2"
- "3"
replicas: 5
metadata:
networking:
clusterNetwork:
- cidr: 10.128.0.0/14
hostPrefix: 23
machineNetwork:
- cidr: 10.0.0.0/16
995
serviceNetwork:
- 172.30.0.0/16
platform:
azure:
osImage: 12
networkResourceGroupName: vnet_resource_group 16
virtualNetwork: vnet 17
controlPlaneSubnet: control_plane_subnet 18
computeSubnet: compute_subnet 19
fips: false 21
value.

IMPORTANT

nodes is 1024 GB.
value.
Optional: A custom Red Hat Enterprise Linux CoreOS (RHCOS) image that should be used to boot
996
16 If you use an existing VNet, specify the name of the resource group that contains it.
17 If you use an existing VNet, specify its name.
18 If you use an existing VNet, specify the name of the subnet to host the control plane machines.
19 If you use an existing VNet, specify the name of the subnet to host the compute machines.
IMPORTANT
NOTE
process uses.
Prerequisites
997
NOTE
(169.254.169.254).
Procedure
apiVersion: v1
proxy:
must be http.

destinations.
NOTE
998
NOTE
NOTE
created.
Azure VMs.

IMPORTANT

Procedure
Portal.
$ tar xvf <file>

999
$ echo $PATH
Verification
$ oc <command>

Procedure
Portal.

C:\> path
Verification
C:\> oc <command>

Procedure
Portal.
NOTE

1000
$ echo $PATH
Verification
$ oc <command>

credentials.
namespace.
Procedure
apiVersion: v1
# ...
command:
1001

--included \ 1
metadata:
...
spec:
providerSpec:
roleBindings:
- role: Contributor
...
metadata:
...
spec:
providerSpec:
roleBindings:
- role: Contributor
1002
...
secretRef:
...
apiVersion: v1
kind: Secret
metadata:
data:
IMPORTANT
NOTE
Prerequisites
1003
Procedure

1004
NOTE

$ chmod 775 ccoctl
Verification
$ ccoctl --help
Usage:
ccoctl [command]
Available Commands:
Flags:
NOTE
Prerequisites
1005
You must have:
Procedure

--included \ 1
NOTE
$ az login
command:

1006
zone.
NOTE
preview parameter.
Verification
Example output
1007
Prerequisites
utility.
Procedure
apiVersion: v1
# ...
shown:
apiVersion: v1
# ...
platform:
azure:
# ...
command:
1008

IMPORTANT
Prerequisites
cluster.
Procedure
deployment:

--log-level=info 2

config.yaml file.
Verification
IMPORTANT
Example output
...
1009
IMPORTANT

7.7.11. Next steps

7.8. INSTALLING A PRIVATE CLUSTER ON AZURE

In OpenShift Container Platform version 4.15, you can install a private cluster into an existing Azure
Virtual Network (VNet) on Microsoft Azure. The installation program provisions the rest of the required
1010
processes.
users.
encryption.

internet.
IMPORTANT
7.8.2.1. Private clusters in Azure
To create a private cluster on Microsoft Azure, you must provide an existing private VNet and subnets to
host the cluster. The installation program must also be able to resolve the DNS records that the cluster
requires. The installation program configures the Ingress Operator and API server for only internal
traffic.
1011
Depending how your network connects to the private VNET, you might need to use a DNS forwarder to
resolve the cluster’s private DNS records. The cluster’s machines use 168.63.129.16 internally for DNS
resolution. For more information, see What is Azure Private DNS? and What is IP address 168.63.129.16?
in the Azure documentation.
The cluster still requires access to internet to access the Azure APIs.
A BaseDomainResourceGroup, since the cluster does not create public records
Public IP addresses
Public DNS records
Public endpoints
The cluster is configured so that the Operators do not create public records for the cluster
and all cluster machines are placed in the private subnets that you specify.
Private clusters on Azure are subject to only the limitations that are associated with the use of an
existing VNet.
7.8.2.2. User-defined outbound routing
In OpenShift Container Platform, you can choose your own outbound routing for a cluster to connect to
the internet. This allows you to skip the creation of public IP addresses and the public load balancer.
You can configure user-defined routing by modifying parameters in the install-config.yaml file before
installing your cluster. A pre-existing VNet is required to use outbound routing when installing a cluster;
the installation program is not responsible for configuring this.
When configuring a cluster to use user-defined routing, the installation program does not create the
following resources:
Outbound rules for access to the internet.
Public IPs for the public load balancer.
Kubernetes Service object to add the cluster machines to the public load balancer for outbound
requests.
You must ensure the following items are available before setting user-defined routing:
Egress to the internet is possible to pull container images, unless using an OpenShift image
registry mirror.
The cluster can access Azure APIs.
Various allowlist endpoints are configured. You can reference these endpoints in the
Configuring your firewall section.
There are several pre-existing networking setups that are supported for internet access using user-
defined routing.
1012
Private cluster with network address translation

You can use Azure VNET network address translation (NAT) to provide outbound internet access for
the subnets in your cluster. You can reference Create a NAT gateway using Azure CLI in the Azure
documentation for configuration instructions.
When using a VNet setup with Azure NAT and user-defined routing configured, you can create a private
cluster with no public endpoints.
Private cluster with Azure Firewall

You can use Azure Firewall to provide outbound routing for the VNet used to install the cluster. You can
learn more about providing user-defined routing with Azure Firewall in the Azure documentation.
When using a VNet setup with Azure Firewall and user-defined routing configured, you can create a
private cluster with no public endpoints.
Private cluster with a proxy configuration

You can use a proxy with user-defined routing to allow egress to the internet. You must ensure that
cluster Operators do not access Azure APIs using a proxy; Operators must have access to Azure APIs
outside of the proxy.
When using the default route table for subnets, with 0.0.0.0/0 populated automatically by Azure, all
Azure API requests are routed over Azure’s internal network even though the IP addresses are public. As
long as the Network Security Group rules allow egress to Azure API endpoints, proxies with user-defined
routing configured allow you to create private clusters with no public endpoints.
Private cluster with no internet access

You can install a private network that restricts all access to the internet, except the Azure API. This is
accomplished by mirroring the release image registry locally. Your cluster must have access to the
following:
An OpenShift image registry mirror that allows for pulling container images
Access to Azure APIs
With these requirements available, you can use user-defined routing to create private clusters with no
public endpoints.
rules.
Subnets
1013
Route tables
VNets
NOTE
cluster.
NOTE
data:
machines.
in availability zones that you do not provide private subnets for.
NOTE
1014
NOTE
IMPORTANT

IMPORTANT
1015

IMPORTANT

authentication.
1016
user.
IMPORTANT
NOTE
Procedure
NOTE
ecdsa algorithm.
NOTE
1017
NOTE

task:
Example output
Agent pid 31874
NOTE

Example output
Next steps
program.

for installation.
Prerequisites
Procedure
1018
IMPORTANT
IMPORTANT
components.

Prerequisites
recovery.
for your cluster.
Procedure
IMPORTANT
1019
IMPORTANT
NOTE
IMPORTANT

System Per Second
(IOPS)[2]

8.6 and later
[3]
1020
IMPORTANT
to true.
Optimizing storage
c4.*
c5.*
c5a.*
i3.*
m4.*
m5.*
m5a.*
m6a.*
m6i.*
r4.*
r5.*
r5a.*
r6i.*
t3.*
t3a.*
1021
Platform.
c6g.*
m6g.*
these features.
IMPORTANT
Prerequisites
Procedure
following stanza:
controlPlane: 1
platform:
azure:
settings:
trustedLaunch:
uefiSettings:

1022
boot.
IMPORTANT

DCasv5-series
DCadsv5-series
ECasv5-series
ECadsv5-series
IMPORTANT
Prerequisites
Procedure
following stanza:
controlPlane: 1
platform:
azure:
settings:
confidentialVM:
1023
uefiSettings:
osDisk:
securityProfile:

boot.
IMPORTANT
apiVersion: v1
controlPlane: 2
name: master
platform:
azure:
osDisk:
diskSizeGB: 1024 5
diskEncryptionSet:
osImage:
1024
replicas: 3
compute: 6
name: worker
platform:
azure:
osDisk:
diskSizeGB: 512 8
diskEncryptionSet:
osImage:
zones: 9
- "1"
- "2"
- "3"
replicas: 5
metadata:
networking:
clusterNetwork:
- cidr: 10.128.0.0/14
hostPrefix: 23
machineNetwork:
- cidr: 10.0.0.0/16
serviceNetwork:
- 172.30.0.0/16
platform:
azure:
osImage: 12
outboundType: UserDefinedRouting 20
1025
fips: false 22
value.

IMPORTANT

nodes is 1024 GB.
value.
1026
20 You can customize your own outbound routing. Configuring user-defined routing prevents
exposing external endpoints in your cluster. User-defined routing for egress requires deploying
IMPORTANT
NOTE
process uses.
Prerequisites
NOTE
1027
NOTE
(169.254.169.254).
Procedure
apiVersion: v1
proxy:
must be http.

destinations.
NOTE
1028
NOTE
NOTE
created.
Azure VMs.

IMPORTANT

Procedure
Portal.
$ tar xvf <file>

1029
$ echo $PATH
Verification
$ oc <command>

Procedure
Portal.

C:\> path
Verification
C:\> oc <command>

Procedure
Portal.
NOTE

1030
$ echo $PATH
Verification
$ oc <command>

credentials.
namespace.
Procedure
apiVersion: v1
# ...
command:
1031

--included \ 1
metadata:
...
spec:
providerSpec:
roleBindings:
- role: Contributor
...
metadata:
...
spec:
providerSpec:
roleBindings:
- role: Contributor
1032
...
secretRef:
...
apiVersion: v1
kind: Secret
metadata:
data:
IMPORTANT
NOTE
Prerequisites
1033
Procedure

1034
NOTE

$ chmod 775 ccoctl
Verification
$ ccoctl --help
Usage:
ccoctl [command]
Available Commands:
Flags:
NOTE
Prerequisites
1035
You must have:
Procedure

--included \ 1
NOTE
$ az login
command:

1036
zone.
NOTE
preview parameter.
Verification
Example output
1037
Prerequisites
utility.
Procedure
apiVersion: v1
# ...
shown:
apiVersion: v1
# ...
platform:
azure:
# ...
command:
1038
7.8.10. Optional: Preparing a private Microsoft Azure cluster for a private image
registry
By installing a private image registry on a private Microsoft Azure cluster, you can create private storage
endpoints. Private storage endpoints disable public facing endpoints to the registry’s storage account,
adding an extra layer of security to your OpenShift Container Platform deployment. Use the following
guide to prepare your private Microsoft Azure cluster for installation with a private image registry.
Prerequisites
You have prepared an install-config.yaml that includes the following information:
The publish field is set to Internal
You have set the permissions for creating a private storage endpoint. For more information, see
"Azure permissions for installer-provisioned infrastructure".
Procedure
command:
This command displays the following messages:
Example output

INFO Manifests created in: <installation_directory>/manifests and
<installation_directory>/openshift
2. Create an image registry configuration object and pass in the networkResourceGroupName,

subnetName, and vnetName provided by Microsoft Azure. For example:
$ touch imageregistry-config.yaml
apiVersion: imageregistry.operator.openshift.io/v1
kind: Config
metadata:
name: cluster
spec:
managementState: "Managed"
replicas: 2
rolloutStrategy: RollingUpdate
storage:
azure:
1039
networkAccess:
internal:
networkResourceGroupName: <vnet_resource_group> 1
subnetName: <subnet_name> 2
vnetName: <vnet_name> 3
type: Internal
1 Optional. If you have an existing VNet and subnet setup, replace <vnet_resource_group>
with the resource group name that contains the existing virtual network (VNet).
2 Optional. If you have an existing VNet and subnet setup, replace <subnet_name> with the
name of the existing compute subnet within the specified resource group.
3 Optional. If you have an existing VNet and subnet setup, replace <vnet_name> with the
name of the existing virtual network (VNet) in the specified resource group.
NOTE
The imageregistry-config.yaml file is consumed during the installation process.

If desired, you must back it up before installation.
3. Move the imageregistry-config.yaml file to the <installation_directory/manifests> folder by

$ mv imageregistry-config.yaml <installation_directory/manifests/>
Next steps
After you have moved the imageregistry-config.yaml file to the

<installation_directory/manifests> folder and set the required permissions, proceed to
"Deploying the cluster".
For the list of permissions needed to create a private storage endpoint, see Required Azure
permissions for installer-provisioned infrastructure.

IMPORTANT
Prerequisites
cluster.
1040
prerequisites:
Procedure
deployment:

--log-level=info 2

config.yaml file.
If the installation program cannot locate the osServicePrincipal.json configuration file from a
previous installation, you are prompted for Azure subscription and authentication values.
3. Enter the following Azure parameter values for your subscription:
4. Depending on the Azure identity you are using to deploy the cluster, do one of the following
when prompted for the azure service principal client id:
when prompted for the azure service principal client secret:
1041
If you are using a user-assigned managed identity,leave this value blank.

platform.
Verification
IMPORTANT
Example output
...
IMPORTANT

1042
Prerequisites
Procedure
$ oc whoami
Example output
system:admin

7.8.14. Next steps

7.9. INSTALLING A CLUSTER ON AZURE INTO A GOVERNMENT

1043
7.9. INSTALLING A CLUSTER ON AZURE INTO A GOVERNMENT

REGION
In OpenShift Container Platform version 4.15, you can install a cluster on Microsoft Azure into a
government region. To configure the government region, you modify parameters in the install-
config.yaml file before you install the cluster.
processes.
users.
government region to deploy the cluster to.
encryption.
7.9.2. Azure government regions

OpenShift Container Platform supports deploying a cluster to Microsoft Azure Government (MAG)
regions. MAG is specifically designed for US government agencies at the federal, state, and local level,
as well as contractors, educational institutions, and other US customers that must run sensitive
workloads on Azure. MAG is composed of government-only data center regions, all granted an Impact
Level 5 Provisional Authorization.
Installing to a MAG region requires manually configuring the Azure Government dedicated cloud
instance and region in the install-config.yaml file. You must also update your service principal to
reference the appropriate government environment.
NOTE
The Azure government region cannot be selected using the guided terminal prompts
from the installation program. You must define the region manually in the install-
config.yaml file. Remember to also set the dedicated cloud instance, like
AzureUSGovernmentCloud, based on the region specified.

internet.
1044
IMPORTANT
7.9.3.1. Private clusters in Azure
To create a private cluster on Microsoft Azure, you must provide an existing private VNet and subnets to
host the cluster. The installation program must also be able to resolve the DNS records that the cluster
traffic.
Depending how your network connects to the private VNET, you might need to use a DNS forwarder to
resolve the cluster’s private DNS records. The cluster’s machines use 168.63.129.16 internally for DNS
resolution. For more information, see What is Azure Private DNS? and What is IP address 168.63.129.16?
in the Azure documentation.
The cluster still requires access to internet to access the Azure APIs.
A BaseDomainResourceGroup, since the cluster does not create public records
Public IP addresses
Public DNS records
Public endpoints
The cluster is configured so that the Operators do not create public records for the cluster
and all cluster machines are placed in the private subnets that you specify.
1045
existing VNet.
requests.
registry mirror.
defined routing.
rules.
Subnets
Route tables
1046
VNets
NOTE
cluster.
NOTE
data:
machines.
NOTE
1047
NOTE
IMPORTANT

IMPORTANT
1048

IMPORTANT

authentication.
1049
user.
IMPORTANT
NOTE
Procedure
NOTE
ecdsa algorithm.
NOTE
1050
NOTE

task:
Example output
Agent pid 31874
NOTE

Example output
Next steps
program.

for installation.
Prerequisites
Procedure
1051
IMPORTANT
IMPORTANT
components.

Prerequisites
recovery.
for your cluster.
Procedure
IMPORTANT
1052
IMPORTANT
NOTE
IMPORTANT

System Per Second
(IOPS)[2]

8.6 and later
[3]
1053
IMPORTANT
to true.
Optimizing storage
c4.*
c5.*
c5a.*
i3.*
m4.*
m5.*
m5a.*
m6a.*
m6i.*
r4.*
r5.*
r5a.*
r6i.*
t3.*
t3a.*
1054
these features.
IMPORTANT
Prerequisites
Procedure
following stanza:
controlPlane: 1
platform:
azure:
settings:
trustedLaunch:
uefiSettings:

boot.
IMPORTANT
1055
IMPORTANT

DCasv5-series
DCadsv5-series
ECasv5-series
ECadsv5-series
IMPORTANT
Prerequisites
Procedure
following stanza:
controlPlane: 1
platform:
azure:
settings:
confidentialVM:
uefiSettings:
osDisk:
securityProfile:

boot.
1056
IMPORTANT
apiVersion: v1
controlPlane: 2
name: master
platform:
azure:
osDisk:
diskSizeGB: 1024 5
diskEncryptionSet:
osImage:
replicas: 3
compute: 6
name: worker
platform:
azure:
osDisk:
diskSizeGB: 512 8
diskEncryptionSet:
1057
osImage:
zones: 9
- "1"
- "2"
- "3"
replicas: 5
metadata:
networking:
clusterNetwork:
- cidr: 10.128.0.0/14
hostPrefix: 23
machineNetwork:
- cidr: 10.0.0.0/16
serviceNetwork:
- 172.30.0.0/16
platform:
azure:
osImage: 12
region: usgovvirginia
cloudName: AzureUSGovernmentCloud 20
fips: false 22
1 10 21 Required.
value.
1058

IMPORTANT

nodes is 1024 GB.
value.
19 You can customize your own outbound routing. Configuring user-defined routing prevents
exposing external endpoints in your cluster. User-defined routing for egress requires deploying
your cluster to an existing VNet.
20 Specify the name of the Azure cloud environment to deploy your cluster to. Set
AzureUSGovernmentCloud to deploy to a Microsoft Azure Government (MAG) region. The
default value is AzurePublicCloud.
IMPORTANT
1059
IMPORTANT
NOTE
process uses.
Prerequisites
NOTE
(169.254.169.254).
Procedure
apiVersion: v1
1060
proxy:
must be http.

destinations.
NOTE
NOTE
NOTE
1061
NOTE
created.
Azure VMs.

IMPORTANT
Prerequisites
cluster.
prerequisites:
Procedure
deployment:

--log-level=info 2

1062

config.yaml file.
If the installation program cannot locate the osServicePrincipal.json configuration file from a
previous installation, you are prompted for Azure subscription and authentication values.
3. Enter the following Azure parameter values for your subscription:
when prompted for the azure service principal client id:
when prompted for the azure service principal client secret:
If you are using a user-assigned managed identity,leave this value blank.

platform.
Verification
IMPORTANT
Example output
...
1063
IMPORTANT

IMPORTANT

Procedure
Portal.
$ tar xvf <file>

$ echo $PATH
1064
Verification
$ oc <command>

Procedure
Portal.

C:\> path
Verification
C:\> oc <command>

Procedure
Portal.
NOTE

1065
$ echo $PATH
Verification
$ oc <command>

Prerequisites
Procedure
$ oc whoami
Example output
system:admin

1066
7.9.13. Next steps

7.10. INSTALLING A CLUSTER ON AZURE IN A RESTRICTED NETWORK

In OpenShift Container Platform, you can install a cluster on Microsoft Azure by using infrastructure that
you provide.
Several Azure Resource Manager (ARM) templates are provided to assist in completing these steps or
to help model your own.
IMPORTANT

ARM templates are provided to assist in completing these steps or to help model your
own. You are also free to create the required resources through other methods.
Prerequisites
processes.
users.
IMPORTANT
Because the installation media is on the mirror host, you must use that computer
1067
namespace, you have manually created long-term credentials.
encryption.

your restrictions.
IMPORTANT

IMPORTANT
1068
IMPORTANT
7.10.2. Configuring your Azure project

Before you can install OpenShift Container Platform, you must configure an Azure project to host it.
IMPORTANT
documentation.
7.10.2.1. Azure account limits
IMPORTANT

nt components limit
required by
default
1069

nt components limit
required by
default


instances:

after installation
Because the bootstrap and control plane machines

use Standard_D8s_v3 virtual machines, which use
8 vCPUs, and the compute machines use
Standard_D4s_v3 virtual machines, which use 4
vCPUs, a default cluster requires 44 vCPUs. The
bootstrap node VM, which uses 8 vCPUs, is used



1070

nt components limit
required by
default


olp
lan
e

load
balancers
t


al


installation.

1071

nt components limit
required by
default
spot VMs, your
cluster must have
two spot VM
NOTE
vCPUs for every
Optimizing storage
7.10.2.2. Configuring a public DNS zone in Azure
Procedure
NOTE
parent domain.
You can view Azure’s DNS solution by visiting this example for creating DNS zones .
7.10.2.3. Increasing Azure account limits
1072
NOTE
Procedure
and approving them.
7.10.2.5. Required Azure roles
resources. Before you create the identity, verify that your environment meets the following
requirements:
The Azure account that you use to create the identity is assigned the User Access
Administrator and Contributor roles. These roles are required when:
Creating a service principal or user-assigned managed identity.
Enabling a system-assigned managed identity on a virtual machine.
If you are going to use a service principal to complete the installation, verify that the Azure
account that you use to create the identity is assigned the
microsoft.directory/servicePrincipals/createAsOwner permission in Azure Active Directory.
1073
portal in the Azure documentation.
7.10.2.6. Required Azure permissions for user-provisioned infrastructure
Microsoft Azure.
1074
Microsoft.Compute/virtualMachines/deallocate/action
1075
1076
Example 7.45. Required permissions for creating deployments
Microsoft.Resources/deployments/read
Microsoft.Resources/deployments/write
Microsoft.Resources/deployments/validate/action
Microsoft.Resources/deployments/operationstatuses/read
1077
Microsoft Azure.
1078
NOTE
1079
NOTE
To install OpenShift Container Platform on Azure, you must scope the permissions
related to resource group creation to your subscription. After the resource group is
created, you can scope the rest of the permissions to the created resource group. If the
public DNS zone is present in a different resource group, then the network DNS zone
related permissions must always be applied to your subscription.
Because OpenShift Container Platform and its installation program create Microsoft Azure resources by
using the Azure Resource Manager, you must create a service principal to represent it.
Prerequisites
Install or update the Azure CLI.
Your Azure account has the required roles for the subscription that you use.
If you want to use a custom role, you have created a custom role with the required permissions
listed in the Required Azure permissions for user-provisioned infrastructure section.
Procedure
1. Log in to the Azure CLI:
$ az login
2. If your Azure account uses subscriptions, ensure that you are using the right subscription:
a. View the list of available accounts and record the tenantId value for the subscription you
want to use for your cluster:
Example output
[
{
"id": "9bab1460-96d5-40b3-a78e-17b15e978a80",
"isDefault": true,
"name": "Subscription Name",
"state": "Enabled",
"tenantId": "6057c7e9-b3ae-489d-a54e-de3f6bf6a8ee",
"user": {
"type": "user"
}
}
]
b. View your active account details and confirm that the tenantId value matches the
1080
subscription you want to use:
$ az account show
Example output
{
"id": "9bab1460-96d5-40b3-a78e-17b15e978a80",
"isDefault": true,
"state": "Enabled",
"tenantId": "6057c7e9-b3ae-489d-a54e-de3f6bf6a8ee", 1
"user": {
"type": "user"
}
}
1 Ensure that the value of the tenantId parameter is the correct subscription ID.
c. If you are not using the right subscription, change the active subscription:
$ az account set -s <subscription_id> 1
1 Specify the subscription ID.
d. Verify the subscription ID update:
$ az account show
Example output
{
"id": "33212d16-bdf6-45cb-b038-f6565b61edda",
"isDefault": true,
"state": "Enabled",
"tenantId": "8049c7e9-c3de-762d-a54e-dc3f6be6a7ee",
"user": {
"type": "user"
}
}
3. Record the tenantId and id parameter values from the output. You need these values during the
OpenShift Container Platform installation.
4. Create the service principal for your account:
1081

Example output

{
"appId": "ac461d78-bf4b-4387-ad16-7e32e328aec6",
"password": "00000000-0000-0000-0000-000000000000",
"tenantId": "8049c7e9-c3de-762d-a54e-dc3f6be6a7ee"
}
5. Record the values of the appId and password parameters from the previous output. You need
these values during OpenShift Container Platform installation.

For more information about CCO modes, see About the Cloud Credential Operator.
7.10.2.8. Supported Azure regions
your subscription.
1082
eastus (East US)
eastus2 (East US 2)
uksouth (UK South)
ukwest (UK West)
1083
westus (West US)
westus2 (West US 2)
westus3 (West US 3)


Hosts Description
control plane.
IMPORTANT
machines.
1084

System Per Second
(IOPS)[2]

8.6 and later
[3]
IMPORTANT
to true.
1085
c4.*
c5.*
c5a.*
i3.*
m4.*
m5.*
m5a.*
m6a.*
m6i.*
r4.*
r5.*
r5a.*
r6i.*
t3.*
t3a.*
Platform.
c6g.*
m6g.*

Red Hat.
1086
IMPORTANT
ARM instances.
Prerequisites
Azure CLI client.
Procedure
commands:
North America:
Example output

------------- -------------- ------------------ ----------------------------------------------------------
---- -----------------
worker:413.92.2023101700 413.92.2023101700
gen1:413.92.2023101700 413.92.2023101700
EMEA:
Example output

Version
------------- -------------- ------------------ ----------------------------------------------------------
---- -----------------
1087
worker:413.92.2023101700 413.92.2023101700
worker-gen1:413.92.2023101700 413.92.2023101700
NOTE
North America:
EMEA:
North America:
EMEA:
North America:
EMEA:
5. Record the image details of your offer. If you use the Azure Resource Manager (ARM) template
to deploy your worker nodes:
a. Update storageProfile.imageReference by deleting the id parameter and adding the

offer, publisher, sku, and version parameters by using the values from your offer.
b. Specify a plan for the virtual machines (VMs).
Example 06_workers.json ARM template with an updated

storageProfile.imageReference object and a specified plan
...
"plan" : {
1088
"name": "rh-ocp-worker",
"product": "rh-ocp-worker",
"publisher": "redhat"
},
"dependsOn" : [
"[concat('Microsoft.Network/networkInterfaces/', concat(variables('vmNames')
[copyIndex()], '-nic'))]"
],
"properties" : {
...
"storageProfile": {
"imageReference": {
"offer": "rh-ocp-worker",
"publisher": "redhat",
"sku": "rh-ocp-worker",
"version": "413.92.2023101700"
}
...
}
...
}
for installation.
Prerequisites
Procedure
IMPORTANT
IMPORTANT
1089
components.
authentication.
user.
IMPORTANT
NOTE
Procedure
NOTE
1090
NOTE
ecdsa algorithm.
NOTE

task:
Example output
Agent pid 31874
NOTE

Example output
Next steps
1091
7.10.5. Creating the installation files for Azure

To install OpenShift Container Platform on Microsoft Azure using user-provisioned infrastructure, you
must generate the files that the installation program needs to deploy your cluster and modify them so
that the cluster creates only the machines that it will use. You generate and customize the install-
config.yaml file, Kubernetes manifests, and Ignition config files. You also have the option to first set up
a separate var partition during the preparation phases of installation.
installation.
IMPORTANT
Procedure
1092
Example output


Example output
...
variant: openshift
version: 4.15.0
metadata:
labels:
storage:
disks:
partitions:
- label: var
number: 5
filesystems:
path: /var
format: xfs
1093
NOTE
name.

partition.yaml

Prerequisites
creation.
You have retrieved a Red Hat Enterprise Linux CoreOS (RHCOS) image and uploaded it to an
accessible location.
prerequisites:
1094
Procedure
command:
NOTE


values.
1095
IMPORTANT
ix. Paste the pull secret from Red Hat OpenShift Cluster Manager .

1096
c. Define the network and subnets for the VNet to install the cluster under the platform.azure
field:
virtualNetwork: <vnet> 2
controlPlaneSubnet: <control_plane_subnet> 3
computeSubnet: <compute_subnet> 4
1 Replace <vnet_resource_group> with the resource group name that contains the
existing virtual network (VNet).
2 Replace <vnet> with the existing virtual network name.
3 Replace <control_plane_subnet> with the existing subnet name to deploy the control
plane machines.
4 Replace <compute_subnet> with the existing subnet name to deploy compute

machines.
- mirrors:
- mirrors:
creation.
publish: Internal
IMPORTANT
Azure Firewall does not work seamlessly with Azure Public Load balancers.
Thus, when using Azure Firewall for restricting internet access, the publish
field in install-config.yaml should be set to Internal.
IMPORTANT
1097
IMPORTANT


platform.
Prerequisites
NOTE
(169.254.169.254).
Procedure
apiVersion: v1
proxy:
must be http.
1098

destinations.
NOTE
NOTE
NOTE
created.
7.10.5.4. Exporting common variables for ARM templates
You must export a common set of variables that are used with the provided Azure Resource Manager
(ARM) templates used to assist in completing a user-provided infrastructure install on Microsoft Azure.
NOTE
Specific ARM templates can also require additional exported variables, which are detailed
in their related procedures.
1099
Prerequisites
Obtain the OpenShift Container Platform installation program and the pull secret for your
cluster.
Procedure
1. Export common variables found in the install-config.yaml to be used by the provided ARM
templates:
$ export CLUSTER_NAME=<cluster_name> 1
$ export AZURE_REGION=<azure_region> 2
$ export SSH_KEY=<ssh_key> 3
$ export BASE_DOMAIN=<base_domain> 4
$ export BASE_DOMAIN_RESOURCE_GROUP=<base_domain_resource_group> 5
1 The value of the .metadata.name attribute from the install-config.yaml file.
2 The region to deploy the cluster into, for example centralus. This is the value of the
.platform.azure.region attribute from the install-config.yaml file.
3 The SSH RSA public key file as a string. You must enclose the SSH key in quotes since it
contains spaces. This is the value of the .sshKey attribute from the install-config.yaml
file.
4 The base domain to deploy the cluster to. The base domain corresponds to the public
DNS zone that you created for your cluster. This is the value of the .baseDomain attribute
from the install-config.yaml file.
5 The resource group where the public DNS zone exists. This is the value of the
.platform.azure.baseDomainResourceGroupName attribute from the install-
config.yaml file.
For example:
$ export CLUSTER_NAME=test-cluster
$ export AZURE_REGION=centralus
$ export SSH_KEY="ssh-rsa xxx/xxx/xxx= user@email.com"
$ export BASE_DOMAIN=example.com
$ export BASE_DOMAIN_RESOURCE_GROUP=ocp-cluster
1100
machines.
IMPORTANT
Prerequisites
Procedure
machines.
machine-set.yaml
4. Remove the Kubernetes manifest files that define the worker machines:
1101
IMPORTANT

these machines.

machines:
kind: DNS
metadata:
name: cluster
spec:
privateZone: 1
publicZone: 2
status: {}
7. When configuring Azure on user-provisioned infrastructure, you must export some common
variables defined in the manifest files to use later in the Azure Resource Manager (ARM)
templates:
a. Export the infrastructure ID by using the following command:
$ export INFRA_ID=<infra_id> 1
1 The OpenShift Container Platform cluster has been assigned an identifier (INFRA_ID)
in the form of <cluster_name>-<random_string>. This will be used as the base name
for most resources created using the provided ARM templates. This is the value of the
.status.infrastructureName attribute from the manifests/cluster-infrastructure-02-
1102
config.yml file.
b. Export the resource group by using the following command:
$ export RESOURCE_GROUP=<resource_group> 1
1 All resources created in this Azure deployment exists as part of a resource group. The
resource group name is also based on the INFRA_ID, in the form of <cluster_name>-
<random_string>-rg. This is the value of the
.status.platformStatus.azure.resourceGroupName attribute from the
manifests/cluster-infrastructure-02-config.yml file.
.
├── auth
7.10.6. Creating the Azure resource group

You must create a Microsoft Azure resource group and an identity for that resource group. These are
both used during the installation of your OpenShift Container Platform cluster on Azure.
Prerequisites
Configure an Azure account.
Generate the Ignition config files for your cluster.
Procedure
1. Create the resource group in a supported Azure region:
$ az group create --name ${RESOURCE_GROUP} --location ${AZURE_REGION}
2. Create an Azure identity for the resource group:
1103
$ az identity create -g ${RESOURCE_GROUP} -n ${INFRA_ID}-identity
This is used to grant the required access to Operators in your cluster. For example, this allows
the Ingress Operator to create a public IP and its load balancer. You must assign the Azure
identity to a role.
3. Grant the Contributor role to the Azure identity:
a. Export the following variables required by the Azure role assignment:
$ export PRINCIPAL_ID=àz identity show -g ${RESOURCE_GROUP} -n ${INFRA_ID}-

identity --query principalId --out tsv`
$ export RESOURCE_GROUP_ID=àz group show -g ${RESOURCE_GROUP} --query

id --out tsv`
b. Assign the Contributor role to the identity:
$ az role assignment create --assignee "${PRINCIPAL_ID}" --role 'Contributor' --scope

"${RESOURCE_GROUP_ID}"
NOTE
If you want to assign a custom role with all the required permissions to the
identity, run the following command:
$ az role assignment create --assignee "${PRINCIPAL_ID}" --role

<custom_role> \ 1
--scope "${RESOURCE_GROUP_ID}"
1 Specifies the custom role name.
7.10.7. Uploading the RHCOS cluster image and bootstrap Ignition config file
The Azure client does not support deployments based on files existing locally. You must copy and store
the RHCOS virtual hard disk (VHD) cluster image and bootstrap Ignition config file in a storage
container so they are accessible during deployment.
Prerequisites
Procedure
1. Create an Azure storage account to store the VHD cluster image:
$ az storage account create -g ${RESOURCE_GROUP} --location ${AZURE_REGION} --

name ${CLUSTER_NAME}sa --kind Storage --sku Standard_LRS
1104

WARNING
The Azure storage account name must be between 3 and 24 characters in

length and use numbers and lower-case letters only. If your
CLUSTER_NAME variable does not follow these restrictions, you must
manually define the Azure storage account name. For more information on
Azure storage account name restrictions, see Resolve errors for storage
account names in the Azure documentation.
2. Export the storage account key as an environment variable:
$ export ACCOUNT_KEY=àz storage account keys list -g ${RESOURCE_GROUP} --

account-name ${CLUSTER_NAME}sa --query "[0].value" -o tsv`
3. Export the URL of the RHCOS VHD to an environment variable:
$ export VHD_URL=òpenshift-install coreos print-stream-json | jq -r '.architectures.

<architecture>."rhel-coreos-extensions"."azure-disk".url'`
where:
<architecture>
Specifies the architecture, valid values include x86_64 or aarch64.
IMPORTANT
The RHCOS images might not change with every release of OpenShift
Container Platform. You must specify an image with the highest version that
is less than or equal to the OpenShift Container Platform version that you
install. Use the image version that matches your OpenShift Container
Platform version if it is available.
4. Create the storage container for the VHD:
$ az storage container create --name vhd --account-name ${CLUSTER_NAME}sa --account-

key ${ACCOUNT_KEY}
5. Copy the local VHD to a blob:
$ az storage blob copy start --account-name ${CLUSTER_NAME}sa --account-key

${ACCOUNT_KEY} --destination-blob "rhcos.vhd" --destination-container vhd --source-uri
"${VHD_URL}"
6. Create a blob storage container and upload the generated bootstrap.ign file:
$ az storage container create --name files --account-name ${CLUSTER_NAME}sa --

account-key ${ACCOUNT_KEY}
1105
$ az storage blob upload --account-name ${CLUSTER_NAME}sa --account-key

${ACCOUNT_KEY} -c "files" -f "<installation_directory>/bootstrap.ign" -n "bootstrap.ign"
7.10.8. Example for creating DNS zones

DNS records are required for clusters that use user-provisioned infrastructure. You should choose the
DNS strategy that fits your scenario.
For this example, Azure’s DNS solution is used, so you will create a new public DNS zone for external
(internet) visibility and a private DNS zone for internal cluster resolution.
NOTE
The public DNS zone is not required to exist in the same resource group as the cluster
deployment and might already exist in your organization for the desired base domain. If
that is the case, you can skip creating the public DNS zone; be sure the installation config
you generated earlier reflects that scenario.
Prerequisites
Procedure
1. Create the new public DNS zone in the resource group exported in the
BASE_DOMAIN_RESOURCE_GROUP environment variable:
$ az network dns zone create -g ${BASE_DOMAIN_RESOURCE_GROUP} -n

${CLUSTER_NAME}.${BASE_DOMAIN}
You can skip this step if you are using a public DNS zone that already exists.
2. Create the private DNS zone in the same resource group as the rest of this deployment:
$ az network private-dns zone create -g ${RESOURCE_GROUP} -n

You can learn more about configuring a public DNS zone in Azure by visiting that section.
7.10.9. Creating a VNet in Azure

You must create a virtual network (VNet) in Microsoft Azure for your OpenShift Container Platform
cluster to use. You can customize the VNet to meet your requirements. One way to create the VNet is to
modify the provided Azure Resource Manager (ARM) template.
NOTE
If you do not use the provided ARM template to create your Azure infrastructure, you
must review the provided information and manually create the infrastructure. If your
installation logs.
1106
Prerequisites
Procedure
1. Copy the template from the ARM template for the VNet section of this topic and save it as
01_vnet.json in your cluster’s installation directory. This template describes the VNet that your
cluster requires.
2. Create the deployment by using the az CLI:
$ az deployment group create -g ${RESOURCE_GROUP} \

--template-file "<installation_directory>/01_vnet.json" \
--parameters baseName="${INFRA_ID}" 1
1 The base name to be used in resource names; this is usually the cluster’s infrastructure ID.
3. Link the VNet template to the private DNS zone:
$ az network private-dns link vnet create -g ${RESOURCE_GROUP} -z

${CLUSTER_NAME}.${BASE_DOMAIN} -n ${INFRA_ID}-network-link -v "${INFRA_ID}-vnet"
-e false
7.10.9.1. ARM template for the VNet
You can use the following Azure Resource Manager (ARM) template to deploy the VNet that you need
for your OpenShift Container Platform cluster:
Example 7.58. 01_vnet.json ARM template
{
"$schema" : "https://schema.management.azure.com/schemas/2015-01-
01/deploymentTemplate.json#",
"contentVersion" : "1.0.0.0",
"parameters" : {
"baseName" : {
"type" : "string",
"minLength" : 1,
"metadata" : {
"description" : "Base name to be used in resource names (usually the cluster's Infra ID)"
}
}
},
"variables" : {
"location" : "[resourceGroup().location]",
"virtualNetworkName" : "[concat(parameters('baseName'), '-vnet')]",
"addressPrefix" : "10.0.0.0/16",
"masterSubnetName" : "[concat(parameters('baseName'), '-master-subnet')]",
"masterSubnetPrefix" : "10.0.0.0/24",
"nodeSubnetName" : "[concat(parameters('baseName'), '-worker-subnet')]",
"nodeSubnetPrefix" : "10.0.1.0/24",
1107
"clusterNsgName" : "[concat(parameters('baseName'), '-nsg')]"

},
"resources" : [
{
"apiVersion" : "2018-12-01",
"type" : "Microsoft.Network/virtualNetworks",
"name" : "[variables('virtualNetworkName')]",
"location" : "[variables('location')]",
"dependsOn" : [
"[concat('Microsoft.Network/networkSecurityGroups/', variables('clusterNsgName'))]"
],
"properties" : {
"addressSpace" : {
"addressPrefixes" : [
"[variables('addressPrefix')]"
]
},
"subnets" : [
{
"name" : "[variables('masterSubnetName')]",
"properties" : {
"addressPrefix" : "[variables('masterSubnetPrefix')]",
"serviceEndpoints": [],
"networkSecurityGroup" : {
"id" : "[resourceId('Microsoft.Network/networkSecurityGroups',
variables('clusterNsgName'))]"
}
}
},
{
"name" : "[variables('nodeSubnetName')]",
"properties" : {
"addressPrefix" : "[variables('nodeSubnetPrefix')]",
}
}
}
]
}
},
{
"type" : "Microsoft.Network/networkSecurityGroups",
"name" : "[variables('clusterNsgName')]",
"apiVersion" : "2018-10-01",
"properties" : {
"securityRules" : [
{
"name" : "apiserver_in",
"properties" : {
"protocol" : "Tcp",
"sourcePortRange" : "*",
"destinationPortRange" : "6443",
1108
"sourceAddressPrefix" : "*",
"destinationAddressPrefix" : "*",
"access" : "Allow",
"priority" : 101,
"direction" : "Inbound"
}
}
]
}
}
]
}
7.10.10. Deploying the RHCOS cluster image for the Azure infrastructure
You must use a valid Red Hat Enterprise Linux CoreOS (RHCOS) image for Microsoft Azure for your
OpenShift Container Platform nodes.
Prerequisites
Store the RHCOS virtual hard disk (VHD) cluster image in an Azure storage container.
Store the bootstrap Ignition config file in an Azure storage container.
Procedure
1. Copy the template from the ARM template for image storage section of this topic and save it
as 02_storage.json in your cluster’s installation directory. This template describes the image
storage that your cluster requires.
2. Export the RHCOS VHD blob URL as a variable:
$ export VHD_BLOB_URL=àz storage blob url --account-name ${CLUSTER_NAME}sa --

account-key ${ACCOUNT_KEY} -c vhd -n "rhcos.vhd" -o tsv`
3. Deploy the cluster image:

--template-file "<installation_directory>/02_storage.json" \
--parameters vhdBlobURL="${VHD_BLOB_URL}" \ 1
--parameters baseName="${INFRA_ID}" \ 2
--parameters storageAccount="${CLUSTER_NAME}sa" \ 3
--parameters architecture="<architecture>" 4
1 The blob URL of the RHCOS VHD to be used to create master and worker machines.
3 The name of your Azure storage account.
1109
4 Specify the system architecture. Valid values are x64 (default) or Arm64.
7.10.10.1. ARM template for image storage
You can use the following Azure Resource Manager (ARM) template to deploy the stored Red Hat
Enterprise Linux CoreOS (RHCOS) image that you need for your OpenShift Container Platform cluster:
Example 7.59. 02_storage.json ARM template
{
"$schema": "https://schema.management.azure.com/schemas/2019-04-
"contentVersion": "1.0.0.0",
"parameters": {
"architecture": {
"type": "string",
"metadata": {
"description": "The architecture of the Virtual Machines"
},
"defaultValue": "x64",
"allowedValues": [
"Arm64",
"x64"
]
},
"baseName": {
"type": "string",
"minLength": 1,
"metadata": {
"description": "Base name to be used in resource names (usually the cluster's Infra ID)"
}
},
"storageAccount": {
"type": "string",
"metadata": {
"description": "The Storage Account name"
}
},
"vhdBlobURL": {
"type": "string",
"metadata": {
"description": "URL pointing to the blob where the VHD to be used to create master and
worker machines is located"
}
}
},
"variables": {
"location": "[resourceGroup().location]",
"galleryName": "[concat('gallery_', replace(parameters('baseName'), '-', '_'))]",
"imageName": "[parameters('baseName')]",
"imageNameGen2": "[concat(parameters('baseName'), '-gen2')]",
"imageRelease": "1.0.0"
},
"resources": [
1110
{
"apiVersion": "2021-10-01",
"type": "Microsoft.Compute/galleries",
"name": "[variables('galleryName')]",
"location": "[variables('location')]",
"resources": [
{
"apiVersion": "2021-10-01",
"type": "images",
"name": "[variables('imageName')]",
"dependsOn": [
"[variables('galleryName')]"
],
"properties": {
"architecture": "[parameters('architecture')]",
"hyperVGeneration": "V1",
"identifier": {
"offer": "rhcos",
"publisher": "RedHat",
"sku": "basic"
},
"osState": "Generalized",
"osType": "Linux"
},
"resources": [
{
"apiVersion": "2021-10-01",
"type": "versions",
"name": "[variables('imageRelease')]",
"dependsOn": [
"[variables('imageName')]"
],
"properties": {
"publishingProfile": {
"storageAccountType": "Standard_LRS",
"targetRegions": [
{
"name": "[variables('location')]",
"regionalReplicaCount": "1"
}
]
},
"storageProfile": {
"osDiskImage": {
"source": {
"id": "[resourceId('Microsoft.Storage/storageAccounts',
parameters('storageAccount'))]",
"uri": "[parameters('vhdBlobURL')]"
}
}
}
}
}
]
1111
},
{
"apiVersion": "2021-10-01",
"type": "images",
"name": "[variables('imageNameGen2')]",
"dependsOn": [
],
"properties": {
"identifier": {
"offer": "rhcos-gen2",
"publisher": "RedHat-gen2",
"sku": "gen2"
},
"osType": "Linux"
},
"resources": [
{
"apiVersion": "2021-10-01",
"type": "versions",
"dependsOn": [
"[variables('imageNameGen2')]"
],
"properties": {
"targetRegions": [
{
}
]
},
"storageProfile": {
"osDiskImage": {
"source": {
}
}
}
}
}
]
}
]
}
]
}
1112
7.10.11. Networking requirements for user-provisioned infrastructure

All the Red Hat Enterprise Linux CoreOS (RHCOS) machines require networking to be configured in
initramfs during boot to fetch their Ignition config files.
7.10.11.1. Network connectivity requirements
You must configure the network connectivity between machines to allow OpenShift Container Platform
cluster components to communicate. Each machine must be able to resolve the hostnames of all other
machines in the cluster.
This section provides details about the ports that are required.
IMPORTANT
In connected OpenShift Container Platform environments, all nodes are required to have
internet access to pull images for platform containers and provide telemetry data to Red
Hat.
Table 7.18. Ports used for all-machine to all-machine communications
Protocol Port Description
ICMP N/A Network reachability tests
TCP 1936 Metrics
9000- 9999 Host level services, including the node exporter on ports
9100- 9101 and the Cluster Version Operator on port9099.
10250 - 10259 The default ports that Kubernetes reserves
UDP 4789 VXLAN
6081 Geneve
9100- 9101.
500 IPsec IKE packets
4500 IPsec NAT-T packets
TCP/UDP 30000 - 32767 Kubernetes node port
ESP N/A IPsec Encapsulating Security Payload (ESP)
Table 7.19. Ports used for all-machine to control plane communications
1113
TCP 6443 Kubernetes API
Table 7.20. Ports used for control plane machine to control plane machine communications
TCP 2379- 2380 etcd server and peer ports
7.10.12. Creating networking and load balancing components in Azure

You must configure networking and load balancing in Microsoft Azure for your OpenShift Container
Platform cluster to use. One way to create these components is to modify the provided Azure Resource
Manager (ARM) template.
NOTE
installation logs.
Prerequisites
Create and configure a VNet and associated subnets in Azure.
Procedure
1. Copy the template from the ARM template for the network and load balancerssection of this
topic and save it as 03_infra.json in your cluster’s installation directory. This template describes
the networking and load balancing objects that your cluster requires.

--template-file "<installation_directory>/03_infra.json" \
--parameters privateDNSZoneName="${CLUSTER_NAME}.${BASE_DOMAIN}" \ 1
1 The name of the private DNS zone.
3. Create an api DNS record in the public zone for the API public load balancer. The
${BASE_DOMAIN_RESOURCE_GROUP} variable must point to the resource group where the
public DNS zone exists.
1114
a. Export the following variable:
$ export PUBLIC_IP=àz network public-ip list -g ${RESOURCE_GROUP} --query "[?

name=='${INFRA_ID}-master-pip'] | [0].ipAddress" -o tsv`
b. Create the api DNS record in a new public zone:
$ az network dns record-set a add-record -g ${BASE_DOMAIN_RESOURCE_GROUP} -

z ${CLUSTER_NAME}.${BASE_DOMAIN} -n api -a ${PUBLIC_IP} --ttl 60
If you are adding the cluster to an existing public zone, you can create the api DNS record in
it instead:

z ${BASE_DOMAIN} -n api.${CLUSTER_NAME} -a ${PUBLIC_IP} --ttl 60
7.10.12.1. ARM template for the network and load balancers
You can use the following Azure Resource Manager (ARM) template to deploy the networking objects
and load balancers that you need for your OpenShift Container Platform cluster:
Example 7.60. 03_infra.json ARM template
{
"parameters" : {
"baseName" : {
"type" : "string",
"minLength" : 1,
"metadata" : {
}
},
"vnetBaseName": {
"type": "string",
"defaultValue": "",
"metadata" : {
"description" : "The specific customer vnet's base name (optional)"
}
},
"privateDNSZoneName" : {
"type" : "string",
"metadata" : {
"description" : "Name of the private DNS zone"
}
}
},
"variables" : {
"virtualNetworkName" : "[concat(if(not(empty(parameters('vnetBaseName'))),
parameters('vnetBaseName'), parameters('baseName')), '-vnet')]",
"virtualNetworkID" : "[resourceId('Microsoft.Network/virtualNetworks',
1115
variables('virtualNetworkName'))]",
"masterSubnetName" : "[concat(if(not(empty(parameters('vnetBaseName'))),
parameters('vnetBaseName'), parameters('baseName')), '-master-subnet')]",
"masterSubnetRef" : "[concat(variables('virtualNetworkID'), '/subnets/',
variables('masterSubnetName'))]",
"masterPublicIpAddressName" : "[concat(parameters('baseName'), '-master-pip')]",
"masterPublicIpAddressID" : "[resourceId('Microsoft.Network/publicIPAddresses',
variables('masterPublicIpAddressName'))]",
"masterLoadBalancerName" : "[parameters('baseName')]",
"masterLoadBalancerID" : "[resourceId('Microsoft.Network/loadBalancers',
variables('masterLoadBalancerName'))]",
"internalLoadBalancerName" : "[concat(parameters('baseName'), '-internal-lb')]",
"internalLoadBalancerID" : "[resourceId('Microsoft.Network/loadBalancers',
variables('internalLoadBalancerName'))]",
"skuName": "Standard"
},
"resources" : [
{
"apiVersion" : "2018-12-01",
"type" : "Microsoft.Network/publicIPAddresses",
"name" : "[variables('masterPublicIpAddressName')]",
"sku": {
"name": "[variables('skuName')]"
},
"properties" : {
"publicIPAllocationMethod" : "Static",
"dnsSettings" : {
"domainNameLabel" : "[variables('masterPublicIpAddressName')]"
}
}
},
{
"apiVersion" : "2018-12-01",
"type" : "Microsoft.Network/loadBalancers",
"name" : "[variables('masterLoadBalancerName')]",
"sku": {
},
"dependsOn" : [
"[concat('Microsoft.Network/publicIPAddresses/', variables('masterPublicIpAddressName'))]"
],
"properties" : {
"frontendIPConfigurations" : [
{
"name" : "public-lb-ip-v4",
"properties" : {
"publicIPAddress" : {
"id" : "[variables('masterPublicIpAddressID')]"
}
}
}
],
"backendAddressPools" : [
{
1116
"name" : "[variables('masterLoadBalancerName')]"
}
],
"loadBalancingRules" : [
{
"name" : "api-internal",
"properties" : {
"frontendIPConfiguration" : {
"id" :"[concat(variables('masterLoadBalancerID'), '/frontendIPConfigurations/public-lb-ip-
v4')]"
},
"backendAddressPool" : {
"id" : "[concat(variables('masterLoadBalancerID'), '/backendAddressPools/',
variables('masterLoadBalancerName'))]"
},
"protocol" : "Tcp",
"loadDistribution" : "Default",
"idleTimeoutInMinutes" : 30,
"frontendPort" : 6443,
"backendPort" : 6443,
"probe" : {
"id" : "[concat(variables('masterLoadBalancerID'), '/probes/api-internal-probe')]"
}
}
}
],
"probes" : [
{
"name" : "api-internal-probe",
"properties" : {
"protocol" : "Https",
"port" : 6443,
"requestPath": "/readyz",
"intervalInSeconds" : 10,
"numberOfProbes" : 3
}
}
]
}
},
{
"apiVersion" : "2018-12-01",
"name" : "[variables('internalLoadBalancerName')]",
"sku": {
},
"properties" : {
{
"name" : "internal-lb-ip",
"properties" : {
"privateIPAllocationMethod" : "Dynamic",
"subnet" : {
"id" : "[variables('masterSubnetRef')]"
1117
},
"privateIPAddressVersion" : "IPv4"
}
}
],
{
"name" : "internal-lb-backend"
}
],
{
"properties" : {
"id" : "[concat(variables('internalLoadBalancerID'), '/frontendIPConfigurations/internal-lb-
ip')]"
},
"enableFloatingIP" : false,
"protocol" : "Tcp",
"enableTcpReset" : false,
"id" : "[concat(variables('internalLoadBalancerID'), '/backendAddressPools/internal-lb-
backend')]"
},
"probe" : {
"id" : "[concat(variables('internalLoadBalancerID'), '/probes/api-internal-probe')]"
}
}
},
{
"name" : "sint",
"properties" : {
ip')]"
},
"protocol" : "Tcp",
backend')]"
},
"probe" : {
"id" : "[concat(variables('internalLoadBalancerID'), '/probes/sint-probe')]"
}
}
1118
}
],
"probes" : [
{
"properties" : {
"port" : 6443,
}
},
{
"name" : "sint-probe",
"properties" : {
"port" : 22623,
"requestPath": "/healthz",
}
}
]
}
},
{
"apiVersion": "2018-09-01",
"type": "Microsoft.Network/privateDnsZones/A",
"name": "[concat(parameters('privateDNSZoneName'), '/api')]",
"dependsOn" : [
"[concat('Microsoft.Network/loadBalancers/', variables('internalLoadBalancerName'))]"
],
"properties": {
"ttl": 60,
"aRecords": [
{
"ipv4Address": "
[reference(variables('internalLoadBalancerName')).frontendIPConfigurations[0].properties.privateIP
Address]"
}
]
}
},
{
"apiVersion": "2018-09-01",
"name": "[concat(parameters('privateDNSZoneName'), '/api-int')]",
"dependsOn" : [
],
"properties": {
"ttl": 60,
"aRecords": [
1119
{
"ipv4Address": "
Address]"
}
]
}
}
]
}
7.10.13. Creating the bootstrap machine in Azure

You must create the bootstrap machine in Microsoft Azure to use during OpenShift Container Platform
cluster initialization. One way to create this machine is to modify the provided Azure Resource Manager
(ARM) template.
NOTE
If you do not use the provided ARM template to create your bootstrap machine, you must
review the provided information and manually create the infrastructure. If your cluster
does not initialize correctly, you might have to contact Red Hat support with your
installation logs.
Prerequisites
Create and configure networking and load balancers in Azure.
Create control plane and compute roles.
Procedure
1. Copy the template from the ARM template for the bootstrap machinesection of this topic
and save it as 04_bootstrap.json in your cluster’s installation directory. This template describes
the bootstrap machine that your cluster requires.
2. Export the bootstrap URL variable:
$ bootstrap_url_expiry=`date -u -d "10 hours" '+%Y-%m-%dT%H:%MZ'`
$ export BOOTSTRAP_URL=àz storage blob generate-sas -c 'files' -n 'bootstrap.ign' --https-

only --full-uri --permissions r --expiry $bootstrap_url_expiry --account-name
${CLUSTER_NAME}sa --account-key ${ACCOUNT_KEY} -o tsv`
3. Export the bootstrap ignition variable:
1120
$ export BOOTSTRAP_IGNITION=`jq -rcnM --arg v "3.2.0" --arg url ${BOOTSTRAP_URL}

'{ignition:{version:$v,config:{replace:{source:$url}}}}' | base64 | tr -d '\n'`

--template-file "<installation_directory>/04_bootstrap.json" \
--parameters bootstrapIgnition="${BOOTSTRAP_IGNITION}" \ 1
--parameter bootstrapVMSize="Standard_D4s_v3" 3
1 The bootstrap Ignition content for the bootstrap cluster.
3 Optional: Specify the size of the bootstrap VM. Use a VM size compatible with your
specified architecture. If this value is not defined, the default value from the template is
set.
7.10.13.1. ARM template for the bootstrap machine
You can use the following Azure Resource Manager (ARM) template to deploy the bootstrap machine
that you need for your OpenShift Container Platform cluster:
Example 7.61. 04_bootstrap.json ARM template
{
"parameters" : {
"baseName" : {
"type" : "string",
"minLength" : 1,
"metadata" : {
}
},
"vnetBaseName": {
"type": "string",
"defaultValue": "",
"metadata" : {
}
},
"bootstrapIgnition" : {
"type" : "string",
"minLength" : 1,
"metadata" : {
"description" : "Bootstrap ignition content for the bootstrap cluster"
}
},
"sshKeyData" : {
1121
"type" : "securestring",
"defaultValue" : "Unused",
"metadata" : {
"description" : "Unused"
}
},
"bootstrapVMSize" : {
"type" : "string",
"defaultValue" : "Standard_D4s_v3",
"metadata" : {
"description" : "The size of the Bootstrap Virtual Machine"
}
},
"hyperVGen": {
"type": "string",
"metadata": {
"description": "VM generation image to use"
},
"defaultValue": "V2",
"allowedValues": [
"V1",
"V2"
]
}
},
"variables" : {
"sshKeyPath" : "/home/core/.ssh/authorized_keys",
"identityName" : "[concat(parameters('baseName'), '-identity')]",
"vmName" : "[concat(parameters('baseName'), '-bootstrap')]",
"nicName" : "[concat(variables('vmName'), '-nic')]",
"imageName" : "[concat(parameters('baseName'), if(equals(parameters('hyperVGen'), 'V2'), '-
gen2', ''))]",
"clusterNsgName" : "[concat(if(not(empty(parameters('vnetBaseName'))),
parameters('vnetBaseName'), parameters('baseName')), '-nsg')]",
"sshPublicIpAddressName" : "[concat(variables('vmName'), '-ssh-pip')]"
},
"resources" : [
{
"apiVersion" : "2018-12-01",
"name" : "[variables('sshPublicIpAddressName')]",
"sku": {
"name": "Standard"
1122
},
"properties" : {
"dnsSettings" : {
"domainNameLabel" : "[variables('sshPublicIpAddressName')]"
}
}
},
{
"apiVersion" : "2018-06-01",
"type" : "Microsoft.Network/networkInterfaces",
"name" : "[variables('nicName')]",
"dependsOn" : [
"[resourceId('Microsoft.Network/publicIPAddresses', variables('sshPublicIpAddressName'))]"
],
"properties" : {
"ipConfigurations" : [
{
"name" : "pipConfig",
"properties" : {
"publicIPAddress": {
"id": "[resourceId('Microsoft.Network/publicIPAddresses',
variables('sshPublicIpAddressName'))]"
},
"subnet" : {
},
"loadBalancerBackendAddressPools" : [
{
"id" : "[concat('/subscriptions/', subscription().subscriptionId, '/resourceGroups/',
resourceGroup().name, '/providers/Microsoft.Network/loadBalancers/',
variables('masterLoadBalancerName'), '/backendAddressPools/',
},
{
variables('internalLoadBalancerName'), '/backendAddressPools/internal-lb-backend')]"
}
]
}
}
]
}
},
{
"apiVersion" : "2018-06-01",
"type" : "Microsoft.Compute/virtualMachines",
"name" : "[variables('vmName')]",
"identity" : {
"type" : "userAssigned",
"userAssignedIdentities" : {
"[resourceID('Microsoft.ManagedIdentity/userAssignedIdentities/',
1123
variables('identityName'))]" : {}
}
},
"dependsOn" : [
"[concat('Microsoft.Network/networkInterfaces/', variables('nicName'))]"
],
"properties" : {
"hardwareProfile" : {
"vmSize" : "[parameters('bootstrapVMSize')]"
},
"osProfile" : {
"computerName" : "[variables('vmName')]",
"adminUsername" : "core",
"adminPassword" : "NotActuallyApplied!",
"customData" : "[parameters('bootstrapIgnition')]",
"linuxConfiguration" : {
"disablePasswordAuthentication" : false
}
},
"storageProfile" : {
"imageReference": {
"id": "[resourceId('Microsoft.Compute/galleries/images', variables('galleryName'),
variables('imageName'))]"
},
"osDisk" : {
"name": "[concat(variables('vmName'),'_OSDisk')]",
"osType" : "Linux",
"createOption" : "FromImage",
"managedDisk": {
"storageAccountType": "Premium_LRS"
},
"diskSizeGB" : 100
}
},
"networkProfile" : {
"networkInterfaces" : [
{
"id" : "[resourceId('Microsoft.Network/networkInterfaces', variables('nicName'))]"
}
]
}
}
},
{
"apiVersion" : "2018-06-01",
"type": "Microsoft.Network/networkSecurityGroups/securityRules",
"name" : "[concat(variables('clusterNsgName'), '/bootstrap_ssh_in')]",
"dependsOn" : [
"[resourceId('Microsoft.Compute/virtualMachines', variables('vmName'))]"
],
"properties": {
"protocol" : "Tcp",
1124
"access" : "Allow",
"priority" : 100,
}
}
]
}
7.10.14. Creating the control plane machines in Azure

You must create the control plane machines in Microsoft Azure for your cluster to use. One way to
create these machines is to modify the provided Azure Resource Manager (ARM) template.
NOTE
By default, Microsoft Azure places control plane machines and compute machines in a
pre-set availability zone. You can manually set an availability zone for a compute node or
control plane node. To do this, modify a vendor’s Azure Resource Manager (ARM)
template by specifying each of your availability zones in the zones parameter of the
virtual machine resource.
If you do not use the provided ARM template to create your control plane machines, you must review the
provided information and manually create the infrastructure. If your cluster does not initialize correctly,
consider contacting Red Hat support with your installation logs.
Prerequisites
Create the bootstrap machine.
Procedure
1. Copy the template from the ARM template for control plane machines section of this topic
and save it as 05_masters.json in your cluster’s installation directory. This template describes
the control plane machines that your cluster requires.
2. Export the following variable needed by the control plane machine deployment:
$ export MASTER_IGNITION=`cat <installation_directory>/master.ign | base64 | tr -d '\n'`
1125
--template-file "<installation_directory>/05_masters.json" \
--parameters masterIgnition="${MASTER_IGNITION}" \ 1
--parameters masterVMSize="Standard_D8s_v3" 3
1 The Ignition content for the control plane nodes.
3 Optional: Specify the size of the Control Plane VM. Use a VM size compatible with your
set.
7.10.14.1. ARM template for control plane machines
You can use the following Azure Resource Manager (ARM) template to deploy the control plane
machines that you need for your OpenShift Container Platform cluster:
Example 7.62. 05_masters.json ARM template
{
"parameters" : {
"baseName" : {
"type" : "string",
"minLength" : 1,
"metadata" : {
}
},
"vnetBaseName": {
"type": "string",
"defaultValue": "",
"metadata" : {
}
},
"masterIgnition" : {
"type" : "string",
"metadata" : {
"description" : "Ignition content for the master nodes"
}
},
"numberOfMasters" : {
"type" : "int",
"defaultValue" : 3,
"minValue" : 2,
"maxValue" : 30,
"metadata" : {
"description" : "Number of OpenShift masters to deploy"
}
},
1126
"sshKeyData" : {
"metadata" : {
}
},
"type" : "string",
"defaultValue" : "",
"metadata" : {
"description" : "unused"
}
},
"masterVMSize" : {
"type" : "string",
"metadata" : {
"description" : "The size of the Master Virtual Machines"
}
},
"diskSizeGB" : {
"type" : "int",
"defaultValue" : 1024,
"metadata" : {
"description" : "Size of the Master VM OS disk, in GB"
}
},
"hyperVGen": {
"type": "string",
"metadata": {
},
"allowedValues": [
"V1",
"V2"
]
}
},
"variables" : {
1127
gen2', ''))]",
"copy" : [
{
"name" : "vmNames",
"count" : "[parameters('numberOfMasters')]",
"input" : "[concat(parameters('baseName'), '-master-', copyIndex('vmNames'))]"
}
]
},
"resources" : [
{
"apiVersion" : "2018-06-01",
"copy" : {
"name" : "nicCopy",
"count" : "[length(variables('vmNames'))]"
},
"name" : "[concat(variables('vmNames')[copyIndex()], '-nic')]",
"properties" : {
{
"properties" : {
"subnet" : {
},
{
},
{
}
]
}
}
]
}
},
{
"apiVersion" : "2018-06-01",
"copy" : {
"name" : "vmCopy",
},
"name" : "[variables('vmNames')[copyIndex()]]",
"identity" : {
1128
}
},
"dependsOn" : [
"[concat('Microsoft.Network/networkInterfaces/', concat(variables('vmNames')[copyIndex()], '-
nic'))]"
],
"properties" : {
"vmSize" : "[parameters('masterVMSize')]"
},
"osProfile" : {
"computerName" : "[variables('vmNames')[copyIndex()]]",
"customData" : "[parameters('masterIgnition')]",
}
},
"imageReference": {
},
"osDisk" : {
"name": "[concat(variables('vmNames')[copyIndex()], '_OSDisk')]",
"osType" : "Linux",
"caching": "ReadOnly",
"writeAcceleratorEnabled": false,
"managedDisk": {
},
"diskSizeGB" : "[parameters('diskSizeGB')]"
}
},
{
"id" : "[resourceId('Microsoft.Network/networkInterfaces', concat(variables('vmNames')
[copyIndex()], '-nic'))]",
"properties": {
"primary": false
}
}
]
}
}
}
]
}
1129
7.10.15. Wait for bootstrap completion and remove bootstrap resources in Azure
After you create all of the required infrastructure in Microsoft Azure, wait for the bootstrap process to
complete on the machines that you provisioned by using the Ignition config files that you generated
with the installation program.
Prerequisites
Create the control plane machines.
Procedure
1. Change to the directory that contains the installation program and run the following command:

--log-level info 2
If the command exits without a FATAL warning, your production control plane has initialized.
2. Delete the bootstrap resources:
$ az network nsg rule delete -g ${RESOURCE_GROUP} --nsg-name ${INFRA_ID}-nsg --

name bootstrap_ssh_in
$ az vm stop -g ${RESOURCE_GROUP} --name ${INFRA_ID}-bootstrap
$ az vm deallocate -g ${RESOURCE_GROUP} --name ${INFRA_ID}-bootstrap
$ az vm delete -g ${RESOURCE_GROUP} --name ${INFRA_ID}-bootstrap --yes
$ az disk delete -g ${RESOURCE_GROUP} --name ${INFRA_ID}-bootstrap_OSDisk --no-
wait --yes
$ az network nic delete -g ${RESOURCE_GROUP} --name ${INFRA_ID}-bootstrap-nic --no-
wait
$ az storage blob delete --account-key ${ACCOUNT_KEY} --account-name
${CLUSTER_NAME}sa --container-name files --name bootstrap.ign
$ az network public-ip delete -g ${RESOURCE_GROUP} --name ${INFRA_ID}-bootstrap-
ssh-pip
NOTE
1130
NOTE
If you do not delete the bootstrap server, installation may not succeed due to API traffic
being routed to the bootstrap server.
7.10.16. Creating additional worker machines in Azure

You can create worker machines in Microsoft Azure for your cluster to use by launching individual
instances discretely or by automated processes outside the cluster, such as auto scaling groups. You can
also take advantage of the built-in cluster scaling mechanisms and the machine API in OpenShift
Container Platform.
In this example, you manually launch one instance by using the Azure Resource Manager (ARM)
template. Additional instances can be launched by including additional resources of type
06_workers.json in the file.
NOTE
control plane node. To do this, modify a vendor’s ARM template by specifying each of
your availability zones in the zones parameter of the virtual machine resource.
Prerequisites
Procedure
1. Copy the template from the ARM template for worker machines section of this topic and save
it as 06_workers.json in your cluster’s installation directory. This template describes the worker
2. Export the following variable needed by the worker machine deployment:
$ export WORKER_IGNITION=`cat <installation_directory>/worker.ign | base64 | tr -d '\n'`
1131

--template-file "<installation_directory>/06_workers.json" \
--parameters workerIgnition="${WORKER_IGNITION}" \ 1
--parameters nodeVMSize="Standard_D4s_v3" 3
1 The Ignition content for the worker nodes.
3 Optional: Specify the size of the compute node VM. Use a VM size compatible with your
set.
7.10.16.1. ARM template for worker machines
You can use the following Azure Resource Manager (ARM) template to deploy the worker machines
Example 7.63. 06_workers.json ARM template
{
"parameters" : {
"baseName" : {
"type" : "string",
"minLength" : 1,
"metadata" : {
}
},
"vnetBaseName": {
"type": "string",
"defaultValue": "",
"metadata" : {
}
},
"workerIgnition" : {
"type" : "string",
"metadata" : {
"description" : "Ignition content for the worker nodes"
}
},
"numberOfNodes" : {
"type" : "int",
"defaultValue" : 3,
"minValue" : 2,
"maxValue" : 30,
"metadata" : {
"description" : "Number of OpenShift compute nodes to deploy"
}
1132
},
"sshKeyData" : {
"metadata" : {
}
},
"nodeVMSize" : {
"type" : "string",
"metadata" : {
"description" : "The size of the each Node Virtual Machine"
}
},
"hyperVGen": {
"type": "string",
"metadata": {
},
"allowedValues": [
"V1",
"V2"
]
}
},
"variables" : {
"nodeSubnetName" : "[concat(if(not(empty(parameters('vnetBaseName'))),
parameters('vnetBaseName'), parameters('baseName')), '-worker-subnet')]",
"nodeSubnetRef" : "[concat(variables('virtualNetworkID'), '/subnets/',
variables('nodeSubnetName'))]",
"infraLoadBalancerName" : "[parameters('baseName')]",
"sshKeyPath" : "/home/capi/.ssh/authorized_keys",
gen2', ''))]",
"copy" : [
{
"name" : "vmNames",
"count" : "[parameters('numberOfNodes')]",
"input" : "[concat(parameters('baseName'), '-worker-', variables('location'), '-',
copyIndex('vmNames', 1))]"
}
]
},
"resources" : [
{
"apiVersion" : "2019-05-01",
"name" : "[concat('node', copyIndex())]",
1133
"type" : "Microsoft.Resources/deployments",
"copy" : {
"name" : "nodeCopy",
},
"properties" : {
"mode" : "Incremental",
"template" : {
"$schema" : "http://schema.management.azure.com/schemas/2015-01-
"resources" : [
{
"apiVersion" : "2018-06-01",
"properties" : {
{
"properties" : {
"subnet" : {
"id" : "[variables('nodeSubnetRef')]"
}
}
}
]
}
},
{
"apiVersion" : "2018-06-01",
"tags" : {
"kubernetes.io-cluster-ffranzupi": "owned"
},
"identity" : {
}
},
"dependsOn" : [
],
"properties" : {
"vmSize" : "[parameters('nodeVMSize')]"
},
"osProfile" : {
1134
"adminUsername" : "capi",
"customData" : "[parameters('workerIgnition')]",
}
},
"imageReference": {
},
"osDisk" : {
"name": "[concat(variables('vmNames')[copyIndex()],'_OSDisk')]",
"osType" : "Linux",
"managedDisk": {
},
"diskSizeGB": 128
}
},
{
"id" : "[resourceId('Microsoft.Network/networkInterfaces',
concat(variables('vmNames')[copyIndex()], '-nic'))]",
"properties": {
"primary": true
}
}
]
}
}
}
]
}
}
}
]
}

IMPORTANT
1135
Procedure
Portal.
$ tar xvf <file>

$ echo $PATH
Verification
$ oc <command>

Procedure
Portal.

C:\> path
Verification
C:\> oc <command>
1136

Procedure
Portal.
NOTE

$ echo $PATH
Verification
$ oc <command>

Prerequisites
Procedure
1137
$ oc whoami
Example output
system:admin

Prerequisites
Procedure
$ oc get nodes
Example output

NOTE
$ oc get csr
Example output

...
1138
list.
NOTE
NOTE
of the node.

NOTE
$ oc get csr
Example output
1139

Pending
Pending
...

$ oc get nodes
Example output

NOTE
7.10.20. Adding the Ingress DNS records

If you removed the DNS Zone configuration when creating Kubernetes manifests and generating
Ignition configs, you must manually create DNS records that point at the Ingress load balancer. You can
create either a wildcard *.apps.{baseDomain}. or specific records. You can use A, CNAME, and other
records per your requirements.
Prerequisites
1140
You deployed an OpenShift Container Platform cluster on Microsoft Azure by using

infrastructure that you provisioned.
Install the OpenShift CLI (oc).
Procedure
1. Confirm the Ingress router has created a load balancer and populated the EXTERNAL-IP field:
Example output
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

router-default LoadBalancer 172.30.20.10 35.130.120.110
80:32288/TCP,443:31215/TCP 20
2. Export the Ingress router IP as a variable:
$ export PUBLIC_IP_ROUTER=òc -n openshift-ingress get service router-default --no-

headers | awk '{print $4}'`
3. Add a *.apps record to the public DNS zone.
a. If you are adding this cluster to a new public zone, run:

z ${CLUSTER_NAME}.${BASE_DOMAIN} -n *.apps -a ${PUBLIC_IP_ROUTER} --ttl 300
b. If you are adding this cluster to an already existing public zone, run:

z ${BASE_DOMAIN} -n *.apps.${CLUSTER_NAME} -a ${PUBLIC_IP_ROUTER} --ttl 300
4. Add a *.apps record to the private DNS zone:
a. Create a *.apps record by using the following command:
$ az network private-dns record-set a create -g ${RESOURCE_GROUP} -z

${CLUSTER_NAME}.${BASE_DOMAIN} -n *.apps --ttl 300
b. Add the *.apps record to the private DNS zone by using the following command:
$ az network private-dns record-set a add-record -g ${RESOURCE_GROUP} -z

${CLUSTER_NAME}.${BASE_DOMAIN} -n *.apps -a ${PUBLIC_IP_ROUTER}
If you prefer to add explicit domains instead of using a wildcard, you can create entries for each of the
cluster’s current routes:
$ oc get --all-namespaces -o jsonpath='{range .items[*]}{range .status.ingress[*]}{.host}{"\n"}{end}

{end}' routes
1141
Example output
oauth-openshift.apps.cluster.basedomain.com
console-openshift-console.apps.cluster.basedomain.com
downloads-openshift-console.apps.cluster.basedomain.com
alertmanager-main-openshift-monitoring.apps.cluster.basedomain.com
prometheus-k8s-openshift-monitoring.apps.cluster.basedomain.com
7.10.21. Completing an Azure installation on user-provisioned infrastructure

After you start the OpenShift Container Platform installation on Microsoft Azure user-provisioned
infrastructure, you can monitor the cluster events until the cluster is ready.
Prerequisites
Deploy the bootstrap machine for an OpenShift Container Platform cluster on user-provisioned
Azure infrastructure.
Install the oc CLI and log in.
Procedure
Complete the cluster installation:
Example output
INFO Waiting up to 30m0s for the cluster to initialize...
IMPORTANT
installation.
1142
7.11. INSTALLING A CLUSTER ON AZURE USING ARM TEMPLATES

In OpenShift Container Platform version 4.15, you can install a cluster on Microsoft Azure by using
infrastructure that you provide.
IMPORTANT

own. You are also free to create the required resources through other methods; the
templates are just an example.
processes.
users.
You configured an Azure account to host the cluster.
You downloaded the Azure CLI and installed it on your computer. See Install the Azure CLI in
the Azure documentation. The documentation below was last tested using version 2.38.0 of the
Azure CLI. Azure CLI commands might perform differently based on the version you use.
namespace, see Alternatives to storing administrator-level secrets in the kube-system project .
NOTE
1143

IMPORTANT
7.11.3. Configuring your Azure project

IMPORTANT
documentation.
7.11.3.1. Azure account limits
IMPORTANT
1144

nt components limit
required by
default


instances:

after installation
Because the bootstrap machine uses

Standard_D4s_v3 machines, which use 4 vCPUs,
the control plane machines use Standard_D8s_v3
virtual machines, which use 8 vCPUs, and the worker
machines use Standard_D4s_v3 virtual machines,
which use 4 vCPUs, a default cluster requires 40
vCPUs. The bootstrap node VM, which uses 4
vCPUs, is used only during installation.



1145

nt components limit
required by
default


olp
lan
e

load
balancers
t


al


installation.

1146

nt components limit
required by
default
spot VMs, your
cluster must have
two spot VM
NOTE
vCPUs for every
Optimizing storage
7.11.3.2. Configuring a public DNS zone in Azure
Procedure
NOTE
parent domain.
7.11.3.3. Increasing Azure account limits
1147
NOTE
Procedure
and approving them.
7.11.3.5. Recording the subscription and tenant IDs
The installation program requires the subscription and tenant IDs that are associated with your Azure
account. You can use the Azure CLI to gather this information.
Prerequisites
Procedure
1. Log in to the Azure CLI by running the following command:
$ az login
2. Ensure that you are using the right subscription:
1148
a. View a list of available subscriptions by running the following command:
Example output
[
{
"isDefault": true,
"state": "Enabled",
"user": {
"type": "user"
}
},
{
"isDefault": false,
"state": "Enabled",
"user": {
"type": "user"
}
}
]
b. View the details of the active account, and confirm that this is the subscription you want to
use, by running the following command:
$ az account show
Example output
{
"isDefault": true,
"state": "Enabled",
"user": {
"type": "user"
}
}
3. If you are not using the right subscription:
1149
a. Change the active subscription by running the following command:
$ az account set -s <subscription_id>
b. Verify that you are using the subscription you need by running the following command:
$ az account show
Example output
{
"isDefault": true,
"state": "Enabled",
"user": {
"type": "user"
}
}
4. Record the id and tenantId parameter values from the output. You require these values to
install an OpenShift Container Platform cluster.
7.11.3.6. Supported identities to access Azure resources
resources. As such, you need one of the following types of identities to complete the installation:
A service principal
A system-assigned managed identity
A user-assigned managed identity
7.11.3.7. Required Azure permissions for user-provisioned infrastructure
1150
Microsoft Azure.
1151
Microsoft.Compute/virtualMachines/deallocate/action
1152
1153
Example 7.72. Required permissions for creating deployments
Microsoft.Resources/deployments/read
Microsoft.Resources/deployments/write
Microsoft.Resources/deployments/validate/action
Microsoft.Resources/deployments/operationstatuses/read
1154
Microsoft Azure.
1155
NOTE
To install OpenShift Container Platform on Azure, you must scope the permissions
related to resource group creation to your subscription. After the resource group is
created, you can scope the rest of the permissions to the created resource group. If the
public DNS zone is present in a different resource group, then the network DNS zone
related permissions must always be applied to your subscription.
7.11.3.8. Using Azure managed identities
The installation program requires an Azure identity to complete the installation. You can use either a
system-assigned or user-assigned managed identity.
If you are unable to use a managed identity, you can use a service principal.
Procedure
1156
will run the installation program from.
2. If you are using a user-assigned managed identity:
a. Assign it to the virtual machine that you will run the installation program from.
b. Record its client ID. You require this value when installing the cluster.
For more information about viewing the details of a user-assigned managed identity, see
the Microsoft Azure documentation for listing user-assigned managed identities .
3. Verify that the required permissions are assigned to the managed identity.
The installation program requires an Azure identity to complete the installation. You can use a service
principal.
If you are unable to use a service principal, you can use a managed identity.
Prerequisites
You have an Azure subscription ID.
service principal, you have created a custom role with the required Azure permissions.
Procedure
1. Create the service principal for your account by running the following command:

Example output

{
"appId": "axxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
1157
"password": "00000000-0000-0000-0000-000000000000",
"tenantId": "8xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
}
2. Record the values of the appId and password parameters from the output. You require these
values when installing the cluster.

--scope /subscriptions/<subscription_id> 2
7.11.3.10. Supported Azure regions
your subscription.
eastus (East US)
eastus2 (East US 2)
1158
uksouth (UK South)
ukwest (UK West)
westus (West US)
westus2 (West US 2)
westus3 (West US 3)

1159

Hosts Description
control plane.
IMPORTANT
machines.
1160

System Per Second
(IOPS)[2]

8.6 and later
[3]
IMPORTANT
to true.
Optimizing storage
c4.*
c5.*
c5a.*
i3.*
1161
m4.*
m5.*
m5a.*
m6a.*
m6i.*
r4.*
r5.*
r5a.*
r6i.*
t3.*
t3a.*
Platform.
c6g.*
m6g.*

Red Hat.
IMPORTANT
1162
IMPORTANT
ARM instances.
Prerequisites
Azure CLI client.
Procedure
commands:
North America:
Example output

------------- -------------- ------------------ ----------------------------------------------------------
---- -----------------
worker:413.92.2023101700 413.92.2023101700
gen1:413.92.2023101700 413.92.2023101700
EMEA:
Example output

Version
------------- -------------- ------------------ ----------------------------------------------------------
---- -----------------
worker:413.92.2023101700 413.92.2023101700
worker-gen1:413.92.2023101700 413.92.2023101700
NOTE
North America:
1163
North America:
EMEA:
North America:
EMEA:
North America:
EMEA:
5. Record the image details of your offer. If you use the Azure Resource Manager (ARM) template
to deploy your worker nodes:
a. Update storageProfile.imageReference by deleting the id parameter and adding the

offer, publisher, sku, and version parameters by using the values from your offer.
b. Specify a plan for the virtual machines (VMs).
Example 06_workers.json ARM template with an updated

storageProfile.imageReference object and a specified plan
...
"plan" : {
"name": "rh-ocp-worker",
"product": "rh-ocp-worker",
"publisher": "redhat"
},
"dependsOn" : [
],
"properties" : {
...
"storageProfile": {
"imageReference": {
"offer": "rh-ocp-worker",
1164
"publisher": "redhat",
"sku": "rh-ocp-worker",
"version": "413.92.2023101700"
}
...
}
...
}

for installation.
Prerequisites
Procedure
IMPORTANT
IMPORTANT
components.
1165

authentication.
user.
IMPORTANT
NOTE
Procedure
NOTE
ecdsa algorithm.
1166
NOTE

task:
Example output
Agent pid 31874
NOTE

Example output
Next steps
7.11.8. Creating the installation files for Azure

To install OpenShift Container Platform on Microsoft Azure using user-provisioned infrastructure, you
must generate the files that the installation program needs to deploy your cluster and modify them so
that the cluster creates only the machines that it will use. You generate and customize the install-
config.yaml file, Kubernetes manifests, and Ignition config files. You also have the option to first set up
a separate var partition during the preparation phases of installation.
1167
installation.
IMPORTANT
Procedure
Example output


1168
Example output
...
variant: openshift
version: 4.15.0
metadata:
labels:
storage:
disks:
partitions:
- label: var
number: 5
filesystems:
path: /var
format: xfs
NOTE
name.
1169

partition.yaml

Prerequisites
cluster.
prerequisites:
Procedure
command:
For <installation_directory>, specify the directory name to store the files that the
1170
NOTE


values.
1171
IMPORTANT
NOTE

more information, see "Installing a three-node cluster on Azure".
IMPORTANT


platform.
Prerequisites
NOTE
1172
NOTE
(169.254.169.254).
Procedure
apiVersion: v1
proxy:
must be http.

destinations.
NOTE
1173
NOTE
NOTE
created.
(ARM) templates used to assist in completing a user-provided infrastructure install on Microsoft Azure.
NOTE
Prerequisites
cluster.
Procedure
templates:
2 The region to deploy the cluster into, for example centralus. This is the value of the
.platform.azure.region attribute from the install-config.yaml file.
file.
1174
4 The base domain to deploy the cluster to. The base domain corresponds to the public
DNS zone that you created for your cluster. This is the value of the .baseDomain attribute
5 The resource group where the public DNS zone exists. This is the value of the
config.yaml file.
For example:
machines.
IMPORTANT
Prerequisites
1175
Procedure
machines.
machine-set.yaml
IMPORTANT

these machines.

WARNING
IMPORTANT
1176

machines:
kind: DNS
metadata:
name: cluster
spec:
privateZone: 1
publicZone: 2
status: {}
templates:
config.yml file.
1177
.
├── auth

You must create a Microsoft Azure resource group and an identity for that resource group. These are
both used during the installation of your OpenShift Container Platform cluster on Azure.
Prerequisites
Procedure
1. Create the resource group in a supported Azure region:
2. Create an Azure identity for the resource group:
$ az identity create -g ${RESOURCE_GROUP} -n ${INFRA_ID}-identity
This is used to grant the required access to Operators in your cluster. For example, this allows
the Ingress Operator to create a public IP and its load balancer. You must assign the Azure
identity to a role.
3. Grant the Contributor role to the Azure identity:
a. Export the following variables required by the Azure role assignment:
$ export PRINCIPAL_ID=àz identity show -g ${RESOURCE_GROUP} -n ${INFRA_ID}-

identity --query principalId --out tsv`
1178
$ export RESOURCE_GROUP_ID=àz group show -g ${RESOURCE_GROUP} --query

id --out tsv`
b. Assign the Contributor role to the identity:
$ az role assignment create --assignee "${PRINCIPAL_ID}" --role 'Contributor' --scope

"${RESOURCE_GROUP_ID}"
NOTE
If you want to assign a custom role with all the required permissions to the
identity, run the following command:
$ az role assignment create --assignee "${PRINCIPAL_ID}" --role

<custom_role> \ 1
--scope "${RESOURCE_GROUP_ID}"
1 Specifies the custom role name.
Prerequisites
Procedure


WARNING

1179

$ export VHD_URL=òpenshift-install coreos print-stream-json | jq -r '.architectures.

<architecture>."rhel-coreos-extensions"."azure-disk".url'`
where:
<architecture>
Specifies the architecture, valid values include x86_64 or aarch64.
IMPORTANT
The RHCOS images might not change with every release of OpenShift
Container Platform. You must specify an image with the highest version that
is less than or equal to the OpenShift Container Platform version that you
install. Use the image version that matches your OpenShift Container
Platform version if it is available.

key ${ACCOUNT_KEY}
$ az storage blob copy start --account-name ${CLUSTER_NAME}sa --account-key

${ACCOUNT_KEY} --destination-blob "rhcos.vhd" --destination-container vhd --source-uri
"${VHD_URL}"



For this example, Azure’s DNS solution is used, so you will create a new public DNS zone for external
(internet) visibility and a private DNS zone for internal cluster resolution.
NOTE
1180
NOTE
The public DNS zone is not required to exist in the same resource group as the cluster
that is the case, you can skip creating the public DNS zone; be sure the installation config
you generated earlier reflects that scenario.
Prerequisites
Procedure
1. Create the new public DNS zone in the resource group exported in the

You can skip this step if you are using a public DNS zone that already exists.
2. Create the private DNS zone in the same resource group as the rest of this deployment:
$ az network private-dns zone create -g ${RESOURCE_GROUP} -n

You can learn more about configuring a public DNS zone in Azure by visiting that section.
7.11.12. Creating a VNet in Azure

You must create a virtual network (VNet) in Microsoft Azure for your OpenShift Container Platform
cluster to use. You can customize the VNet to meet your requirements. One way to create the VNet is to
modify the provided Azure Resource Manager (ARM) template.
NOTE
installation logs.
Prerequisites
Procedure
cluster requires.
1181

3. Link the VNet template to the private DNS zone:
$ az network private-dns link vnet create -g ${RESOURCE_GROUP} -z

${CLUSTER_NAME}.${BASE_DOMAIN} -n ${INFRA_ID}-network-link -v "${INFRA_ID}-vnet"
-e false
{
"parameters" : {
"baseName" : {
"type" : "string",
"minLength" : 1,
"metadata" : {
}
}
},
"variables" : {
"virtualNetworkName" : "[concat(parameters('baseName'), '-vnet')]",
"addressPrefix" : "10.0.0.0/16",
"masterSubnetName" : "[concat(parameters('baseName'), '-master-subnet')]",
"masterSubnetPrefix" : "10.0.0.0/24",
"nodeSubnetName" : "[concat(parameters('baseName'), '-worker-subnet')]",
"nodeSubnetPrefix" : "10.0.1.0/24",
"clusterNsgName" : "[concat(parameters('baseName'), '-nsg')]"
},
"resources" : [
{
"apiVersion" : "2018-12-01",
"type" : "Microsoft.Network/virtualNetworks",
"name" : "[variables('virtualNetworkName')]",
"dependsOn" : [
"[concat('Microsoft.Network/networkSecurityGroups/', variables('clusterNsgName'))]"
],
"properties" : {
1182
"addressSpace" : {
"addressPrefixes" : [
"[variables('addressPrefix')]"
]
},
"subnets" : [
{
"name" : "[variables('masterSubnetName')]",
"properties" : {
"addressPrefix" : "[variables('masterSubnetPrefix')]",
}
}
},
{
"name" : "[variables('nodeSubnetName')]",
"properties" : {
"addressPrefix" : "[variables('nodeSubnetPrefix')]",
}
}
}
]
}
},
{
"type" : "Microsoft.Network/networkSecurityGroups",
"name" : "[variables('clusterNsgName')]",
"apiVersion" : "2018-10-01",
"properties" : {
"securityRules" : [
{
"name" : "apiserver_in",
"properties" : {
"protocol" : "Tcp",
"access" : "Allow",
"priority" : 101,
}
}
]
}
}
]
}
1183
7.11.13. Deploying the RHCOS cluster image for the Azure infrastructure
You must use a valid Red Hat Enterprise Linux CoreOS (RHCOS) image for Microsoft Azure for your
OpenShift Container Platform nodes.
Prerequisites
Procedure


1184
"$schema": "https://schema.management.azure.com/schemas/2019-04-
"contentVersion": "1.0.0.0",
"parameters": {
"architecture": {
"type": "string",
"metadata": {
"description": "The architecture of the Virtual Machines"
},
"defaultValue": "x64",
"allowedValues": [
"Arm64",
"x64"
]
},
"baseName": {
"type": "string",
"minLength": 1,
"metadata": {
"description": "Base name to be used in resource names (usually the cluster's Infra ID)"
}
},
"storageAccount": {
"type": "string",
"metadata": {
"description": "The Storage Account name"
}
},
"vhdBlobURL": {
"type": "string",
"metadata": {
"description": "URL pointing to the blob where the VHD to be used to create master and
worker machines is located"
}
}
},
"variables": {
"location": "[resourceGroup().location]",
"imageName": "[parameters('baseName')]",
"imageNameGen2": "[concat(parameters('baseName'), '-gen2')]",
"imageRelease": "1.0.0"
},
"resources": [
{
"apiVersion": "2021-10-01",
"type": "Microsoft.Compute/galleries",
"name": "[variables('galleryName')]",
"resources": [
{
"apiVersion": "2021-10-01",
"type": "images",
"name": "[variables('imageName')]",
"dependsOn": [
1185
],
"properties": {
"identifier": {
"offer": "rhcos",
"publisher": "RedHat",
"sku": "basic"
},
"osType": "Linux"
},
"resources": [
{
"apiVersion": "2021-10-01",
"type": "versions",
"dependsOn": [
"[variables('imageName')]"
],
"properties": {
"targetRegions": [
{
}
]
},
"storageProfile": {
"osDiskImage": {
"source": {
}
}
}
}
}
]
},
{
"apiVersion": "2021-10-01",
"type": "images",
"name": "[variables('imageNameGen2')]",
"dependsOn": [
],
"properties": {
1186
"identifier": {
"offer": "rhcos-gen2",
"publisher": "RedHat-gen2",
"sku": "gen2"
},
"osType": "Linux"
},
"resources": [
{
"apiVersion": "2021-10-01",
"type": "versions",
"dependsOn": [
"[variables('imageNameGen2')]"
],
"properties": {
"targetRegions": [
{
}
]
},
"storageProfile": {
"osDiskImage": {
"source": {
}
}
}
}
}
]
}
]
}
]
}

1187
IMPORTANT
Hat.
TCP 1936 Metrics
UDP 4789 VXLAN
6081 Geneve
9100- 9101.
1188
7.11.15. Creating networking and load balancing components in Azure

You must configure networking and load balancing in Microsoft Azure for your OpenShift Container
Platform cluster to use. One way to create these components is to modify the provided Azure Resource
Manager (ARM) template.
NOTE
installation logs.
Prerequisites
Procedure

--parameters privateDNSZoneName="${CLUSTER_NAME}.${BASE_DOMAIN}" \ 1
1 The name of the private DNS zone.
3. Create an api DNS record in the public zone for the API public load balancer. The
public DNS zone exists.

b. Create the api DNS record in a new public zone:
1189

If you are adding the cluster to an existing public zone, you can create the api DNS record in
it instead:

{
"parameters" : {
"baseName" : {
"type" : "string",
"minLength" : 1,
"metadata" : {
}
},
"vnetBaseName": {
"type": "string",
"defaultValue": "",
"metadata" : {
}
},
"type" : "string",
"metadata" : {
"description" : "Name of the private DNS zone"
}
}
},
"variables" : {
"masterPublicIpAddressName" : "[concat(parameters('baseName'), '-master-pip')]",
"masterPublicIpAddressID" : "[resourceId('Microsoft.Network/publicIPAddresses',
1190
variables('masterPublicIpAddressName'))]",
"masterLoadBalancerID" : "[resourceId('Microsoft.Network/loadBalancers',
variables('masterLoadBalancerName'))]",
"internalLoadBalancerID" : "[resourceId('Microsoft.Network/loadBalancers',
variables('internalLoadBalancerName'))]",
"skuName": "Standard"
},
"resources" : [
{
"apiVersion" : "2018-12-01",
"name" : "[variables('masterPublicIpAddressName')]",
"sku": {
},
"properties" : {
"dnsSettings" : {
"domainNameLabel" : "[variables('masterPublicIpAddressName')]"
}
}
},
{
"apiVersion" : "2018-12-01",
"name" : "[variables('masterLoadBalancerName')]",
"sku": {
},
"dependsOn" : [
"[concat('Microsoft.Network/publicIPAddresses/', variables('masterPublicIpAddressName'))]"
],
"properties" : {
{
"name" : "public-lb-ip-v4",
"properties" : {
"publicIPAddress" : {
"id" : "[variables('masterPublicIpAddressID')]"
}
}
}
],
{
"name" : "[variables('masterLoadBalancerName')]"
}
],
{
"properties" : {
1191
"id" :"[concat(variables('masterLoadBalancerID'), '/frontendIPConfigurations/public-lb-ip-
v4')]"
},
"id" : "[concat(variables('masterLoadBalancerID'), '/backendAddressPools/',
},
"protocol" : "Tcp",
"probe" : {
"id" : "[concat(variables('masterLoadBalancerID'), '/probes/api-internal-probe')]"
}
}
}
],
"probes" : [
{
"properties" : {
"port" : 6443,
}
}
]
}
},
{
"apiVersion" : "2018-12-01",
"name" : "[variables('internalLoadBalancerName')]",
"sku": {
},
"properties" : {
{
"name" : "internal-lb-ip",
"properties" : {
"subnet" : {
},
"privateIPAddressVersion" : "IPv4"
}
}
],
{
1192
"name" : "internal-lb-backend"
}
],
{
"properties" : {
ip')]"
},
"protocol" : "Tcp",
backend')]"
},
"probe" : {
"id" : "[concat(variables('internalLoadBalancerID'), '/probes/api-internal-probe')]"
}
}
},
{
"name" : "sint",
"properties" : {
ip')]"
},
"protocol" : "Tcp",
backend')]"
},
"probe" : {
"id" : "[concat(variables('internalLoadBalancerID'), '/probes/sint-probe')]"
}
}
}
],
"probes" : [
{
"properties" : {
1193
"port" : 6443,
}
},
{
"name" : "sint-probe",
"properties" : {
"port" : 22623,
"requestPath": "/healthz",
}
}
]
}
},
{
"apiVersion": "2018-09-01",
"name": "[concat(parameters('privateDNSZoneName'), '/api')]",
"dependsOn" : [
],
"properties": {
"ttl": 60,
"aRecords": [
{
"ipv4Address": "
Address]"
}
]
}
},
{
"apiVersion": "2018-09-01",
"name": "[concat(parameters('privateDNSZoneName'), '/api-int')]",
"dependsOn" : [
],
"properties": {
"ttl": 60,
"aRecords": [
{
"ipv4Address": "
Address]"
}
]
}
1194
}
]
}
7.11.16. Creating the bootstrap machine in Azure

You must create the bootstrap machine in Microsoft Azure to use during OpenShift Container Platform
cluster initialization. One way to create this machine is to modify the provided Azure Resource Manager
(ARM) template.
NOTE
installation logs.
Prerequisites
Procedure

$ export BOOTSTRAP_IGNITION=`jq -rcnM --arg v "3.2.0" --arg url ${BOOTSTRAP_URL}

'{ignition:{version:$v,config:{replace:{source:$url}}}}' | base64 | tr -d '\n'`

1195
--parameter bootstrapVMSize="Standard_D4s_v3" 3
3 Optional: Specify the size of the bootstrap VM. Use a VM size compatible with your
set.
{
"parameters" : {
"baseName" : {
"type" : "string",
"minLength" : 1,
"metadata" : {
}
},
"vnetBaseName": {
"type": "string",
"defaultValue": "",
"metadata" : {
}
},
"bootstrapIgnition" : {
"type" : "string",
"minLength" : 1,
"metadata" : {
"description" : "Bootstrap ignition content for the bootstrap cluster"
}
},
"sshKeyData" : {
"metadata" : {
}
},
"bootstrapVMSize" : {
"type" : "string",
1196
"metadata" : {
"description" : "The size of the Bootstrap Virtual Machine"
}
},
"hyperVGen": {
"type": "string",
"metadata": {
},
"allowedValues": [
"V1",
"V2"
]
}
},
"variables" : {
"vmName" : "[concat(parameters('baseName'), '-bootstrap')]",
"nicName" : "[concat(variables('vmName'), '-nic')]",
gen2', ''))]",
"clusterNsgName" : "[concat(if(not(empty(parameters('vnetBaseName'))),
parameters('vnetBaseName'), parameters('baseName')), '-nsg')]",
"sshPublicIpAddressName" : "[concat(variables('vmName'), '-ssh-pip')]"
},
"resources" : [
{
"apiVersion" : "2018-12-01",
"name" : "[variables('sshPublicIpAddressName')]",
"sku": {
"name": "Standard"
},
"properties" : {
"dnsSettings" : {
"domainNameLabel" : "[variables('sshPublicIpAddressName')]"
}
}
},
1197
{
"apiVersion" : "2018-06-01",
"name" : "[variables('nicName')]",
"dependsOn" : [
"[resourceId('Microsoft.Network/publicIPAddresses', variables('sshPublicIpAddressName'))]"
],
"properties" : {
{
"properties" : {
"publicIPAddress": {
"id": "[resourceId('Microsoft.Network/publicIPAddresses',
variables('sshPublicIpAddressName'))]"
},
"subnet" : {
},
{
},
{
}
]
}
}
]
}
},
{
"apiVersion" : "2018-06-01",
"name" : "[variables('vmName')]",
"identity" : {
}
},
"dependsOn" : [
"[concat('Microsoft.Network/networkInterfaces/', variables('nicName'))]"
],
"properties" : {
1198
"vmSize" : "[parameters('bootstrapVMSize')]"
},
"osProfile" : {
"computerName" : "[variables('vmName')]",
"customData" : "[parameters('bootstrapIgnition')]",
}
},
"imageReference": {
},
"osDisk" : {
"name": "[concat(variables('vmName'),'_OSDisk')]",
"osType" : "Linux",
"managedDisk": {
},
"diskSizeGB" : 100
}
},
{
"id" : "[resourceId('Microsoft.Network/networkInterfaces', variables('nicName'))]"
}
]
}
}
},
{
"apiVersion" : "2018-06-01",
"type": "Microsoft.Network/networkSecurityGroups/securityRules",
"name" : "[concat(variables('clusterNsgName'), '/bootstrap_ssh_in')]",
"dependsOn" : [
"[resourceId('Microsoft.Compute/virtualMachines', variables('vmName'))]"
],
"properties": {
"protocol" : "Tcp",
"access" : "Allow",
"priority" : 100,
}
}
]
}
1199
7.11.17. Creating the control plane machines in Azure

You must create the control plane machines in Microsoft Azure for your cluster to use. One way to
create these machines is to modify the provided Azure Resource Manager (ARM) template.
NOTE
control plane node. To do this, modify a vendor’s Azure Resource Manager (ARM)
template by specifying each of your availability zones in the zones parameter of the
virtual machine resource.
Prerequisites
Procedure

--parameters masterVMSize="Standard_D8s_v3" 3
1 The Ignition content for the control plane nodes.
1200
3 Optional: Specify the size of the Control Plane VM. Use a VM size compatible with your
{
"parameters" : {
"baseName" : {
"type" : "string",
"minLength" : 1,
"metadata" : {
}
},
"vnetBaseName": {
"type": "string",
"defaultValue": "",
"metadata" : {
}
},
"masterIgnition" : {
"type" : "string",
"metadata" : {
"description" : "Ignition content for the master nodes"
}
},
"numberOfMasters" : {
"type" : "int",
"defaultValue" : 3,
"minValue" : 2,
"maxValue" : 30,
"metadata" : {
"description" : "Number of OpenShift masters to deploy"
}
},
"sshKeyData" : {
"metadata" : {
}
},
"type" : "string",
"defaultValue" : "",
1201
"metadata" : {
"description" : "unused"
}
},
"masterVMSize" : {
"type" : "string",
"metadata" : {
"description" : "The size of the Master Virtual Machines"
}
},
"diskSizeGB" : {
"type" : "int",
"defaultValue" : 1024,
"metadata" : {
"description" : "Size of the Master VM OS disk, in GB"
}
},
"hyperVGen": {
"type": "string",
"metadata": {
},
"allowedValues": [
"V1",
"V2"
]
}
},
"variables" : {
gen2', ''))]",
"copy" : [
{
"name" : "vmNames",
"count" : "[parameters('numberOfMasters')]",
"input" : "[concat(parameters('baseName'), '-master-', copyIndex('vmNames'))]"
}
]
},
"resources" : [
1202
{
"apiVersion" : "2018-06-01",
"copy" : {
"name" : "nicCopy",
},
"properties" : {
{
"properties" : {
"subnet" : {
},
{
},
{
}
]
}
}
]
}
},
{
"apiVersion" : "2018-06-01",
"copy" : {
"name" : "vmCopy",
},
"identity" : {
}
},
"dependsOn" : [
"[concat('Microsoft.Network/networkInterfaces/', concat(variables('vmNames')[copyIndex()], '-
nic'))]"
],
1203
"properties" : {
"vmSize" : "[parameters('masterVMSize')]"
},
"osProfile" : {
"customData" : "[parameters('masterIgnition')]",
}
},
"imageReference": {
},
"osDisk" : {
"name": "[concat(variables('vmNames')[copyIndex()], '_OSDisk')]",
"osType" : "Linux",
"caching": "ReadOnly",
"writeAcceleratorEnabled": false,
"managedDisk": {
},
"diskSizeGB" : "[parameters('diskSizeGB')]"
}
},
{
"id" : "[resourceId('Microsoft.Network/networkInterfaces', concat(variables('vmNames')
[copyIndex()], '-nic'))]",
"properties": {
"primary": false
}
}
]
}
}
}
]
}
After you create all of the required infrastructure in Microsoft Azure, wait for the bootstrap process to
complete on the machines that you provisioned by using the Ignition config files that you generated
with the installation program.
Prerequisites
1204
Procedure

--log-level info 2

wait --yes
wait
ssh-pip
NOTE
7.11.19. Creating additional worker machines in Azure

You can create worker machines in Microsoft Azure for your cluster to use by launching individual
instances discretely or by automated processes outside the cluster, such as auto scaling groups. You can
also take advantage of the built-in cluster scaling mechanisms and the machine API in OpenShift
1205
Container Platform.
NOTE
NOTE
control plane node. To do this, modify a vendor’s ARM template by specifying each of
your availability zones in the zones parameter of the virtual machine resource.
Prerequisites
Procedure

1206
--parameters nodeVMSize="Standard_D4s_v3" 3
3 Optional: Specify the size of the compute node VM. Use a VM size compatible with your
set.
{
"parameters" : {
"baseName" : {
"type" : "string",
"minLength" : 1,
"metadata" : {
}
},
"vnetBaseName": {
"type": "string",
"defaultValue": "",
"metadata" : {
}
},
"workerIgnition" : {
"type" : "string",
"metadata" : {
"description" : "Ignition content for the worker nodes"
}
},
"numberOfNodes" : {
"type" : "int",
"defaultValue" : 3,
"minValue" : 2,
"maxValue" : 30,
"metadata" : {
"description" : "Number of OpenShift compute nodes to deploy"
}
},
"sshKeyData" : {
1207
"metadata" : {
}
},
"nodeVMSize" : {
"type" : "string",
"metadata" : {
"description" : "The size of the each Node Virtual Machine"
}
},
"hyperVGen": {
"type": "string",
"metadata": {
},
"allowedValues": [
"V1",
"V2"
]
}
},
"variables" : {
"nodeSubnetName" : "[concat(if(not(empty(parameters('vnetBaseName'))),
parameters('vnetBaseName'), parameters('baseName')), '-worker-subnet')]",
"nodeSubnetRef" : "[concat(variables('virtualNetworkID'), '/subnets/',
variables('nodeSubnetName'))]",
"infraLoadBalancerName" : "[parameters('baseName')]",
"sshKeyPath" : "/home/capi/.ssh/authorized_keys",
gen2', ''))]",
"copy" : [
{
"name" : "vmNames",
"count" : "[parameters('numberOfNodes')]",
"input" : "[concat(parameters('baseName'), '-worker-', variables('location'), '-',
copyIndex('vmNames', 1))]"
}
]
},
"resources" : [
{
"apiVersion" : "2019-05-01",
"name" : "[concat('node', copyIndex())]",
"type" : "Microsoft.Resources/deployments",
"copy" : {
"name" : "nodeCopy",
1208
},
"properties" : {
"mode" : "Incremental",
"template" : {
"$schema" : "http://schema.management.azure.com/schemas/2015-01-
"resources" : [
{
"apiVersion" : "2018-06-01",
"properties" : {
{
"properties" : {
"subnet" : {
"id" : "[variables('nodeSubnetRef')]"
}
}
}
]
}
},
{
"apiVersion" : "2018-06-01",
"tags" : {
"kubernetes.io-cluster-ffranzupi": "owned"
},
"identity" : {
}
},
"dependsOn" : [
],
"properties" : {
"vmSize" : "[parameters('nodeVMSize')]"
},
"osProfile" : {
"adminUsername" : "capi",
"customData" : "[parameters('workerIgnition')]",
1209
}
},
"imageReference": {
},
"osDisk" : {
"name": "[concat(variables('vmNames')[copyIndex()],'_OSDisk')]",
"osType" : "Linux",
"managedDisk": {
},
"diskSizeGB": 128
}
},
{
"id" : "[resourceId('Microsoft.Network/networkInterfaces',
concat(variables('vmNames')[copyIndex()], '-nic'))]",
"properties": {
"primary": true
}
}
]
}
}
}
]
}
}
}
]
}

IMPORTANT

Procedure
1210
Portal.
$ tar xvf <file>

$ echo $PATH
Verification
$ oc <command>

Procedure
Portal.

C:\> path
Verification
C:\> oc <command>

Procedure
1211
Portal.
NOTE

$ echo $PATH
Verification
$ oc <command>

Prerequisites
Procedure
$ oc whoami
Example output
1212
system:admin

Prerequisites
Procedure
$ oc get nodes
Example output

NOTE
$ oc get csr
Example output

...
list.
NOTE
1213
NOTE
NOTE
of the node.

NOTE
$ oc get csr
Example output

Pending
Pending
...
1214

$ oc get nodes
Example output

NOTE

Prerequisites
You deployed an OpenShift Container Platform cluster on Microsoft Azure by using

1215
Procedure
Example output

80:32288/TCP,443:31215/TCP 20

3. Add a *.apps record to the public DNS zone.
a. If you are adding this cluster to a new public zone, run:

b. If you are adding this cluster to an already existing public zone, run:

4. Add a *.apps record to the private DNS zone:
a. Create a *.apps record by using the following command:
$ az network private-dns record-set a create -g ${RESOURCE_GROUP} -z

${CLUSTER_NAME}.${BASE_DOMAIN} -n *.apps --ttl 300
b. Add the *.apps record to the private DNS zone by using the following command:
$ az network private-dns record-set a add-record -g ${RESOURCE_GROUP} -z

${CLUSTER_NAME}.${BASE_DOMAIN} -n *.apps -a ${PUBLIC_IP_ROUTER}

{end}' routes
Example output
1216
7.11.24. Completing an Azure installation on user-provisioned infrastructure

After you start the OpenShift Container Platform installation on Microsoft Azure user-provisioned
infrastructure, you can monitor the cluster events until the cluster is ready.
Prerequisites
Azure infrastructure.
Procedure
Example output
IMPORTANT
installation.

1217
7.12. INSTALLING A CLUSTER ON AZURE IN A RESTRICTED NETWORK

In OpenShift Container Platform version 4.15, you can install a cluster on Microsoft Azure in a restricted
network by creating an internal mirror of the installation release content on an existing Azure Virtual
Network (VNet).
IMPORTANT
You can install an OpenShift Container Platform cluster by using mirrored installation
release content, but your cluster requires internet access to use the Azure APIs.
processes.
users.
region to deploy the cluster.
IMPORTANT
You have an existing VNet in Azure. While installing a cluster in a restricted network that uses
installer-provisioned infrastructure, you cannot use the installer-provisioned VNet. You must
use a user-provisioned VNet that satisfies one of the following requirements:
The VNet contains the mirror registry
The VNet has firewall rules or a peering connection to access the mirror registry hosted
elsewhere
encryption.
1218
your restrictions.
requests.
registry mirror.
defined routing.
Restricted cluster with Azure Firewall
1219
You can use Azure Firewall to restrict the outbound routing for the Virtual Network (VNet) that is used
to install the OpenShift Container Platform cluster. For more information, see providing user-defined
routing with Azure Firewall. You can create a OpenShift Container Platform cluster in a restricted
network by using VNet with Azure Firewall and configuring the user-defined routing.
IMPORTANT
If you are using Azure Firewall for restricting internet access, you must set the publish
field to Internal in the install-config.yaml file. This is because Azure Firewall does not
work properly with Azure public load balancers.
rules.
Subnets
Route tables
VNets
NOTE
cluster.
1220
NOTE
data:
machines.
NOTE
IMPORTANT
1221

* Allows connections to Azure APIs. You must set a x x

[1]
Destination Service Tag to AzureCloud .
* Denies connections to the internet. You must set a x x

[1]
Destination Service Tag to Internet.
IMPORTANT
1222


authentication.
user.
IMPORTANT
NOTE
Procedure
1223
NOTE
ecdsa algorithm.
NOTE

task:
Example output
Agent pid 31874
NOTE

1224
Example output
Next steps
program.

Prerequisites
creation.
prerequisites:
Procedure
1225
command:
NOTE


values.
1226
IMPORTANT
ix. Paste the pull secret from Red Hat OpenShift Cluster Manager .

c. Define the network and subnets for the VNet to install the cluster under the platform.azure
field:
virtualNetwork: <vnet> 2
controlPlaneSubnet: <control_plane_subnet> 3
computeSubnet: <compute_subnet> 4
1 Replace <vnet_resource_group> with the resource group name that contains the
existing virtual network (VNet).
2 Replace <vnet> with the existing virtual network name.
1227
3 Replace <control_plane_subnet> with the existing subnet name to deploy the control
plane machines.
4 Replace <compute_subnet> with the existing subnet name to deploy compute

machines.
- mirrors:
- mirrors:
creation.
publish: Internal
IMPORTANT
Azure Firewall does not work seamlessly with Azure Public Load balancers.
Thus, when using Azure Firewall for restricting internet access, the publish
field in install-config.yaml should be set to Internal.
IMPORTANT


platform.
1228

System Per Second
(IOPS)[2]

8.6 and later
[3]
IMPORTANT
to true.
c4.*
c5.*
c5a.*
i3.*
m4.*
1229
m5.*
m5a.*
m6a.*
m6i.*
r4.*
r5.*
r5a.*
r6i.*
t3.*
t3a.*
Platform.
c6g.*
m6g.*
these features.
IMPORTANT
Prerequisites
1230
Procedure
following stanza:
controlPlane: 1
platform:
azure:
settings:
trustedLaunch:
uefiSettings:

boot.
IMPORTANT

DCasv5-series
DCadsv5-series
ECasv5-series
ECadsv5-series
IMPORTANT
1231
IMPORTANT
Prerequisites
Procedure
following stanza:
controlPlane: 1
platform:
azure:
settings:
confidentialVM:
uefiSettings:
osDisk:
securityProfile:

boot.
IMPORTANT
apiVersion: v1
controlPlane: 2
1232
name: master
platform:
azure:
osDisk:
diskSizeGB: 1024 5
diskEncryptionSet:
osImage:
replicas: 3
compute: 6
name: worker
platform:
azure:
osDisk:
diskSizeGB: 512 8
diskEncryptionSet:
osImage:
zones: 9
- "1"
- "2"
- "3"
replicas: 5
metadata:
networking:
clusterNetwork:
- cidr: 10.128.0.0/14
hostPrefix: 23
machineNetwork:
- cidr: 10.0.0.0/16
serviceNetwork:
1233
- 172.30.0.0/16
platform:
azure:
osImage: 12
fips: false 22
- mirrors:
- mirrors:
value.

IMPORTANT
1234
IMPORTANT

nodes is 1024 GB.
value.
20 When using Azure Firewall to restrict Internet access, you must configure outbound routing to send
traffic through the Azure Firewall. Configuring user-defined routing prevents exposing external
endpoints in your cluster.
IMPORTANT
system in FIPS mode. The use of FIPS validated or Modules In Process
cryptographic libraries is only supported on OpenShift Container Platform
deployments on the x86_64, ppc64le, and s390x architectures.
NOTE
1235
NOTE
process uses.
repository.
26 How to publish the user-facing endpoints of your cluster. When using Azure Firewall to restrict
Internet access, set publish to Internal to deploy a private cluster. The user-facing endpoints then
cannot be accessed from the internet. The default value is External.
Prerequisites
NOTE
(169.254.169.254).
Procedure
apiVersion: v1
proxy:
1236
must be http.

destinations.
NOTE
NOTE
NOTE
created.
1237
IMPORTANT

Procedure
Portal.
$ tar xvf <file>

$ echo $PATH
Verification
$ oc <command>

Procedure
Portal.

1238
C:\> path
Verification
C:\> oc <command>

Procedure
Portal.
NOTE

$ echo $PATH
Verification
$ oc <command>

project
credentials.
1239
namespace.
Procedure
apiVersion: v1
# ...
command:

--included \ 1
1240
metadata:
...
spec:
providerSpec:
roleBindings:
- role: Contributor
...
metadata:
...
spec:
providerSpec:
roleBindings:
- role: Contributor
...
secretRef:
...
apiVersion: v1
kind: Secret
metadata:
data:
IMPORTANT
1241
IMPORTANT
NOTE
Prerequisites
1242
Procedure

NOTE

$ chmod 775 ccoctl
Verification
$ ccoctl --help
1243
Usage:
ccoctl [command]
Available Commands:
Flags:
NOTE
Prerequisites
You must have:
Procedure

1244
--included \ 1
NOTE
$ az login
command:

zone.
NOTE
1245
NOTE
preview parameter.
Verification
Example output
Prerequisites
utility.
Procedure
1246
apiVersion: v1
# ...
shown:
apiVersion: v1
# ...
platform:
azure:
# ...
command:

IMPORTANT
Prerequisites
1247
cluster.
Procedure
deployment:

--log-level=info 2

config.yaml file.
Verification
IMPORTANT
Example output
...
IMPORTANT
1248
IMPORTANT

Prerequisites
Procedure
$ oc whoami
Example output
system:admin

1249
7.12.12. Next steps

7.13. INSTALLING A THREE-NODE CLUSTER ON AZURE

In OpenShift Container Platform version 4.15, you can install a three-node cluster on Microsoft Azure. A
three-node cluster consists of three control plane machines, which also act as compute machines. This
type of cluster provides a smaller, more resource efficient cluster, for cluster administrators and
developers to use for testing, development, and production.
NOTE
Deploying a three-node cluster using an Azure Marketplace image is not supported.

NOTE
Prerequisites
Procedure
apiVersion: v1
compute:
- name: worker
1250
platform: {}
replicas: 0
# ...
Azure using ARM templates".
kind: Scheduler
metadata:
name: cluster
spec:
policy:
name: ""
status: {}
7.13.2. Next steps

Installing a cluster on Azure with customizations
Installing a cluster on Azure using ARM templates
7.14. UNINSTALLING A CLUSTER ON AZURE

You can remove a cluster that you deployed to Microsoft Azure.

NOTE
Prerequisites
Procedure
1251
Procedure

NOTE
7.14.2. Deleting Microsoft Azure resources with the Cloud Credential Operator
utility
outside the cluster, you can use the CCO utility (ccoctl) to remove the Microsoft Azure (Azure)
Prerequisites
Uninstall an OpenShift Container Platform cluster on Azure that uses short-term credentials.
Procedure
Delete the Azure resources that ccoctl created by running the following command:
$ ccoctl azure delete \

--name=<name> \ 1
--delete-oidc-resource-group
2 <azure_region> is the Azure region in which to delete cloud resources.
3 <azure_subscription_id> is the Azure subscription ID for which to delete cloud resources.
Verification
1252
To verify that the resources are deleted, query Azure. For more information, refer to Azure
documentation.
7.15. INSTALLATION CONFIGURATION PARAMETERS FOR AZURE

Before you deploy an OpenShift Container Platform cluster on Microsoft Azure, you provide parameters
to customize your cluster and the platform that hosts it. When you create the install-config.yaml file,
you provide values for the required parameters through the command line. You can then modify the
install-config.yaml file to customize your cluster further.
7.15.1. Available installation configuration parameters for Azure

The following tables specify the required, optional, and Azure-specific installation configuration
NOTE

versions.

combination of the
baseDomain and
<metadata.name>.

consumed.
1253
name: subdomains of
{{.metadata.name}}.
{{.baseDomain}}.

alibabacloud, aws,
ibmcloud, nutanix,

"auth":"b3Blb=",
}
}
}
NOTE
1254

NOTE
You cannot modify

by the networking
object after
installation.


networking:
clusterNetwork:
- cidr: 10.128.0.0/14
hostPrefix: 23

An IPv4 network.

23 provides 510 (2^(32 - 23) - 2) pod
IP addresses.
serviceNetwork:
The OVN-Kubernetes network plugins networking:
serviceNetwork:
for the service network. - 172.30.0.0/16
1255

networking:
machineNetwork: If you specify multiple IP address networking:
machineNetwork:
- cidr: 10.0.0.0/16

NOTE
the CIDR that the
in.



et:
1256


section.


1257

cores.
IMPORTANT
If you disable
simultaneous
that your capacity
the dramatically
decreased machine
performance.

name:
value.


1258


cores.
IMPORTANT
If you disable
simultaneous
that your capacity
the dramatically
decreased machine
performance.

name:

parameter value.

replicas:
1259


IMPORTANT
To enable FIPS mode

must run the
from a Red Hat
Enterprise Linux
(RHEL) computer
in FIPS mode. For
more information
about configuring
FIPS mode on RHEL,
see Installing the
Enterprise Linux
(RHEL) or Red Hat
Enterprise Linux
CoreOS (RHCOS)
OpenShift Container
Platform core
components use the
RHEL cryptographic
libraries that have
been submitted to
NIST for FIPS 140-
only the x86_64,
ppc64le, and s390x
architectures.
NOTE
1260
NOTE
File storage, you
cannot enable FIPS
mode.



ces:
mirrors:
How to publish or expose the user- Internal , External, or Mixed. To

publish: facing endpoints of your cluster, such deploy a private cluster, which cannot
as the Kubernetes API, OpenShift be accessed from the internet, set
value is External. To deploy a cluster
where the API and the ingress server
have different publishing strategies,
set publish to Mixed and use the
operatorPublishingStrategy
parameter.
1261

NOTE
For production
OpenShift Container
which you want to
SSH key that your
ssh-agent process
uses.
IMPORTANT

7.15.1.4. Additional Azure configuration parameters
Additional Azure configuration parameters are described in the following table.
NOTE
Table 7.31. Additional Azure parameters
1262
Enables host-level encryption for true or false. The default is false.

compute: compute machines. You can enable
platform: this encryption alongside user-
azure: managed server-side encryption. This
feature encrypts temporary,
encryptionAtHost: ephemeral, cached and un-managed
disks on the VM host. This is not a
prerequisite for user-managed server-
side encryption.
The Azure disk size for the VM. Integer that represents the size of the
compute: disk in GB. The default is 128.
platform:
azure:
osDisk:
diskSizeGB:
Defines the type of disk. standard_LRS , premium_LRS, or

compute: standardSSD_LRS. The default is
platform: premium_LRS.
azure:
osDisk:
diskType:
Enables the use of Azure ultra disks for Enabled, Disabled. The default is
compute: persistent storage on compute nodes. Disabled.
platform: This requires that your Azure region
azure: and zone have ultra disks available.
ultraSSDCapability:
The name of the Azure resource group String, for example

compute: that contains the disk encryption set production_encryption_resource
platform: from the installation prerequisites. This _group.
azure: resource group should be different
osDisk: from the resource group where you
install the cluster to avoid deleting
diskEncryptionSet: your Azure encryption key when the
cluster is destroyed. This value is only
resourceGroup: necessary if you intend to install the
cluster with user-managed disk
encryption.
1263
The name of the disk encryption set String, for example

compute: that contains the encryption key from production_disk_encryption_set.
platform: the installation prerequisites.
azure:
osDisk:
diskEncryptionSet:
name:
Defines the Azure subscription of the String, in the format 00000000-0000-

compute: disk encryption set where the disk 0000-0000-000000000000 .
platform: encryption set resides. This secondary
azure: disk encryption set is used to encrypt
osDisk: compute machines.
diskEncryptionSet:
subscriptionId:
Optional. By default, the installation String. The name of the image

compute: program downloads and installs the publisher.
platform: Red Hat Enterprise Linux CoreOS
azure: (RHCOS) image that is used to boot
osImage: compute machines. You can override
publisher: the default behavior by using a custom
RHCOS image that is available from
the Azure Marketplace. The installation
program uses this image for compute
machines only.
The name of Azure Marketplace offer String. The name of the image offer.
compute: that is associated with the custom
platform: RHCOS image. If you use
azure: compute.platform.azure.osImage
osImage: .publisher , this field is required.
offer:
An instance of the Azure Marketplace String. The SKU of the image offer.
compute: offer. If you use
platform: compute.platform.azure.osImage
azure: .publisher , this field is required.
osImage:
sku:
1264
The version number of the image SKU. String. The version of the image to use.
compute: If you use
platform: compute.platform.azure.osImage
azure: .publisher , this field is required.
osImage:
version:
Enables accelerated networking. Accelerated or Basic.

compute: Accelerated networking enables single
platform: root I/O virtualization (SR-IOV) to a
azure: VM, improving its networking
performance. If instance type of
vmNetworkingType compute machines support
: Accelerated networking, by default,
the installer enables Accelerated
networking, otherwise the default
networking type is Basic.
Defines the Azure instance type for String

compute: compute machines.
platform:
azure:
type:
The availability zones where the String list

compute: installation program creates compute
platform: machines.
azure:
zones:
Enables confidential VMs or trusted ConfidentialVM or

compute: launch for compute nodes. This option TrustedLaunch.
platform: is not enabled by default.
azure:
settings:
securityType:
Enables secure boot on compute Enabled or Disabled. The default is

compute: nodes if you are using confidential Disabled.
platform: VMs.
azure:
settings:
confidentialVM:
uefiSettings:
secureBoot:
1265
Enables the virtualized Trusted Enabled or Disabled. The default is

compute: Platform Module (vTPM) feature on Disabled.
platform: compute nodes if you are using
azure: confidential VMs.
settings:
confidentialVM:
uefiSettings:
virtualizedTrustedPl
atformModule:
Enables secure boot on compute Enabled or Disabled. The default is

compute: nodes if you are using trusted launch. Disabled.
platform:
azure:
settings:
trustedLaunch:
uefiSettings:
secureBoot:
Enables the vTPM feature on compute Enabled or Disabled. The default is

compute: nodes if you are using trusted launch. Disabled.
platform:
azure:
settings:
trustedLaunch:
uefiSettings:
atformModule:
Enables the encryption of the virtual VMGuestStateOnly is the only

compute: machine guest state for compute supported value.
platform: nodes. This parameter can only be
azure: used if you use Confidential VMs.
osDisk:
securityProfile:
securityEncryptionT
ype:
1266

controlPlane: launch for control plane nodes. This TrustedLaunch.
platform: option is not enabled by default.
azure:
settings:
securityType:
Enables secure boot on control plane Enabled or Disabled. The default is

controlPlane: nodes if you are using confidential Disabled.
platform: VMs.
azure:
settings:
confidentialVM:
uefiSettings:
secureBoot:
Enables the vTPM feature on control Enabled or Disabled. The default is

controlPlane: plane nodes if you are using Disabled.
platform: confidential VMs.
azure:
settings:
confidentialVM:
uefiSettings:
atformModule:
Enables secure boot on control plane Enabled or Disabled. The default is

controlPlane: nodes if you are using trusted launch. Disabled.
platform:
azure:
settings:
trustedLaunch:
uefiSettings:
secureBoot:
1267
Enables the vTPM feature on control Enabled or Disabled. The default is

controlPlane: plane nodes if you are using trusted Disabled.
platform: launch.
azure:
settings:
trustedLaunch:
uefiSettings:
atformModule:

controlPlane: machine guest state for control plane supported value.
platform: nodes. This parameter can only be
azure: used if you use Confidential VMs.
osDisk:
securityProfile:
securityEncryptionT
ype:
Defines the Azure instance type for String

controlPlane: control plane machines.
platform:
azure:
type:
The availability zones where the String list

controlPlane: installation program creates control
platform: plane machines.
azure:
zones:

platform: launch for all nodes. This option is not TrustedLaunch.
azure: enabled by default.
defaultMachinePlatf
orm:
settings:
securityType:
1268
Enables secure boot on all nodes if you Enabled or Disabled. The default is
platform: are using confidential VMs. Disabled.
azure:
defaultMachinePlatf
orm:
settings:
confidentialVM:
uefiSettings:
secureBoot:
Enables the virtualized Trusted Enabled or Disabled. The default is

platform: Platform Module (vTPM) feature on all Disabled.
azure: nodes if you are using confidential
VMs.
defaultMachinePlatf
orm:
settings:
confidentialVM:
uefiSettings:
atformModule:
Enables secure boot on all nodes if you Enabled or Disabled. The default is
platform: are using trusted launch. Disabled.
azure:
defaultMachinePlatf
orm:
settings:
trustedLaunch:
uefiSettings:
secureBoot:
1269
Enables the vTPM feature on all nodes Enabled or Disabled. The default is
platform: if you are using trusted launch. Disabled.
azure:
defaultMachinePlatf
orm:
settings:
trustedLaunch:
uefiSettings:
atformModule:

platform: machine guest state for all nodes. This supported value.
azure: parameter can only be used if you use
Confidential VMs.
defaultMachinePlatf
orm:
osDisk:
securityProfile:
securityEncryptionT
ype:

platform: compute machines. You can enable
azure: this encryption alongside user-
managed server-side encryption. This
defaultMachinePlatf feature encrypts temporary,
orm: ephemeral, cached, and un-managed
disks on the VM host. This parameter is
encryptionAtHost: not a prerequisite for user-managed
server-side encryption.
The name of the disk encryption set String, for example,

platform: that contains the encryption key from production_disk_encryption_set.
azure: the installation prerequisites.
defaultMachinePlatf
orm:
osDisk:
diskEncryptionSet:
name:
1270
The name of the Azure resource group String, for example,

platform: that contains the disk encryption set production_encryption_resource
azure: from the installation prerequisites. To _group.
avoid deleting your Azure encryption
defaultMachinePlatf key when the cluster is destroyed, this
orm: resource group must be different from
osDisk: the resource group where you install
the cluster. This value is necessary only
diskEncryptionSet: if you intend to install the cluster with
user-managed disk encryption.
resourceGroup:

platform: disk encryption set where the disk 0000-0000-000000000000 .
azure: encryption set resides. This secondary
disk encryption set is used to encrypt
defaultMachinePlatf compute machines.
orm:
osDisk:
diskEncryptionSet:
subscriptionId:
platform: disk in GB. The default is 128.
azure:
defaultMachinePlatf
orm:
osDisk:
diskSizeGB:
Defines the type of disk. premium_LRS or

platform: standardSSD_LRS. The default is
azure: premium_LRS.
defaultMachinePlatf
orm:
osDisk:
diskType:
1271

platform: program downloads and installs the publisher.
azure: Red Hat Enterprise Linux CoreOS
(RHCOS) image that is used to boot
defaultMachinePlatf control plane and compute machines.
orm: You can override the default behavior
osImage: by using a custom RHCOS image that
publisher: is available from the Azure
Marketplace. The installation program
uses this image for both types of
machines.
platform: that is associated with the custom
azure: RHCOS image. If you use
platform.azure.defaultMachinePl
defaultMachinePlatf atform.osImage.publisher, this
orm: field is required.
osImage:
offer:
platform: offer. If you use
azure: platform.azure.defaultMachinePl
atform.osImage.publisher, this
defaultMachinePlatf field is required.
orm:
osImage:
sku:
platform: If you use
azure: platform.azure.defaultMachinePl
atform.osImage.publisher, this
defaultMachinePlatf field is required.
orm:
osImage:
version:
The Azure instance type for control The Azure instance type.
platform: plane and compute machines.
azure:
defaultMachinePlatf
orm:
type:
1272
The availability zones where the String list.

platform: installation program creates compute
azure: and control plane machines.
defaultMachinePlatf
orm:
zones:

controlPlane: control plane machines. You can
platform: enable this encryption alongside user-
azure: managed server-side encryption. This
feature encrypts temporary,
encryptionAtHost: ephemeral, cached and un-managed
disks on the VM host. This is not a
prerequisite for user-managed server-
side encryption.
The name of the Azure resource group String, for example

controlPlane: that contains the disk encryption set production_encryption_resource
platform: from the installation prerequisites. This _group.
azure: resource group should be different
osDisk: from the resource group where you
install the cluster to avoid deleting
diskEncryptionSet: your Azure encryption key when the
cluster is destroyed. This value is only
resourceGroup: necessary if you intend to install the
cluster with user-managed disk
encryption.
The name of the disk encryption set String, for example

controlPlane: that contains the encryption key from production_disk_encryption_set.
platform: the installation prerequisites.
azure:
osDisk:
diskEncryptionSet:
name:
1273

controlPlane: disk encryption set where the disk 0000-0000-000000000000 .
platform: encryption set resides. This secondary
azure: disk encryption set is used to encrypt
osDisk: control plane machines.
diskEncryptionSet:
subscriptionId:
controlPlane: disk in GB. The default is 1024.
platform:
azure:
osDisk:
diskSizeGB:
Defines the type of disk. premium_LRS or

controlPlane: standardSSD_LRS. The default is
platform: premium_LRS.
azure:
osDisk:
diskType:

controlPlane: program downloads and installs the publisher.
platform: Red Hat Enterprise Linux CoreOS
azure: (RHCOS) image that is used to boot
osImage: control plane machines. You can
publisher: override the default behavior by using
a custom RHCOS image that is
available from the Azure Marketplace.
The installation program uses this
image for control plane machines only.
controlPlane: that is associated with the custom
platform: RHCOS image. If you use
azure: controlPlane.platform.azure.osIm
osImage: age.publisher, this field is required.
offer:
controlPlane: offer. If you use
platform: controlPlane.platform.azure.osIm
azure: age.publisher, this field is required.
osImage:
sku:
1274
controlPlane: If you use
platform: controlPlane.platform.azure.osIm
azure: age.publisher, this field is required.
osImage:
version:
controlPlane: persistent storage on control plane Disabled.
platform: machines. This requires that your
azure: Azure region and zone have ultra disks
available.
ultraSSDCapability:
Enables accelerated networking. Accelerated or Basic.

controlPlane: Accelerated networking enables single
platform: root I/O virtualization (SR-IOV) to a
azure: VM, improving its networking
performance. If instance type of
vmNetworkingType control plane machines support
: Accelerated networking, by default,
the installer enables Accelerated
networking, otherwise the default
networking type is Basic.
The name of the resource group that String, for example

platform: contains the DNS zone for your base production_cluster .
azure: domain.
baseDomainResou
rceGroupName:
The name of an already existing String, for example

platform: resource group to install your cluster existing_resource_group.
azure: to. This resource group must be empty
and only used for this specific cluster;
resourceGroupNa the cluster components assume
me: ownership of all resources in the
resource group. If you limit the service
principal scope of the installation
program to this resource group, you
must ensure all other resources used
by the installation program in your
environment have the necessary
permissions, such as the public DNS
zone and virtual network. Destroying
the cluster by using the installation
program deletes this resource group.
1275
The outbound routing strategy used to LoadBalancer ,

platform: connect your cluster to the internet. If UserDefinedRouting, or
azure: you are using user-defined routing, NatGateway. The default is
outboundType: you must have pre-existing networking LoadBalancer .
available where the outbound routing
has already been configured prior to
installing a cluster. The installation
program is not responsible for
configuring user-defined routing. If
you specify the NatGateway routing
strategy, the installation program will
only create one NAT gateway. If you
specify the NatGateway routing
strategy, your account must have the
Microsoft.Network/natGateways/r
ead and
Microsoft.Network/natGateways/
write permissions.
IMPORTANT
NatGateway is a
Technology Preview
feature only.
Technology Preview
features are not
supported with Red
Hat production service
level agreements
(SLAs) and might not
be functionally
complete. Red Hat
does not recommend
using them in
production. These
features provide early
access to upcoming
product features,
enabling customers to
test functionality and
provide feedback
during the
development process.
For more information

about the support
scope of Red Hat
Technology Preview
features, see
Technology Preview
Features Support
Scope.
1276
The name of the Azure region that Any valid region name, such as
platform: hosts your cluster. centralus.
azure:
region:
List of availability zones to place List of zones, for example ["1", "2",
platform: machines in. For high availability, "3"].
azure: specify at least two zones.
zone:
Specifies the name of the key vault String.

platform: that contains the encryption key that is
azure: used to encrypt Azure storage.
customerManaged
Key:
keyVault:
name:
Specifies the name of the user- String.

platform: managed encryption key that is used
azure: to encrypt Azure storage.
customerManaged
Key:
keyVault:
keyName:
Specifies the name of the resource String.

platform: group that contains the key vault and
azure: managed identity.
customerManaged
Key:
keyVault:
resourceGroup:
Specifies the subscription ID that is String, in the format 00000000-0000-

platform: associated with the key vault. 0000-0000-000000000000 .
azure:
customerManaged
Key:
keyVault:
subscriptionId:
1277
Specifies the name of the user- String.

platform: assigned managed identity that resides
azure: in the resource group with the key
vault and has access to the user-
customerManaged managed key.
Key:
userAssignedIdentit
yKey:
platform: persistent storage on control plane Disabled.
azure: and compute machines. This requires
that your Azure region and zone have
defaultMachinePlatf ultra disks available.
orm:
ultraSSDCapability:
The name of the resource group that String.

platform: contains the existing VNet that you
azure: want to deploy your cluster to. This
name cannot be the same as the
networkResourceG platform.azure.baseDomainReso
roupName: urceGroupName.
The name of the existing VNet that String.

platform: you want to deploy your cluster to.
azure:
virtualNetwork:
The name of the existing subnet in Valid CIDR, for example 10.0.0.0/16.
platform: your VNet that you want to deploy
azure: your control plane machines to.
controlPlaneSubnet
:
The name of the existing subnet in Valid CIDR, for example 10.0.0.0/16.
platform: your VNet that you want to deploy
azure: your compute machines to.
computeSubnet:
1278
The name of the Azure cloud Any valid cloud environment, such as
platform: environment that is used to configure AzurePublicCloud or
azure: the Azure SDK with the appropriate AzureUSGovernmentCloud .
cloudName: Azure API endpoints. If empty, the
default value AzurePublicCloud is
used.
Enables accelerated networking. Accelerated or Basic. If instance

platform: Accelerated networking enables single type of control plane and compute
azure: root I/O virtualization (SR-IOV) to a machines support Accelerated
VM, improving its networking networking, by default, the installer
defaultMachinePlatf performance. enables Accelerated networking,
orm: otherwise the default networking type
is Basic.
vmNetworkingType
:
Determines whether the load balancers External or Internal . The default

operatorPublishing that service the API are public or value is External.
Strategy: private. Set this parameter to Internal
apiserver: to prevent the API server from being
accessible outside of your VNet. Set
this parameter to External to make
the API server accessible outside of
your VNet. If you set this parameter,
you must set the publish parameter
to Mixed.
Determines whether the DNS External or Internal . The default

operatorPublishing resources that the cluster creates for value is External.
Strategy: ingress traffic are publicly visible. Set
ingress: this parameter to Internal to prevent
the ingress VIP from being publicly
accessible. Set this parameter to
External to make the ingress VIP
publicly accessible. If you set this
parameter, you must set the publish
parameter to Mixed.
NOTE
You cannot customize Azure Availability Zones or Use tags to organize your Azure
resources with an Azure cluster.
1279
CHAPTER 8. INSTALLING ON AZURE STACK HUB
8.1. PREPARING TO INSTALL ON AZURE STACK HUB
processes.
users.
You have installed Azure Stack Hub version 2008 or later.
8.1.2. Requirements for installing OpenShift Container Platform on Azure Stack Hub
Before installing OpenShift Container Platform on Microsoft Azure Stack Hub, you must configure an
Azure account.
See Configuring an Azure Stack Hub account for details about account configuration, account limits,
DNS zone configuration, required roles, and creating service principals.
8.1.3. Choosing a method to install OpenShift Container Platform on Azure Stack

Hub
You can install a cluster on Azure Stack Hub infrastructure that is provisioned by the OpenShift
Container Platform installation program, by using the following method:
Installing a cluster on Azure Stack Hub with an installer-provisioned infrastructure : You can
install OpenShift Container Platform on Azure Stack Hub infrastructure that is provisioned by
the OpenShift Container Platform installation program.
You can install a cluster on Azure Stack Hub infrastructure that you provision, by using the following
method:
Installing a cluster on Azure Stack Hub using ARM templates: You can install OpenShift
Container Platform on Azure Stack Hub by using infrastructure that you provide. You can use
the provided Azure Resource Manager (ARM) templates to assist with an installation.
8.1.4. Next steps
1280
Configuring an Azure Stack Hub account
8.2. CONFIGURING AN AZURE STACK HUB ACCOUNT

Before you can install OpenShift Container Platform, you must configure a Microsoft Azure account.
IMPORTANT
documentation.
8.2.1. Azure Stack Hub account limits

The OpenShift Container Platform cluster uses a number of Microsoft Azure Stack Hub components,
and the default Quota types in Azure Stack Hub affect your ability to install OpenShift Container
Platform clusters.
The following table summarizes the Azure Stack Hub components whose limits can impact your ability to
install and run OpenShift Container Platform clusters.
Component Number of Description

components required
by default
vCPU 56 A default cluster requires 56 vCPUs, so you must increase the

account limit.
By default, each cluster creates the following instances:
One bootstrap machine, which is removed after

installation
Because the bootstrap, control plane, and worker machines use

Standard_DS4_v2 virtual machines, which use 8 vCPUs, a
default cluster requires 56 vCPUs. The bootstrap node VM is used
To deploy more worker nodes, enable autoscaling, deploy large

workloads, or use a different instance type, you must further
increase the vCPU limit for your account to ensure that your
cluster can deploy the machines that you require.
VNet 1 Each default cluster requires one Virtual Network (VNet), which
contains two subnets.
Network 7 Each default cluster requires seven network interfaces. If you

interfaces create more machines or your deployed workloads create load
balancers, your cluster uses more network interfaces.
1281

components required
by default
Network 2 Each cluster creates network security groups for each subnet in
security the VNet. The default cluster creates network security groups for
groups the control plane and for the compute node subnets:
contr Allows the control plane machines to be reached on port

olpla 6443 from anywhere
ne
node Allows worker nodes to be reached from the internet on

ports 80 and 443
Network load 3 Each cluster creates the following load balancers:

balancers
defa Public IP address that load balances requests to ports 80

ult and 443 across worker machines
inter Private IP address that load balances requests to ports

nal 6443 and 22623 across control plane machines
exter Public IP address that load balances requests to port

nal 6443 across control plane machines
If your applications create more Kubernetes LoadBalancer

service objects, your cluster uses more load balancers.
Public IP 2 The public load balancer uses a public IP address. The bootstrap
addresses machine also uses a public IP address so that you can SSH into the
machine to troubleshoot issues during installation. The IP address
for the bootstrap node is used only during installation.
Private IP 7 The internal load balancer, each of the three control plane
addresses machines, and each of the three worker machines each use a
private IP address.
8.2.2. Configuring a DNS zone in Azure Stack Hub

To successfully install OpenShift Container Platform on Azure Stack Hub, you must create DNS records
in an Azure Stack Hub DNS zone. The DNS zone must be authoritative for the domain. To delegate a
registrar’s DNS zone to Azure Stack Hub, see Microsoft’s documentation for Azure Stack Hub
datacenter DNS integration.
1282
8.2.3. Required Azure Stack Hub roles

Your Microsoft Azure Stack Hub account must have the following roles for the subscription that you use:
Owner
To set roles on the Azure portal, see the Manage access to resources in Azure Stack Hub with role-
based access control in the Microsoft documentation.
8.2.4. Creating a service principal

Prerequisites
Procedure
1. Register your environment:
$ az cloud register -n AzureStackCloud --endpoint-resource-manager <endpoint> 1
1 Specify the Azure Resource Manager endpoint, `https://management.<region>.<fqdn>/`.
See the Microsoft documentation for details.
2. Set the active environment:
$ az cloud set -n AzureStackCloud
3. Update your environment configuration to use the specific API version for Azure Stack Hub:
$ az cloud update --profile 2019-03-01-hybrid
$ az login
If you are in a multitenant environment, you must also supply the tenant ID.
Example output
1283
[
{
"cloudName": AzureStackCloud",
"id": "9bab1460-96d5-40b3-a78e-17b15e978a80",
"isDefault": true,
"state": "Enabled",
"user": {
"type": "user"
}
}
]
$ az account show
Example output
{
"environmentName": AzureStackCloud",
"id": "9bab1460-96d5-40b3-a78e-17b15e978a80",
"isDefault": true,
"state": "Enabled",
"user": {
"type": "user"
}
}
$ az account show
Example output
{
1284
"isDefault": true,
"state": "Enabled",
"user": {
"type": "user"
}
}
$ az ad sp create-for-rbac --role Contributor --name <service_principal> \ 1

--years <years> 3
1 Specify the service principal name.
3 Specify the number of years. By default, a service principal expires in one year. By using the
--years option you can extend the validity of your service principal.
Example output

{
"password": "00000000-0000-0000-0000-000000000000",
}
8.2.5. Next steps

Installing a cluster quickly on Azure Stack Hub .
Install an OpenShift Container Platform cluster on Azure Stack Hub with user-provisioned
1285
Install an OpenShift Container Platform cluster on Azure Stack Hub with user-provisioned
infrastructure by following Installing a cluster on Azure Stack Hub using ARM templates .
8.3. INSTALLING A CLUSTER ON AZURE STACK HUB WITH AN

INSTALLER-PROVISIONED INFRASTRUCTURE
In OpenShift Container Platform version 4.15, you can install a cluster on Microsoft Azure Stack Hub with
an installer-provisioned infrastructure. However, you must manually configure the install-config.yaml
file to specify values that are specific to Azure Stack Hub.
NOTE
While you can select azure when using the installation program to deploy a cluster using
installer-provisioned infrastructure, this option is only supported for the Azure Public
Cloud.
processes.
users.
You configured an Azure Stack Hub account to host the cluster.
You verified that you have approximately 16 GB of local disk space. Installing the cluster
requires that you download the RHCOS virtual hard disk (VHD) cluster image and upload it to
your Azure Stack Hub environment so that it is accessible during deployment. Decompressing
the VHD files requires this amount of local disk space.

IMPORTANT
1286

authentication.
user.
IMPORTANT
NOTE
Procedure
NOTE
ecdsa algorithm.
1287
NOTE

task:
Example output
Agent pid 31874
NOTE

Example output
Next steps
program.
8.3.4. Uploading the RHCOS cluster image

You must download the RHCOS virtual hard disk (VHD) cluster image and upload it to your Azure Stack
Hub environment so that it is accessible during deployment.
Prerequisites
Procedure
1. Obtain the RHCOS VHD cluster image:
1288
a. Export the URL of the RHCOS VHD to an environment variable.
$ export COMPRESSED_VHD_URL=$(openshift-install coreos print-stream-json | jq -r

'.architectures.x86_64.artifacts.azurestack.formats."vhd.gz".disk.location')
b. Download the compressed RHCOS VHD file locally.
$ curl -O -L ${COMPRESSED_VHD_URL}
2. Decompress the VHD file.
NOTE
The decompressed VHD file is approximately 16 GB, so be sure that your host
system has 16 GB of free space available. The VHD file can be deleted once you
have uploaded it.
3. Upload the local VHD to the Azure Stack Hub environment, making sure that the blob is publicly
available. For example, you can upload the VHD to a blob using the az cli or the web portal.

for installation.
Prerequisites
Procedure
2. Select Azure as the cloud provider.
IMPORTANT
IMPORTANT
1289
components.

Prerequisites
recovery.
for your cluster.
Procedure
IMPORTANT
NOTE
Make the following modifications:
a. Specify the required installation parameters.
b. Update the platform.azure section to specify the parameters that are specific to Azure
Stack Hub.
c. Optional: Update one or more of the default configuration parameters to customize the
1290
installation.
IMPORTANT
Installation configuration parameters for Azure Stack Hub
8.3.6.1. Sample customized install-config.yaml file for Azure Stack Hub
IMPORTANT
apiVersion: v1
controlPlane: 2 3
name: master
platform:
azure:
osDisk:
diskSizeGB: 1024 4
diskType: premium_LRS
replicas: 3
compute: 5
- name: worker
platform:
azure:
osDisk:
diskSizeGB: 512 6
replicas: 3
metadata:
name: test-cluster 7 8
networking:
clusterNetwork:
- cidr: 10.128.0.0/14
hostPrefix: 23
machineNetwork:
- cidr: 10.0.0.0/16
1291
serviceNetwork:
- 172.30.0.0/16
platform:
azure:
armEndpoint: azurestack_arm_endpoint 10 11
baseDomainResourceGroupName: resource_group 12 13
region: azure_stack_local_region 14 15
cloudName: AzureStackCloud 17
clusterOSimage: https://vhdsa.blob.example.example.com/vhd/rhcos-410.84.202112040202-0-
azurestack.x86_64.vhd 18 19
pullSecret: '{"auths": ...}' 20 21
fips: false 22
1 7 10 12 14 17 18 20 Required.
value.
must begin with a hyphen, -, and the first line of the controlPlane section must not. Although both
sections currently define a single machine pool, it is possible that future versions of OpenShift
Container Platform will support defining multiple compute pools during installation. Only one
nodes is 1024 GB.
8 The name of the cluster.
value.
11 The Azure Resource Manager endpoint that your Azure Stack Hub operator provides.
13 The name of the resource group that contains the DNS zone for your base domain.
15 The name of your Azure Stack Hub local region.
16 The name of an existing resource group to install your cluster to. If undefined, a new resource
group is created for the cluster.
19 The URL of a storage blob in the Azure Stack environment that contains an RHCOS VHD.
21 The pull secret required to authenticate your cluster.
1292
IMPORTANT
When running Red Hat Enterprise Linux (RHEL) or Red Hat Enterprise Linux
CoreOS (RHCOS) booted in FIPS mode, OpenShift Container Platform core
components use the RHEL cryptographic libraries that have been submitted to NIST
for FIPS 140-2/140-3 Validation on only the x86_64, ppc64le, and s390x
architectures.
NOTE
process uses.
24 If the Azure Stack Hub environment is using an internal Certificate Authority (CA), adding the CA
certificate is required.
8.3.7. Manually manage cloud credentials

The Cloud Credential Operator (CCO) only supports your cloud provider in manual mode. As a result,
you must specify the identity and access management (IAM) secrets for your cloud provider.
Procedure
command:

--included \ 1
1293
metadata:
...
spec:
providerSpec:
roleBindings:
- role: Contributor
...
metadata:
...
spec:
providerSpec:
roleBindings:
- role: Contributor
...
secretRef:
...
apiVersion: v1
kind: Secret
metadata:
1294
data:
IMPORTANT
Updating cloud provider resources with manually maintained credentials
8.3.8. Configuring the cluster to use an internal CA

If the Azure Stack Hub environment is using an internal Certificate Authority (CA), update the cluster-
proxy-01-config.yaml file to configure the cluster to use the internal CA.
Prerequisites
Create the install-config.yaml file and specify the certificate trust bundle in .pem format.
Create the cluster manifests.
Procedure
1. From the directory in which the installation program creates files, go to the manifests directory.
2. Add user-ca-bundle to the spec.trustedCA.name field.
Example cluster-proxy-01-config.yaml file
kind: Proxy
metadata:
name: cluster
spec:
trustedCA:
name: user-ca-bundle
status: {}
3. Optional: Back up the manifests/ cluster-proxy-01-config.yaml file. The installation program

consumes the manifests/ directory when you deploy the cluster.

1295
IMPORTANT
Prerequisites
cluster.
Procedure
deployment:

--log-level=info 2

config.yaml file.
Verification
IMPORTANT
Example output
...
1296
IMPORTANT

IMPORTANT

Procedure
Portal.
$ tar xvf <file>

$ echo $PATH
Verification
$ oc <command>
1297

Procedure
Portal.

C:\> path
Verification
C:\> oc <command>

Procedure
Portal.
NOTE

$ echo $PATH
Verification
1298
$ oc <command>

Prerequisites
Procedure
$ oc whoami
Example output
system:admin

Prerequisites
Procedure
installation host:
NOTE
1299
NOTE

NOTE
Example output


8.3.14. Next steps

8.4. INSTALLING A CLUSTER ON AZURE STACK HUB WITH NETWORK

CUSTOMIZATIONS
1300
configuration on infrastructure that the installation program provisions on Azure Stack Hub. By
customizing your network configuration, your cluster can coexist with existing IP address allocations in
your environment and integrate with existing MTU and VXLAN configurations.
NOTE
While you can select azure when using the installation program to deploy a cluster using
installer-provisioned infrastructure, this option is only supported for the Azure Public
Cloud.
processes.
users.
You verified that you have approximately 16 GB of local disk space. Installing the cluster
requires that you download the RHCOS virtual hard disk (VHD) cluster image and upload it to
your Azure Stack Hub environment so that it is accessible during deployment. Decompressing
the VHD files requires this amount of local disk space.

IMPORTANT

1301
authentication.
user.
IMPORTANT
NOTE
Procedure
NOTE
ecdsa algorithm.
1302
NOTE

task:
Example output
Agent pid 31874
NOTE

Example output
Next steps
program.
8.4.4. Uploading the RHCOS cluster image

You must download the RHCOS virtual hard disk (VHD) cluster image and upload it to your Azure Stack
Hub environment so that it is accessible during deployment.
Prerequisites
Procedure
1. Obtain the RHCOS VHD cluster image:
a. Export the URL of the RHCOS VHD to an environment variable.

1303
b. Download the compressed RHCOS VHD file locally.
NOTE
system has 16 GB of free space available. The VHD file can be deleted once you
have uploaded it.
3. Upload the local VHD to the Azure Stack Hub environment, making sure that the blob is publicly
available. For example, you can upload the VHD to a blob using the az cli or the web portal.

for installation.
Prerequisites
Procedure
IMPORTANT
IMPORTANT
1304
components.

Prerequisites
recovery.
for your cluster.
Procedure
IMPORTANT
NOTE
Make the following modifications:
a. Specify the required installation parameters.
b. Update the platform.azure section to specify the parameters that are specific to Azure
Stack Hub.
installation.
IMPORTANT
1305
IMPORTANT
IMPORTANT
apiVersion: v1
controlPlane: 2 3
name: master
platform:
azure:
osDisk:
diskSizeGB: 1024 4
replicas: 3
compute: 5
- name: worker
platform:
azure:
osDisk:
diskSizeGB: 512 6
replicas: 3
metadata:
name: test-cluster 7 8
networking:
clusterNetwork:
- cidr: 10.128.0.0/14
hostPrefix: 23
machineNetwork:
- cidr: 10.0.0.0/16
serviceNetwork:
- 172.30.0.0/16
platform:
azure:
armEndpoint: azurestack_arm_endpoint 10 11
baseDomainResourceGroupName: resource_group 12 13
1306
region: azure_stack_local_region 14 15
clusterOSimage: https://vhdsa.blob.example.example.com/vhd/rhcos-410.84.202112040202-0-
azurestack.x86_64.vhd 18 19
pullSecret: '{"auths": ...}' 20 21
fips: false 22
1 7 10 12 14 17 18 20 Required.
value.
nodes is 1024 GB.
8 The name of the cluster.
value.
11 The Azure Resource Manager endpoint that your Azure Stack Hub operator provides.
13 The name of the resource group that contains the DNS zone for your base domain.
15 The name of your Azure Stack Hub local region.
16 The name of an existing resource group to install your cluster to. If undefined, a new resource
group is created for the cluster.
19 The URL of a storage blob in the Azure Stack environment that contains an RHCOS VHD.
21 The pull secret required to authenticate your cluster.
IMPORTANT
1307
IMPORTANT
architectures.
NOTE
process uses.
24 If the Azure Stack Hub environment is using an internal Certificate Authority (CA), adding the CA
certificate is required.
8.4.7. Manually manage cloud credentials

The Cloud Credential Operator (CCO) only supports your cloud provider in manual mode. As a result,
you must specify the identity and access management (IAM) secrets for your cloud provider.
Procedure
command:

--included \ 1
1308
metadata:
...
spec:
providerSpec:
roleBindings:
- role: Contributor
...
metadata:
...
spec:
providerSpec:
roleBindings:
- role: Contributor
...
secretRef:
...
apiVersion: v1
kind: Secret
metadata:
data:
1309
IMPORTANT
Updating a cluster using the web console
Updating a cluster using the CLI
8.4.8. Configuring the cluster to use an internal CA

If the Azure Stack Hub environment is using an internal Certificate Authority (CA), update the cluster-
proxy-01-config.yaml file to configure the cluster to use the internal CA.
Prerequisites
Create the install-config.yaml file and specify the certificate trust bundle in .pem format.
Create the cluster manifests.
Procedure
1. From the directory in which the installation program creates files, go to the manifests directory.
2. Add user-ca-bundle to the spec.trustedCA.name field.
Example cluster-proxy-01-config.yaml file
kind: Proxy
metadata:
name: cluster
spec:
trustedCA:
status: {}
3. Optional: Back up the manifests/ cluster-proxy-01-config.yaml file. The installation program

consumes the manifests/ directory when you deploy the cluster.

1310
Phase 1
NOTE

NIC resides in.
IMPORTANT
cluster.
Phase 2

the cluster.
IMPORTANT

Prerequisites
Procedure
1311
kind: Network
metadata:
name: cluster
spec:
kind: Network
metadata:
name: cluster
spec:
defaultNetwork:
ipsecConfig:
mode: Full


clusterNetwork
serviceNetwork
defaultNetwork.type
1312
spec:
clusterNetwork:
- cidr: 10.128.0.0/19
hostPrefix: 23
- cidr: 10.128.32.0/19
hostPrefix: 23
spec:
serviceNetwork:
- 172.30.0.0/14

manifest file.
work

1313

NOTE


network plugin.

detected MTU.


value to 1400.

configuration.
1314


NOTE

1315

t network
infrastructure
overlaps with the
100.64.0.0/16
IPv4 subnet, you
can specify a
different IP
address range for
internal use by
OVN-Kubernetes.
You must ensure
that the IP address
range does not
overlap with any
other subnet used
by your OpenShift
Container
Platform
address range
must be larger
than the maximum
number of nodes
that can be added
to the cluster. For
example, if the
clusterNetwork.
cidr value is
10.128.0.0/14
and the
clusterNetwork.
hostPrefix value
is /23, then the
maximum number
of nodes is 2^(23-
14)=512 .
This field cannot

be changed after
installation.
1316

t network
infrastructure
overlaps with the
fd98::/48 IPv6
subnet, you can
specify a different
IP address range
for internal use by
OVN-Kubernetes.
You must ensure
that the IP address
range does not
overlap with any
other subnet used
by your OpenShift
Container
Platform
address range
must be larger
than the maximum
number of nodes
that can be added
to the cluster.
This field cannot

be changed after
installation.

50000000 or 50 MB.
1317
libc
host.
udp:<host>:<port>
unix:<file>
null

1318


external hosts.

defaultNetwork:
type: OVNKubernetes
mtu: 1400
genevePort: 6081
ipsecConfig:
mode: Full
IMPORTANT

documentation.
NOTE
1319

kubeProxyConfig:
proxyArguments:
- 0s

NOTE
Prerequisites

Procedure
where:
cluster.

kind: Network
metadata:
name: cluster
spec:
EOF
1320
where:

kind: Network
metadata:
name: cluster
spec:
defaultNetwork:
- cidr: 10.132.0.0/14
hostPrefix: 23
NOTE

port.

NOTE

1321
IMPORTANT
Prerequisites
cluster.
Procedure
deployment:

--log-level=info 2

config.yaml file.
Verification
IMPORTANT
Example output
...
1322
IMPORTANT

IMPORTANT

Procedure
Portal.
$ tar xvf <file>

$ echo $PATH
Verification
$ oc <command>
1323

Procedure
Portal.

C:\> path
Verification
C:\> oc <command>

Procedure
Portal.
NOTE

$ echo $PATH
Verification
1324
$ oc <command>

Prerequisites
Procedure
$ oc whoami
Example output
system:admin

Prerequisites
Procedure
installation host:
NOTE
1325
NOTE

NOTE
Example output

Accessing the web console.

8.4.18. Next steps

8.5. INSTALLING A CLUSTER ON AZURE STACK HUB USING ARM

TEMPLATES
1326
In OpenShift Container Platform version 4.15, you can install a cluster on Microsoft Azure Stack Hub by
using infrastructure that you provide.
IMPORTANT

own. You are also free to create the required resources through other methods; the
templates are just an example.
processes.
users.
You downloaded the Azure CLI and installed it on your computer. See Install the Azure CLI in
the Azure documentation. The documentation below was tested using version 2.28.0 of the
Azure CLI. Azure CLI commands might perform differently based on the version you use.
NOTE

IMPORTANT
1327
IMPORTANT
8.5.3. Configuring your Azure Stack Hub project

IMPORTANT
All Azure Stack Hub resources that are available through public endpoints are subject to
resource name restrictions, and you cannot create resources that use certain terms. For a
list of terms that Azure Stack Hub restricts, see Resolve reserved resource name errors in
the Azure documentation.
8.5.3.1. Azure Stack Hub account limits
The OpenShift Container Platform cluster uses a number of Microsoft Azure Stack Hub components,
and the default Quota types in Azure Stack Hub affect your ability to install OpenShift Container
Platform clusters.
The following table summarizes the Azure Stack Hub components whose limits can impact your ability to
install and run OpenShift Container Platform clusters.

components required
by default
vCPU 56 A default cluster requires 56 vCPUs, so you must increase the

account limit.
By default, each cluster creates the following instances:
One bootstrap machine, which is removed after

installation
Because the bootstrap, control plane, and worker machines use

Standard_DS4_v2 virtual machines, which use 8 vCPUs, a
default cluster requires 56 vCPUs. The bootstrap node VM is used
To deploy more worker nodes, enable autoscaling, deploy large

workloads, or use a different instance type, you must further
increase the vCPU limit for your account to ensure that your
cluster can deploy the machines that you require.
1328

components required
by default
VNet 1 Each default cluster requires one Virtual Network (VNet), which
contains two subnets.
Network 7 Each default cluster requires seven network interfaces. If you

interfaces create more machines or your deployed workloads create load
balancers, your cluster uses more network interfaces.
Network 2 Each cluster creates network security groups for each subnet in
security the VNet. The default cluster creates network security groups for
groups the control plane and for the compute node subnets:
contr Allows the control plane machines to be reached on port

olpla 6443 from anywhere
ne
node Allows worker nodes to be reached from the internet on

ports 80 and 443
Network load 3 Each cluster creates the following load balancers:

balancers
defa Public IP address that load balances requests to ports 80

ult and 443 across worker machines
inter Private IP address that load balances requests to ports

nal 6443 and 22623 across control plane machines
exter Public IP address that load balances requests to port

nal 6443 across control plane machines
If your applications create more Kubernetes LoadBalancer

service objects, your cluster uses more load balancers.
Public IP 2 The public load balancer uses a public IP address. The bootstrap
addresses machine also uses a public IP address so that you can SSH into the
machine to troubleshoot issues during installation. The IP address
for the bootstrap node is used only during installation.
Private IP 7 The internal load balancer, each of the three control plane
addresses machines, and each of the three worker machines each use a
private IP address.
1329
8.5.3.2. Configuring a DNS zone in Azure Stack Hub
To successfully install OpenShift Container Platform on Azure Stack Hub, you must create DNS records
in an Azure Stack Hub DNS zone. The DNS zone must be authoritative for the domain. To delegate a
registrar’s DNS zone to Azure Stack Hub, see Microsoft’s documentation for Azure Stack Hub
datacenter DNS integration.
and approving them.
8.5.3.4. Required Azure Stack Hub roles
Your Microsoft Azure Stack Hub account must have the following roles for the subscription that you use:
Owner
To set roles on the Azure portal, see the Manage access to resources in Azure Stack Hub with role-
based access control in the Microsoft documentation.
Prerequisites
Procedure
1. Register your environment:
$ az cloud register -n AzureStackCloud --endpoint-resource-manager <endpoint> 1
1 Specify the Azure Resource Manager endpoint, `https://management.<region>.<fqdn>/`.
See the Microsoft documentation for details.
2. Set the active environment:
$ az cloud set -n AzureStackCloud
3. Update your environment configuration to use the specific API version for Azure Stack Hub:
1330
$ az cloud update --profile 2019-03-01-hybrid
$ az login
If you are in a multitenant environment, you must also supply the tenant ID.
Example output
[
{
"cloudName": AzureStackCloud",
"id": "9bab1460-96d5-40b3-a78e-17b15e978a80",
"isDefault": true,
"state": "Enabled",
"user": {
"type": "user"
}
}
]
$ az account show
Example output
{
"id": "9bab1460-96d5-40b3-a78e-17b15e978a80",
"isDefault": true,
"state": "Enabled",
"user": {
"type": "user"
}
}
1331
$ az account show
Example output
{
"isDefault": true,
"state": "Enabled",
"user": {
"type": "user"
}
}
$ az ad sp create-for-rbac --role Contributor --name <service_principal> \ 1

--years <years> 3
1 Specify the service principal name.
3 Specify the number of years. By default, a service principal expires in one year. By using the
--years option you can extend the validity of your service principal.
Example output

{
1332
"password": "00000000-0000-0000-0000-000000000000",
}

for installation.
Prerequisites
Procedure
IMPORTANT
IMPORTANT
1333
components.

authentication.
user.
IMPORTANT
NOTE
Procedure
NOTE
ecdsa algorithm.
1334
NOTE

task:
Example output
Agent pid 31874
NOTE

Example output
Next steps
program.
8.5.6. Creating the installation files for Azure Stack Hub

To install OpenShift Container Platform on Microsoft Azure Stack Hub using user-provisioned
and modify them so that the cluster creates only the machines that it will use. You manually create the
install-config.yaml file, and then generate and customize the Kubernetes manifests and Ignition config
files. You also have the option to first set up a separate var partition during the preparation phases of
installation.
8.5.6.1. Manually creating the installation configuration file
1335
Prerequisites
recovery.
for your cluster.
Procedure
IMPORTANT
NOTE
Make the following modifications for Azure Stack Hub:
a. Set the replicas parameter to 0 for the compute pool:
compute:
name: worker
platform: {}
replicas: 0 1
1 Set to 0.
The compute machines will be provisioned manually later.
b. Update the platform.azure section of the install-config.yaml file to configure your Azure
Stack Hub configuration:
platform:
azure:
1336
armEndpoint: <azurestack_arm_endpoint> 1
baseDomainResourceGroupName: <resource_group> 2
region: <azurestack_region> 4
1 Specify the Azure Resource Manager endpoint of your Azure Stack Hub environment,
like https://management.local.azurestack.external.
2 Specify the name of the resource group that contains the DNS zone for your base
domain.
3 Specify the Azure Stack Hub environment, which is used to configure the Azure SDK
with the appropriate Azure API endpoints.
4 Specify the name of your Azure Stack Hub region.
IMPORTANT
IMPORTANT
apiVersion: v1
controlPlane: 1
name: master
platform:
azure:
osDisk:
diskSizeGB: 1024 2
replicas: 3
compute: 3
- name: worker
platform:
azure:
osDisk:
1337
diskSizeGB: 512 4
replicas: 0
metadata:
networking:
clusterNetwork:
- cidr: 10.128.0.0/14
hostPrefix: 23
machineNetwork:
- cidr: 10.0.0.0/16
serviceNetwork:
- 172.30.0.0/16
platform:
azure:
armEndpoint: azurestack_arm_endpoint 7
region: azure_stack_local_region 9
fips: false 13
nodes is 1024 GB.
5 Specify the name of the cluster.
value.
7 Specify the Azure Resource Manager endpoint that your Azure Stack Hub operator provides.
9 Specify the name of your Azure Stack Hub local region.
11 Specify the Azure Stack Hub environment as your target platform.
12 Specify the pull secret required to authenticate your cluster.
1338
IMPORTANT
14 If your Azure Stack Hub environment uses an internal certificate authority (CA), add the necessary
certificate bundle in .pem format.
NOTE
process uses.
Prerequisites
NOTE
(169.254.169.254).
Procedure
1339
apiVersion: v1
proxy:
must be http.

destinations.
NOTE
NOTE
1340
NOTE
created.
(ARM) templates used to assist in completing a user-provided infrastructure install on Microsoft Azure
Stack Hub.
NOTE
Prerequisites
cluster.
Procedure
templates:
2 The region to deploy the cluster into. This is the value of the .platform.azure.region
attribute from the install-config.yaml file.
file.
4 The base domain to deploy the cluster to. The base domain corresponds to the DNS zone
that you created for your cluster. This is the value of the .baseDomain attribute from the
install-config.yaml file.
5 The resource group where the DNS zone exists. This is the value of the
config.yaml file.
For example:
1341
machines.
IMPORTANT
Prerequisites
Procedure
1342
machines.
machine-set.yaml
IMPORTANT

these machines.

machines:
kind: DNS
metadata:
name: cluster
spec:
privateZone: 1
publicZone: 2
status: {}
1343
7. Optional: If your Azure Stack Hub environment uses an internal certificate authority (CA), you
must update the .spec.trustedCA.name field in the
<installation_directory>/manifests/cluster-proxy-01-config.yaml file to use user-ca-bundle:
...
spec:
trustedCA:
...
Later, you must update your bootstrap ignition to include the CA.
templates:
config.yml file.
9. Manually create your cloud credentials.
a. From the directory that contains the installation program, obtain details of the OpenShift
Container Platform release image that your openshift-install binary is built to use:
$ openshift-install version
Example output
release image quay.io/openshift-release-dev/ocp-release:4.y.z-x86_64
b. Set a $RELEASE_IMAGE variable with the release image from your installation file by
1344
c. Extract the list of CredentialsRequest custom resources (CRs) from the OpenShift
Container Platform release image by running the following command:

--included \ 1
2
metadata:
labels:
controller-tools.k8s.io: "1.0"
name: openshift-image-registry-azure
spec:
secretRef:
name: installer-cloud-credentials
namespace: openshift-image-registry
providerSpec:
roleBindings:
- role: Contributor
d. Create YAML files for secrets in the openshift-install manifests directory that you
generated previously. The secrets must be stored using the namespace and secret name
defined in the spec.secretRef for each CredentialsRequest object. The format for the
secret data varies for each cloud provider.
Sample secrets.yaml file:
apiVersion: v1
kind: Secret
metadata:
name: ${secret_name}
namespace: ${secret_namespace}
stringData:
1345
azure_subscription_id: ${subscription_id}
azure_client_id: ${app_id}
azure_client_secret: ${client_secret}
azure_tenant_id: ${tenant_id}
azure_resource_prefix: ${cluster_name}
azure_resourcegroup: ${resource_group}
azure_region: ${azure_region}
e. Create a cco-configmap.yaml file in the manifests directory with the Cloud Credential
Operator (CCO) disabled:
Sample ConfigMap object
apiVersion: v1
kind: ConfigMap
metadata:
name: cloud-credential-operator-config
annotations:
release.openshift.io/create-only: "true"
data:
disabled: "true"
.
├── auth
Manually manage cloud credentials
1346
installation.
IMPORTANT
Procedure
Example output


Example output
1347
...
variant: openshift
version: 4.15.0
metadata:
labels:
storage:
disks:
partitions:
- label: var
number: 5
filesystems:
path: /var
format: xfs
NOTE
name.
1348

partition.yaml


You must create a Microsoft Azure resource group. This is used during the installation of your OpenShift
Container Platform cluster on Azure Stack Hub.
Prerequisites
Procedure
Create the resource group in a supported Azure region:
Prerequisites
Procedure

1349

WARNING



IMPORTANT
The RHCOS images might not change with every release of OpenShift Container
Platform. You must specify an image with the highest version that is less than or
equal to the OpenShift Container Platform version that you install. Use the image
version that matches your OpenShift Container Platform version if it is available.

key ${ACCOUNT_KEY}
5. Download the compressed RHCOS VHD file locally:
NOTE
system has 16 GB of free space available. You can delete the VHD file after you
upload it.

${ACCOUNT_KEY} -c vhd -n "rhcos.vhd" -f rhcos-<rhcos_version>-azurestack.x86_64.vhd
1350



For this example, Azure Stack Hub’s datacenter DNS integration is used, so you will create a DNS zone.
NOTE
The DNS zone is not required to exist in the same resource group as the cluster
that is the case, you can skip creating the DNS zone; be sure the installation config you
generated earlier reflects that scenario.
Prerequisites
Procedure
Create the new DNS zone in the resource group exported in the

You can skip this step if you are using a DNS zone that already exists.
You can learn more about configuring a DNS zone in Azure Stack Hub by visiting that section.
8.5.10. Creating a VNet in Azure Stack Hub

You must create a virtual network (VNet) in Microsoft Azure Stack Hub for your OpenShift Container
Platform cluster to use. You can customize the VNet to meet your requirements. One way to create the
VNet is to modify the provided Azure Resource Manager (ARM) template.
NOTE
If you do not use the provided ARM template to create your Azure Stack Hub
1351
Prerequisites
Procedure
cluster requires.

link:https://raw.githubusercontent.com/openshift/installer/release-
4.15/upi/azurestack/01_vnet.json[]
8.5.11. Deploying the RHCOS cluster image for the Azure Stack Hub infrastructure
You must use a valid Red Hat Enterprise Linux CoreOS (RHCOS) image for Microsoft Azure Stack Hub
for your OpenShift Container Platform nodes.
Prerequisites
Procedure
1352


4.15/upi/azurestack/02_storage.json[]

IMPORTANT
Hat.
1353
TCP 1936 Metrics
UDP 4789 VXLAN
6081 Geneve
9100- 9101.
8.5.13. Creating networking and load balancing components in Azure Stack Hub
You must configure networking and load balancing in Microsoft Azure Stack Hub for your OpenShift
Container Platform cluster to use. One way to create these components is to modify the provided Azure
Resource Manager (ARM) template.
Load balancing requires the following DNS records:
An api DNS record for the API public load balancer in the DNS zone.
1354
An api-int DNS record for the API internal load balancer in the DNS zone.
NOTE
If you do not use the provided ARM template to create your Azure Stack Hub
Prerequisites
Create and configure a VNet and associated subnets in Azure Stack Hub.
Procedure

3. Create an api DNS record and an api-int DNS record. When creating the API DNS records, the
DNS zone exists.

b. Export the following variable:
$ export PRIVATE_IP=àz network lb frontend-ip show -g "$RESOURCE_GROUP" --lb-

name "${INFRA_ID}-internal" -n internal-lb-ip --query "privateIpAddress" -o tsv`
c. Create the api DNS record in a new DNS zone:

If you are adding the cluster to an existing DNS zone, you can create the api DNS record in
it instead:
1355

d. Create the api-int DNS record in a new DNS zone:

z "${CLUSTER_NAME}.${BASE_DOMAIN}" -n api-int -a ${PRIVATE_IP} --ttl 60
If you are adding the cluster to an existing DNS zone, you can create the api-int DNS record
in it instead:

z ${BASE_DOMAIN} -n api-int.${CLUSTER_NAME} -a ${PRIVATE_IP} --ttl 60
4.15/upi/azurestack/03_infra.json[]
8.5.14. Creating the bootstrap machine in Azure Stack Hub

You must create the bootstrap machine in Microsoft Azure Stack Hub to use during OpenShift
Container Platform cluster initialization. One way to create this machine is to modify the provided Azure
Resource Manager (ARM) template.
NOTE
installation logs.
Prerequisites
Create and configure networking and load balancers in Azure Stack Hub.
Procedure
1356

a. If your environment uses a public certificate authority (CA), run this command:
$ export BOOTSTRAP_IGNITION=`jq -rcnM --arg v "3.2.0" --arg url

${BOOTSTRAP_URL} '{ignition:{version:$v,config:{replace:{source:$url}}}}' | base64 | tr -
d '\n'`
b. If your environment uses an internal CA, you must add your PEM encoded bundle to the
bootstrap ignition stub so that your bootstrap virtual machine can pull the bootstrap
ignition from the storage account. Run the following commands, which assume your CA is in
a file called CA.pem:
$ export CA="data:text/plain;charset=utf-8;base64,$(cat CA.pem |base64 |tr -d '\n')"
$ export BOOTSTRAP_IGNITION=`jq -rcnM --arg v "3.2.0" --arg url

"$BOOTSTRAP_URL" --arg cert "$CA" '{ignition:{version:$v,security:{tls:
{certificateAuthorities:[{source:$cert}]}},config:{replace:{source:$url}}}}' | base64 | tr -d
'\n'`
$ az deployment group create --verbose -g ${RESOURCE_GROUP} \

--parameters diagnosticsStorageAccountName="${CLUSTER_NAME}sa" 3
3 The name of the storage account for your cluster.
1357
4.15/upi/azurestack/04_bootstrap.json[]
8.5.15. Creating the control plane machines in Azure Stack Hub

You must create the control plane machines in Microsoft Azure Stack Hub for your cluster to use. One
way to create these machines is to modify the provided Azure Resource Manager (ARM) template.
Prerequisites
Procedure

1 The Ignition content for the control plane nodes (also known as the master nodes).
1358
4.15/upi/azurestack/05_masters.json[]
Stack Hub
After you create all of the required infrastructure in Microsoft Azure Stack Hub, wait for the bootstrap
process to complete on the machines that you provisioned by using the Ignition config files that you
generated with the installation program.
Prerequisites
Procedure

--log-level info 2

1359

wait --yes
wait
ssh-pip
NOTE
8.5.17. Creating additional worker machines in Azure Stack Hub

You can create worker machines in Microsoft Azure Stack Hub for your cluster to use by launching
individual instances discretely or by automated processes outside the cluster, such as auto scaling
groups. You can also take advantage of the built-in cluster scaling mechanisms and the machine API in
Prerequisites
Procedure
1360

4.15/upi/azurestack/06_workers.json[]

IMPORTANT

Procedure
Portal.
$ tar xvf <file>
1361

$ echo $PATH
Verification
$ oc <command>

Procedure
Portal.

C:\> path
Verification
C:\> oc <command>

Procedure
Portal.
NOTE
1362

$ echo $PATH
Verification
$ oc <command>

Prerequisites
Procedure
$ oc whoami
Example output
system:admin

Prerequisites
Procedure
1363
Procedure
$ oc get nodes
Example output

NOTE
$ oc get csr
Example output

...
list.
NOTE
NOTE
1364
NOTE
of the node.

NOTE
$ oc get csr
Example output

Pending
Pending
...
1365

$ oc get nodes
Example output

NOTE

Prerequisites
You deployed an OpenShift Container Platform cluster on Microsoft Azure Stack Hub by using
Procedure
Example output
1366

80:32288/TCP,443:31215/TCP 20

3. Add a *.apps record to the DNS zone.
a. If you are adding this cluster to a new DNS zone, run:

b. If you are adding this cluster to an already existing DNS zone, run:


{end}' routes
Example output
8.5.22. Completing an Azure Stack Hub installation on user-provisioned

infrastructure
After you start the OpenShift Container Platform installation on Microsoft Azure Stack Hub user-
provisioned infrastructure, you can monitor the cluster events until the cluster is ready.
Prerequisites
Azure Stack Hub infrastructure.
Procedure
1367
Example output
IMPORTANT
installation.
8.6. INSTALLATION CONFIGURATION PARAMETERS FOR AZURE

STACK HUB
Before you deploy an OpenShift Container Platform cluster on Azure Stack Hub, you provide a
customized install-config.yaml installation configuration file that describes the details for your
environment.
8.6.1. Available installation configuration parameters for Azure Stack Hub

The following tables specify the required, optional, and Azure Stack Hub-specific installation
configuration parameters that you can set as part of the installation process.
NOTE
1368

versions.

combination of the
baseDomain and
<metadata.name>.

consumed.
name: subdomains of
{{.metadata.name}}.
{{.baseDomain}}.

alibabacloud, aws,
ibmcloud, nutanix,
1369

"auth":"b3Blb=",
}
}
}
NOTE

NOTE
You cannot modify

by the networking
object after
installation.

1370

networking:
clusterNetwork: The default value is 10.128.0.0/14
networking:
- cidr: 10.128.0.0/14
hostPrefix: 23

An IPv4 network.

23 provides 510 (2^(32 - 23) - 2) pod
IP addresses.
serviceNetwork:
networking:
supports only a single IP address block serviceNetwork:
- 172.30.0.0/16

networking:
networking:
blocks, the blocks must not overlap. machineNetwork:
- cidr: 10.0.0.0/16

than libvirt and IBM Power® Virtual NOTE
the CIDR that the
in.
1371



et:


section.

1372

(the default).

cores.
IMPORTANT
If you disable
simultaneous
that your capacity
the dramatically
decreased machine
performance.

name:
value.

1373


(the default).

cores.
IMPORTANT
If you disable
simultaneous
that your capacity
the dramatically
decreased machine
performance.

name:

parameter value.
1374

replicas:

1375

IMPORTANT
To enable FIPS mode

must run the
from a Red Hat
Enterprise Linux
(RHEL) computer
in FIPS mode. For
more information
about configuring
FIPS mode on RHEL,
see Installing the
Enterprise Linux
(RHEL) or Red Hat
Enterprise Linux
CoreOS (RHCOS)
OpenShift Container
Platform core
components use the
RHEL cryptographic
libraries that have
been submitted to
NIST for FIPS 140-
only the x86_64,
ppc64le, and s390x
architectures.
NOTE

File storage, you
cannot enable FIPS
mode.

1376


ces:
mirrors:

IMPORTANT

cluster will become
non-functional. For
more information,

NOTE
For production
OpenShift Container
which you want to
SSH key that your
ssh-agent process
uses.
8.6.1.4. Additional Azure Stack Hub configuration parameters
1377
Additional Azure configuration parameters are described in the following table:
Table 8.14. Additional Azure Stack Hub parameters
compute: disk in GB. The default is 128.
platform:
azure:
osDisk:
diskSizeGB:
Defines the type of disk. standard_LRS or premium_LRS.

compute: The default is premium_LRS.
platform:
azure:
osDisk:
diskType:
Defines the azure instance type for String

platform:
azure:
type:
controlPlane: disk in GB. The default is 1024.
platform:
azure:
osDisk:
diskSizeGB:
Defines the type of disk. premium_LRS.

controlPlane:
platform:
azure:
osDisk:
diskType:
Defines the azure instance type for String

platform:
azure:
type:
1378
platform: disk in GB. The default is 128.
azure:
defaultMachinePlatf
orm:
osDisk:
diskSizeGB:
Defines the type of disk. standard_LRS or premium_LRS.

platform: The default is premium_LRS.
azure:
defaultMachinePlatf
orm:
osDisk:
diskType:
The Azure instance type for control The Azure instance type.
platform: plane and compute machines.
azure:
defaultMachinePlatf
orm:
type:
The URL of the Azure Resource String

platform: Manager endpoint that your Azure
azure: Stack Hub operator provides.
armEndpoint:
The name of the resource group that String, for example

platform: contains the DNS zone for your base production_cluster .
azure: domain.
baseDomainResou
rceGroupName:
The name of your Azure Stack Hub String

platform: local region.
azure:
region:
1379
The name of an already existing String, for example

platform: resource group to install your cluster existing_resource_group.
azure: to. This resource group must be empty
and only used for this specific cluster;
resourceGroupNa the cluster components assume
me: ownership of all resources in the
resource group. If you limit the service
principal scope of the installation
program to this resource group, you
must ensure all other resources used
by the installation program in your
environment have the necessary
permissions, such as the public DNS
zone and virtual network. Destroying
the cluster by using the installation
program deletes this resource group.
The outbound routing strategy used to LoadBalancer or

platform: connect your cluster to the internet. If UserDefinedRouting. The default is
azure: you are using user-defined routing, LoadBalancer .
outboundType: you must have pre-existing networking
available where the outbound routing
has already been configured prior to
installing a cluster. The installation
program is not responsible for
configuring user-defined routing.
The name of the Azure cloud AzureStackCloud

platform: environment that is used to configure
azure: the Azure SDK with the appropriate
cloudName: Azure API endpoints.
The URL of a storage blob in the Azure String, for example,

clusterOSImage: Stack environment that contains an https://vhdsa.blob.example.example.c
RHCOS VHD. om/vhd/rhcos-410.84.202112040202-
0-azurestack.x86_64.vhd
8.7. UNINSTALLING A CLUSTER ON AZURE STACK HUB

You can remove a cluster that you deployed to Azure Stack Hub.

NOTE
1380
NOTE
Prerequisites
Procedure

NOTE
1381
CHAPTER 9. INSTALLING ON GCP
9.1. PREPARING TO INSTALL ON GCP
processes.
users.
9.1.2. Requirements for installing OpenShift Container Platform on GCP

Before installing OpenShift Container Platform on Google Cloud Platform (GCP), you must create a
service account and configure a GCP project. See Configuring a GCP project for details about creating
a project, enabling API services, configuring DNS, GCP account limits, and supported GCP regions.
Manually creating long-term credentials for GCP for other options.
9.1.3. Choosing a method to install OpenShift Container Platform on GCP

You can install a cluster on GCP infrastructure that is provisioned by the OpenShift Container Platform
Installing a cluster quickly on GCP: You can install OpenShift Container Platform on GCP
Installing a customized cluster on GCP: You can install a customized cluster on GCP
Installing a cluster on GCP with network customizations: You can customize your OpenShift
Installing a cluster on GCP in a restricted network: You can install OpenShift Container
Platform on GCP on installer-provisioned infrastructure by using an internal mirror of the
installation release content. You can use this method to install a cluster that does not require an
1382
active internet connection to obtain the software components. While you can install OpenShift
Container Platform by using the mirrored content, your cluster still requires internet access to
use the GCP APIs.
Installing a cluster into an existing Virtual Private Cloud: You can install OpenShift Container
Platform on an existing GCP Virtual Private Cloud (VPC). You can use this installation method if
you have constraints set by the guidelines of your company, such as limits on creating new
accounts or infrastructure.
GCP VPC. You can use this method to deploy OpenShift Container Platform on an internal
network that is not visible to the internet.
You can install a cluster on GCP infrastructure that you provision, by using one of the following methods:
Installing a cluster on GCP with user-provisioned infrastructure: You can install OpenShift
Container Platform on GCP infrastructure that you provide. You can use the provided
Deployment Manager templates to assist with the installation.
Installing a cluster with shared VPC on user-provisioned infrastructure in GCP: You can use
the provided Deployment Manager templates to create GCP resources in a shared VPC
infrastructure.
Installing a cluster on GCP in a restricted network with user-provisioned infrastructure : You

can install OpenShift Container Platform on GCP in a restricted network with user-provisioned
infrastructure. By creating an internal mirror of the installation release content, you can install a
cluster that does not require an active internet connection to obtain the software components.
You can also use this installation method to ensure that your clusters only use container images
that satisfy your organizational controls on external content.
9.1.4. Next steps

Configuring a GCP project
9.2. CONFIGURING A GCP PROJECT

Before you can install OpenShift Container Platform, you must configure a Google Cloud Platform
(GCP) project to host it.
9.2.1. Creating a GCP project

To install OpenShift Container Platform, you must create a project in your Google Cloud Platform
(GCP) account to host the cluster.
Procedure
Create a project to host your OpenShift Container Platform cluster. See Creating and
Managing Projects in the GCP documentation.
IMPORTANT
1383
IMPORTANT
Your GCP project must use the Premium Network Service Tier if you are using
installer-provisioned infrastructure. The Standard Network Service Tier is not
supported for clusters installed using the installation program. The installation
program configures internal load balancing for the api-int.<cluster_name>.
<base_domain> URL; the Premium Tier is required for internal load balancing.
9.2.2. Enabling API services in GCP

Your Google Cloud Platform (GCP) project requires access to several API services to complete
Prerequisites
You created a project to host your cluster.
Procedure
Enable the following required API services in the project that hosts your cluster. You may also
enable optional API services which are not required for installation. See Enabling services in the
GCP documentation.
Table 9.1. Required API services
API service Console service name
Compute Engine API compute.googleapis.com
Cloud Resource Manager API cloudresourcemanager.googleapis.com
Google DNS API dns.googleapis.com
IAM Service Account Credentials API iamcredentials.googleapis.com
Identity and Access Management iam.googleapis.com

(IAM) API
Service Usage API serviceusage.googleapis.com
Table 9.2. Optional API services
Google Cloud APIs cloudapis.googleapis.com
Service Management API servicemanagement.googleapis.com
Google Cloud Storage JSON API storage-api.googleapis.com
1384
Cloud Storage storage-component.googleapis.com
9.2.3. Configuring DNS for GCP

To install OpenShift Container Platform, the Google Cloud Platform (GCP) account you use must have
a dedicated public hosted zone in the same project that you host the OpenShift Container Platform
cluster. This zone must be authoritative for the domain. The DNS service provides cluster DNS
resolution and name lookup for external connections to the cluster.
Procedure
registrar or obtain a new one through GCP or another source.
NOTE
If you purchase a new domain, it can take time for the relevant DNS changes to
propagate. For more information about purchasing domains through Google, see
Google Domains.
2. Create a public hosted zone for your domain or subdomain in your GCP project. See Creating
public zones in the GCP documentation.
3. Extract the new authoritative name servers from the hosted zone records. See Look up your
Cloud DNS name servers in the GCP documentation.
You typically have four name servers.
4. Update the registrar records for the name servers that your domain uses. For example, if you
registered your domain to Google Domains, see the following topic in the Google Domains Help:
How to switch to custom name servers .
5. If you migrated your root domain to Google Cloud DNS, migrate your DNS records. See
Migrating to Cloud DNS in the GCP documentation.
parent domain. This process might include a request to your company’s IT department or the
division that controls the root domain and DNS services for your company.
9.2.4. GCP account limits

The OpenShift Container Platform cluster uses a number of Google Cloud Platform (GCP)
components, but the default Quotas do not affect your ability to install a default OpenShift Container
Platform cluster.
A default cluster, which contains three compute and three control plane machines, uses the following
resources. Note that some resources are required only during the bootstrap process and are removed
after the cluster deploys.
Table 9.3. GCP resources used in a default cluster
1385
Service Component Location Total resources Resources

required removed after
bootstrap
Service account IAM Global 6 1
Firewall rules Compute Global 11 1
Forwarding rules Compute Global 2 0
In-use global IP Compute Global 4 1

addresses
Health checks Compute Global 3 0
Images Compute Global 1 0
Networks Compute Global 2 0
Static IP addresses Compute Region 4 1
Routers Compute Global 1 0
Routes Compute Global 2 0
Subnetworks Compute Global 2 0
Target pools Compute Global 3 0
CPUs Compute Region 28 4
Persistent disk Compute Region 896 128

SSD (GB)
NOTE
If any of the quotas are insufficient during installation, the installation program displays an
error that states both which quota was exceeded and the region.
Be sure to consider your actual cluster size, planned cluster growth, and any usage from other clusters
that are associated with your account. The CPU, static IP addresses, and persistent disk SSD (storage)
quotas are the ones that are most likely to be insufficient.
If you plan to deploy your cluster in one of the following regions, you will exceed the maximum storage
quota and are likely to exceed the CPU quota limit:
asia-east2
asia-northeast2
1386
asia-south1
australia-southeast1
europe-north1
europe-west2
europe-west3
europe-west6
northamerica-northeast1
southamerica-east1
us-west2
You can increase resource quotas from the GCP console, but you might need to file a support ticket. Be
sure to plan your cluster size early so that you can allow time to resolve the support ticket before you
install your OpenShift Container Platform cluster.
9.2.5. Creating a service account in GCP

OpenShift Container Platform requires a Google Cloud Platform (GCP) service account that provides
authentication and authorization to access data in the Google APIs. If you do not have an existing IAM
service account that contains the required roles in your project, you must create one.
Prerequisites
Procedure
1. Create a service account in the project that you use to host your OpenShift Container Platform
cluster. See Creating a service account in the GCP documentation.
2. Grant the service account the appropriate permissions. You can either grant the individual
permissions that follow or assign the Owner role to it. See Granting roles to a service account
for specific resources.
NOTE
While making the service account an owner of the project is the easiest way to
gain the required permissions, it means that service account has complete
control over the project. You must determine if the risk that comes from offering
that power is acceptable.
3. You can create the service account key in JSON format, or attach the service account to a GCP
virtual machine. See Creating service account keys and Creating and enabling service accounts
for instances in the GCP documentation.
You must have a service account key or a virtual machine with an attached service account to
create the cluster.
NOTE
1387
NOTE
If you use a virtual machine with an attached service account to create your
cluster, you must set credentialsMode: Manual in the install-config.yaml file
before installation.
9.2.5.1. Required GCP roles
When you attach the Owner role to the service account that you create, you grant that service account
all permissions, including those that are required to install OpenShift Container Platform. If your
organization’s security policies require a more restrictive set of permissions, you can create a service
account with the following permissions. If you deploy your cluster into an existing virtual private cloud
(VPC), the service account does not require certain networking permissions, which are noted in the
following lists:
Required roles for the installation program
Compute Admin
Role Administrator
Security Admin
Service Account Admin
Service Account Key Admin
Service Account User
Storage Admin
Required roles for creating network resources during installation
DNS Administrator
Required roles for using the Cloud Credential Operator in passthrough mode
Compute Load Balancer Admin
The following roles are applied to the service accounts that the control plane and compute machines
use:
Table 9.4. GCP service account roles
Account Roles
Control Plane roles/compute.instanceAdmin
roles/compute.networkAdmin
roles/compute.securityAdmin
roles/storage.admin
1388
Account Roles
roles/iam.serviceAccountUser
Compute roles/compute.viewer
roles/storage.admin
9.2.5.2. Required GCP permissions for installer-provisioned infrastructure
all permissions, including those that are required to install OpenShift Container Platform.
If your organization’s security policies require a more restrictive set of permissions, you can create
custom roles with the necessary permissions. The following permissions are required for the installer-
provisioned infrastructure for creating and deleting the OpenShift Container Platform cluster.
compute.addresses.create
compute.addresses.createInternal
compute.addresses.delete
compute.addresses.get
compute.addresses.list
compute.addresses.use
compute.addresses.useInternal
compute.firewalls.create
compute.firewalls.delete
compute.firewalls.get
compute.firewalls.list
compute.forwardingRules.create
compute.forwardingRules.get
compute.forwardingRules.list
compute.forwardingRules.setLabels
compute.networks.create
compute.networks.get
compute.networks.list
1389
compute.networks.updatePolicy
compute.routers.create
compute.routers.get
compute.routers.list
compute.routers.update
compute.routes.list
compute.subnetworks.create
compute.subnetworks.get
compute.subnetworks.list
compute.subnetworks.use
compute.subnetworks.useExternalIp
Example 9.2. Required permissions for creating load balancer resources
compute.regionBackendServices.create
compute.regionBackendServices.get
compute.regionBackendServices.list
compute.regionBackendServices.update
compute.regionBackendServices.use
compute.targetPools.addInstance
compute.targetPools.create
compute.targetPools.get
compute.targetPools.list
compute.targetPools.removeInstance
compute.targetPools.use
Example 9.3. Required permissions for creating DNS resources
dns.changes.create
dns.changes.get
dns.managedZones.create
dns.managedZones.get
1390
dns.managedZones.list
dns.networks.bindPrivateDNSZone
dns.resourceRecordSets.create
dns.resourceRecordSets.list
Example 9.4. Required permissions for creating Service Account resources
iam.serviceAccountKeys.create
iam.serviceAccountKeys.delete
iam.serviceAccountKeys.get
iam.serviceAccountKeys.list
iam.serviceAccounts.actAs
iam.serviceAccounts.create
iam.serviceAccounts.delete
iam.serviceAccounts.get
iam.serviceAccounts.list
resourcemanager.projects.get
resourcemanager.projects.getIamPolicy
resourcemanager.projects.setIamPolicy
compute.disks.create
compute.disks.get
compute.disks.list
compute.disks.setLabels
compute.instanceGroups.create
compute.instanceGroups.delete
compute.instanceGroups.get
compute.instanceGroups.list
compute.instanceGroups.update
compute.instanceGroups.use
1391
compute.instances.create
compute.instances.delete
compute.instances.get
compute.instances.list
compute.instances.setLabels
compute.instances.setMetadata
compute.instances.setServiceAccount
compute.instances.setTags
compute.instances.use
compute.machineTypes.get
compute.machineTypes.list
Example 9.6. Required for creating storage resources
storage.buckets.create
storage.buckets.delete
storage.buckets.get
storage.buckets.list
storage.objects.create
storage.objects.delete
storage.objects.get
storage.objects.list
Example 9.7. Required permissions for creating health check resources
compute.healthChecks.create
compute.healthChecks.get
compute.healthChecks.list
compute.healthChecks.useReadOnly
compute.httpHealthChecks.create
compute.httpHealthChecks.get
compute.httpHealthChecks.list
1392
compute.httpHealthChecks.useReadOnly
Example 9.8. Required permissions to get GCP zone and region related information
compute.globalOperations.get
compute.regionOperations.get
compute.regions.list
compute.zoneOperations.get
compute.zones.get
compute.zones.list
Example 9.9. Required permissions for checking services and quotas
monitoring.timeSeries.list
serviceusage.quotas.get
serviceusage.services.list
iam.roles.get
Example 9.11. Optional Images permissions for installation
compute.images.list
compute.instances.getSerialPortOutput
compute.addresses.deleteInternal
compute.forwardingRules.delete
1393
compute.networks.delete
compute.routers.delete
compute.routes.list
compute.subnetworks.delete
Example 9.14. Required permissions for deleting load balancer resources
compute.regionBackendServices.delete
compute.targetPools.delete
Example 9.15. Required permissions for deleting DNS resources
dns.changes.create
dns.managedZones.delete
dns.resourceRecordSets.delete
Example 9.16. Required permissions for deleting Service Account resources
1394
compute.disks.delete
compute.disks.list
compute.instances.stop
Example 9.18. Required for deleting storage resources
storage.buckets.getIamPolicy
Example 9.19. Required permissions for deleting health check resources
compute.healthChecks.delete
compute.httpHealthChecks.delete
Example 9.20. Required Images permissions for deletion
compute.images.list
9.2.5.3. Required GCP permissions for shared VPC installations
When you are installing a cluster to a shared VPC, you must configure the service account for both the
host project and the service project. If you are not installing to a shared VPC, you can skip this section.
You must apply the minimum roles required for a standard installation as listed above, to the service
1395
project. Note that custom roles, and therefore fine-grained permissions, cannot be used in shared VPC
installations because GCP does not support adding the required permission
compute.organizations.administerXpn to custom roles.
In addition, the host project must apply one of the following configurations to the service account:
Example 9.21. Required permissions for creating firewalls in the host project
projects/<host-project>/roles/dns.networks.bindPrivateDNSZone
Example 9.22. Required minimal permissions
projects/<host-project>/roles/dns.networks.bindPrivateDNSZone
roles/compute.networkUser
9.2.6. Supported GCP regions

You can deploy an OpenShift Container Platform cluster to the following Google Cloud Platform (GCP)
regions:
asia-east1 (Changhua County, Taiwan)
asia-east2 (Hong Kong)
asia-northeast1 (Tokyo, Japan)
asia-northeast2 (Osaka, Japan)
asia-northeast3 (Seoul, South Korea)
asia-south1 (Mumbai, India)
asia-south2 (Delhi, India)
asia-southeast1 (Jurong West, Singapore)
asia-southeast2 (Jakarta, Indonesia)
australia-southeast1 (Sydney, Australia)
australia-southeast2 (Melbourne, Australia)
europe-central2 (Warsaw, Poland)
europe-north1 (Hamina, Finland)
europe-southwest1 (Madrid, Spain)
europe-west1 (St. Ghislain, Belgium)
1396
europe-west2 (London, England, UK)
europe-west3 (Frankfurt, Germany)
europe-west4 (Eemshaven, Netherlands)
europe-west6 (Zürich, Switzerland)
europe-west8 (Milan, Italy)
europe-west9 (Paris, France)
europe-west12 (Turin, Italy)
me-central1 (Doha, Qatar, Middle East)
me-west1 (Tel Aviv, Israel)
northamerica-northeast1 (Montréal, Québec, Canada)
northamerica-northeast2 (Toronto, Ontario, Canada)
southamerica-east1 (São Paulo, Brazil)
southamerica-west1 (Santiago, Chile)
us-central1 (Council Bluffs, Iowa, USA)
us-east1 (Moncks Corner, South Carolina, USA)
us-east4 (Ashburn, Northern Virginia, USA)
us-east5 (Columbus, Ohio)
us-south1 (Dallas, Texas)
us-west1 (The Dalles, Oregon, USA)
us-west2 (Los Angeles, California, USA)
us-west3 (Salt Lake City, Utah, USA)
us-west4 (Las Vegas, Nevada, USA)
NOTE
To determine which machine type instances are available by region and zone, see the
Google documentation.
9.2.7. Next steps

Install an OpenShift Container Platform cluster on GCP. You can install a customized cluster or
quickly install a cluster with default options.
9.3. INSTALLING A CLUSTER QUICKLY ON GCP
In OpenShift Container Platform version 4.15, you can install a cluster on Google Cloud Platform (GCP)
1397
that uses the default configuration options.
processes.
users.
You configured a GCP project to host the cluster.

IMPORTANT

authentication.
user.
IMPORTANT
1398
IMPORTANT
NOTE
Procedure
NOTE
ecdsa algorithm.
NOTE

task:
Example output
1399
Example output
Agent pid 31874
NOTE

Example output
Next steps
program.

for installation.
Prerequisites
Procedure
IMPORTANT
IMPORTANT
1400
IMPORTANT
components.

IMPORTANT
Prerequisites
cluster.
Procedure
1. Remove any existing GCP credentials that do not use the service account key for the GCP
account that you configured for your cluster and that are stored in the following locations:
The GOOGLE_CREDENTIALS, GOOGLE_CLOUD_KEYFILE_JSON, or

GCLOUD_KEYFILE_JSON environment variables
The ~/.gcp/osServiceAccount.json file
The gcloud cli default credentials
deployment:

--log-level=info 2
1401
Platform version.
NOTE

b. Select gcp as the platform to target.
c. If you have not configured the service account key for your GCP account on your host, you
must obtain it from GCP and paste the contents of the file or enter the absolute path to the
file.
d. Select the project ID to provision the cluster in. The default value is specified by the service
account that you configured.
e. Select the region to deploy the cluster to.
f. Select the base domain to deploy the cluster to. The base domain corresponds to the
g. Enter a descriptive name for your cluster. If you provide a name that is longer than 6
characters, only the first 6 characters will be used in the infrastructure ID that is generated
from the cluster name.
h. Paste the pull secret from Red Hat OpenShift Cluster Manager .
4. Optional: You can reduce the number of permissions for the service account that you used to
If you assigned the Owner role to your service account, you can remove that role and
replace it with the Viewer role.
If you included the Service Account Key Admin role, you can remove it.
Verification
1402
IMPORTANT
Example output
...
IMPORTANT

IMPORTANT

Procedure
1403
Portal.
$ tar xvf <file>

$ echo $PATH
Verification
$ oc <command>

Procedure
Portal.

C:\> path
Verification
C:\> oc <command>

Procedure
1404
Portal.
NOTE

$ echo $PATH
Verification
$ oc <command>

Prerequisites
Procedure
$ oc whoami
Example output
1405
system:admin

9.3.9. Next steps

9.4. INSTALLING A CLUSTER ON GCP WITH CUSTOMIZATIONS

the installation program provisions on Google Cloud Platform (GCP). To customize the installation, you
modify parameters in the install-config.yaml file before you install the cluster.
processes.
users.

1406
IMPORTANT

authentication.
user.
IMPORTANT
NOTE
Procedure
NOTE
1407
NOTE
ecdsa algorithm.
NOTE

task:
Example output
Agent pid 31874
NOTE

Example output
Next steps
1408
program.

for installation.
Prerequisites
Procedure
IMPORTANT
IMPORTANT
components.

You can customize the OpenShift Container Platform cluster you install on Google Cloud Platform
(GCP).
Prerequisites
1409
cluster.
Procedure
command:
NOTE

ii. Select gcp as the platform to target.
iii. If you have not configured the service account key for your GCP account on your
computer, you must obtain it from GCP and paste the contents of the file or enter the
absolute path to the file.
iv. Select the project ID to provision the cluster in. The default value is specified by the
service account that you configured.
v. Select the region to deploy the cluster to.
vi. Select the base domain to deploy the cluster to. The base domain corresponds to the
vii. Enter a descriptive name for your cluster.
1410
NOTE

more information, see "Installing a three-node cluster on GCP".
IMPORTANT

Installation configuration parameters for GCP

System Per Second
(IOPS)[2]

8.6 and later
[3]
1411
Optimizing storage
9.4.5.2. Tested instance types for GCP
The following Google Cloud Platform instance types have been tested with OpenShift Container
Platform.
Example 9.23. Machine series
A2
C2
C2D
C3
E2
M1
N1
N2
N2D
Tau T2D
9.4.5.3. Tested instance types for GCP on 64-bit ARM infrastructures
The following Google Cloud Platform (GCP) 64-bit ARM instance types have been tested with
Example 9.24. Machine series for 64-bit ARM machines
Tau T2A
9.4.5.4. Using custom machine types
Using a custom machine type to install a OpenShift Container Platform cluster is supported.
Consider the following when using a custom machine type:
Similar to predefined instance types, custom machine types must meet the minimum resource
requirements for control plane and compute machines. For more information, see "Minimum
resource requirements for cluster installation".
The name of the custom machine type must adhere to the following syntax:
custom-<number_of_cpus>-<amount_of_memory_in_mb>
For example, custom-6-20480.
1412
As part of the installation process, you specify the custom machine type in the install-config.yaml file.
Sample install-config.yaml file with a custom machine type
compute:
name: worker
platform:
gcp:
type: custom-6-20480
replicas: 2
controlPlane:
architecture: amd64
name: master
platform:
gcp:
replicas: 3
9.4.5.5. Enabling Shielded VMs
You can use Shielded VMs when installing your cluster. Shielded VMs have extra security features
including secure boot, firmware and integrity monitoring, and rootkit detection. For more information,
see Google’s documentation on Shielded VMs.
NOTE
Shielded VMs are currently not supported on clusters with 64-bit ARM infrastructures.
Prerequisites
Procedure
Use a text editor to edit the install-config.yaml file prior to deploying your cluster and add one
of the following stanzas:
a. To use shielded VMs for only control plane machines:
controlPlane:
platform:
gcp:
secureBoot: Enabled
b. To use shielded VMs for only compute machines:
compute:
- platform:
gcp:
secureBoot: Enabled
1413
c. To use shielded VMs for all machines:
platform:
gcp:
secureBoot: Enabled
9.4.5.6. Enabling Confidential VMs
You can use Confidential VMs when installing your cluster. Confidential VMs encrypt data while it is
being processed. For more information, see Google’s documentation on Confidential Computing. You
can enable Confidential VMs and Shielded VMs at the same time, although they are not dependent on
each other.
NOTE
Prerequisites
Procedure
a. To use confidential VMs for only control plane machines:
controlPlane:
platform:
gcp:
confidentialCompute: Enabled 1
type: n2d-standard-8 2
onHostMaintenance: Terminate 3
2 Specify a machine type that supports Confidential VMs. Confidential VMs require the
N2D or C2D series of machine types. For more information on supported machine
types, see Supported operating systems and machine types .
3 Specify the behavior of the VM during a host maintenance event, such as a hardware or
software update. For a machine that uses Confidential VM, this value must be set to
Terminate, which stops the VM. Confidential VMs do not support live VM migration.
b. To use confidential VMs for only compute machines:
compute:
- platform:
gcp:
1414
confidentialCompute: Enabled
type: n2d-standard-8
onHostMaintenance: Terminate
c. To use confidential VMs for all machines:
platform:
gcp:
9.4.5.7. Sample customized install-config.yaml file for GCP
IMPORTANT
apiVersion: v1
controlPlane: 3 4
name: master
platform:
gcp:
type: n2-standard-4
zones:
- us-central1-a
- us-central1-c
osDisk:
diskType: pd-ssd
diskSizeGB: 1024
encryptionKey: 6
kmsKey:
name: worker-key
keyRing: test-machine-keys
location: global
projectID: project-id
tags: 7
- control-plane-tag1
osImage: 8
project: example-project-name
name: example-image-name
replicas: 3
compute: 9 10
1415
name: worker
platform:
gcp:
type: n2-standard-4
zones:
- us-central1-a
- us-central1-c
osDisk:
diskType: pd-standard
diskSizeGB: 128
encryptionKey: 12
kmsKey:
name: worker-key
location: global
tags: 13
- compute-tag1
- compute-tag2
osImage: 14
replicas: 3
metadata:
networking:
clusterNetwork:
- cidr: 10.128.0.0/14
hostPrefix: 23
machineNetwork:
- cidr: 10.0.0.0/16
serviceNetwork:
- 172.30.0.0/16
platform:
gcp:
projectID: openshift-production 17
region: us-central1 18
tags: 19
- global-tag1
- global-tag2
osImage: 20
fips: false 22
1 15 17 18 21 Required. The installation program prompts you for this value.
1416
value.

IMPORTANT

accounts for the dramatically decreased machine performance. Use larger machine
types, such as n1-standard-8, for your machines if you disable simultaneous
multithreading.
6 12 Optional: The custom encryption key section to encrypt both virtual machines and persistent
volumes. Your default compute service account must have the permissions granted to use your
KMS key and have the correct IAM role assigned. The default service account name follows the
service-<project_number>@compute-system.iam.gserviceaccount.com pattern. For more
information about granting the correct permissions for your service account, see "Machine
management" → "Creating compute machine sets" → "Creating a compute machine set on GCP".
7 13 19 Optional: A set of network tags to apply to the control plane or compute machine sets. The
platform.gcp.defaultMachinePlatform.tags parameter will apply to both control plane and
compute machines. If the compute.platform.gcp.tags or controlPlane.platform.gcp.tags
parameters are set, they override the platform.gcp.defaultMachinePlatform.tags parameter.
8 14 20 Optional: A custom Red Hat Enterprise Linux CoreOS (RHCOS) that should be used to boot
control plane and compute machines. The project and name parameters under
platform.gcp.defaultMachinePlatform.osImage apply to both control plane and compute
machines. If the project and name parameters under controlPlane.platform.gcp.osImage or
compute.platform.gcp.osImage are set, they override the
platform.gcp.defaultMachinePlatform.osImage parameters.
value.
IMPORTANT
1417
IMPORTANT
architectures.
NOTE
process uses.
Enabling customer-managed encryption keys for a compute machine set
Prerequisites
NOTE
(169.254.169.254).
Procedure
apiVersion: v1
proxy:
1418
must be http.

destinations.
NOTE
NOTE
NOTE
1419
NOTE
created.
9.4.6. Managing user-defined labels and tags for GCP
IMPORTANT
Support for user-defined labels and tags for GCP is a Technology Preview feature only.
Google Cloud Platform (GCP) provides labels and tags that help to identify and organize the resources
created for a specific OpenShift Container Platform cluster, making them easier to manage.
You can define labels and tags for each GCP resource only during OpenShift Container Platform
IMPORTANT
User-defined labels and tags are not supported for OpenShift Container Platform
clusters upgraded to OpenShift Container Platform 4.15.
User-defined labels
User-defined labels and OpenShift Container Platform specific labels are applied only to resources
created by OpenShift Container Platform installation program and its core components such as:
GCP filestore CSI Driver Operator
GCP PD CSI Driver Operator
Image Registry Operator
Machine API provider for GCP
User-defined labels and OpenShift Container Platform specific labels are not applied on the resources
created by any other operators or the Kubernetes in-tree components that create resources, for
example, the Ingress load balancers.
User-defined labels and OpenShift Container Platform labels are available on the following GCP
resources:
Compute disk
Compute instance
Compute image
1420
Compute forwarding rule
DNS managed zone
Filestore instance
Storage bucket
Limitations to user-defined labels
Labels for ComputeAddress are supported in the GCP beta version. OpenShift Container
Platform does not add labels to the resource.
User-defined tags
User-defined tags are attached to resources created by the OpenShift Container Platform Image
Registry Operator and not on the resources created by any other Operators or the Kubernetes in-tree
components.
User-defined tags are available on the following GCP resources: * Storage bucket
Limitations to the user-defined tags
Tags will not be attached to the following items:
Control plane instances and storage buckets created by the installation program
Compute instances created by the Machine API provider for GCP
Filestore instance resources created by the GCP filestore CSI driver Operator
Compute disk and compute image resources created by the GCP PD CSI driver Operator
Tags are not supported for buckets located in the following regions:
us-east2
us-east3
Image Registry Operator does not throw any error but skips processing tags when the buckets
are created in the tags unsupported region.
Tags must not be restricted to particular service accounts, because Operators create and use
service accounts with minimal roles.
OpenShift Container Platform does not create any key and value resources of the tag.
OpenShift Container Platform specific tags are not added to any resource.
For more information about identifying the OrganizationID, see: OrganizationID
For more information about identifying the ProjectID, see: ProjectID
For more information about labels, see Labels Overview.
1421
For more information about tags, see Tags Overview.
9.4.6.1. Configuring user-defined labels and tags for GCP
Prerequisites
The installation program requires that a service account includes a TagUser role, so that the
program can create the OpenShift Container Platform cluster with defined tags at both
organization and project levels.
Procedure
Update the install-config.yaml file to define the list of desired labels and tags.
NOTE
Labels and tags are defined during the install-config.yaml creation phase, and
cannot be modified or updated with new labels and tags after cluster creation.
Sample install-config.yaml file
apiVersion: v1
featureSet: TechPreviewNoUpgrade
platform:
gcp:
userLabels: 1
- key: <label_key> 2
value: <label_value> 3
userTags: 4
- parentID: <OrganizationID/ProjectID> 5
key: <tag_key_short_name>
value: <tag_value_short_name>
1 Adds keys and values as labels to the resources created on GCP.
2 Defines the label name.
3 Defines the label content.
4 Adds keys and values as tags to the resources created on GCP.
5 The ID of the hierarchical resource where the tags are defined, at the organization or the
project level.
The following are the requirements for user-defined labels:
A label key and value must have a minimum of 1 character and can have a maximum of 63
characters.
A label key and value must contain only lowercase letters, numeric characters, underscore (_),
and dash (-).
A label key must start with a lowercase letter.
1422
You can configure a maximum of 32 labels per resource. Each resource can have a maximum of
64 labels, and 32 labels are reserved for internal use by OpenShift Container Platform.
The following are the requirements for user-defined tags:
Tag key and tag value must already exist. OpenShift Container Platform does not create the key
and the value.
A tag parentID can be either OrganizationID or ProjectID:
OrganizationID must consist of decimal numbers without leading zeros.
ProjectID must be 6 to 30 characters in length, that includes only lowercase letters,

numbers, and hyphens.
ProjectID must start with a letter, and cannot end with a hyphen.
A tag key must contain only uppercase and lowercase alphanumeric characters, hyphen (-),
underscore (_), and period ( .).
A tag value must contain only uppercase and lowercase alphanumeric characters, hyphen (-),
underscore (_), period (.), at sign ( @), percent sign (%), equals sign ( =), plus (+), colon (:),
comma (,), asterisk (*), pound sign ($), ampersand (&), parentheses (()), square braces ( []), curly
braces ({}), and space.
A tag key and value must begin and end with an alphanumeric character.
Tag value must be one of the pre-defined values for the key.
You can configure a maximum of 50 tags.
There should be no tag key defined with the same value as any of the existing tag keys that will
be inherited from the parent resource.
9.4.6.2. Querying user-defined labels and tags for GCP
After creating the OpenShift Container Platform cluster, you can access the list of the labels and tags
defined for the GCP resources in the infrastructures.config.openshift.io/cluster object as shown in
the following sample infrastructure.yaml file.
Sample infrastructure.yaml file
kind: Infrastructure
metadata:
name: cluster
spec:
platformSpec:
type: GCP
status:
infrastructureName: <cluster_id> 1
platform: GCP
platformStatus:
gcp:
resourceLabels:
- key: <label_key>
1423
value: <label_value>
resourceTags:
- key: <tag_key_short_name>
parentID: <OrganizationID/ProjectID>
value: <tag_value_short_name>
type: GCP
1 The cluster ID that is generated during cluster installation.
Along with the user-defined labels, resources have a label defined by the OpenShift Container Platform.
The format of the OpenShift Container Platform labels is kubernetes-io-cluster-<cluster_id>:owned.

IMPORTANT

Procedure
Portal.
$ tar xvf <file>

$ echo $PATH
Verification
$ oc <command>
1424
Procedure
Portal.

C:\> path
Verification
C:\> oc <command>

Procedure
Portal.
NOTE

$ echo $PATH
Verification
$ oc <command>
1425

components, follow the procedures in Configuring a GCP cluster to use short-term credentials .
namespace.
Procedure
1. Add the following granular permissions to the GCP account that the installation program uses:
Example 9.25. Required GCP permissions
compute.zones.list
dns.changes.create
dns.changes.get
1426
apiVersion: v1
# ...
command:

--included \ 1
metadata:
...
spec:
providerSpec:
kind: GCPProviderSpec
predefinedRoles:
- roles/storage.admin
1427
- roles/iam.serviceAccountUser
skipServiceCheck: true
...
metadata:
...
spec:
providerSpec:
...
secretRef:
...
apiVersion: v1
kind: Secret
metadata:
data:
service_account.json: <base64_encoded_gcp_service_account_file>
IMPORTANT
9.4.8.2. Configuring a GCP cluster to use short-term credentials
To install a cluster that is configured to use GCP Workload Identity, you must configure the CCO utility
and create the required GCP resources for your cluster.
NOTE
1428
Prerequisites
You have added one of the following authentication options to the GCP account that the
installation program uses:
The IAM Workload Identity Pool Adminrole.
The following granular permissions:

compute.projects.get
iam.googleapis.com/workloadIdentityPoolProviders.create
iam.googleapis.com/workloadIdentityPoolProviders.get
iam.googleapis.com/workloadIdentityPools.create
iam.googleapis.com/workloadIdentityPools.delete
iam.googleapis.com/workloadIdentityPools.get
iam.googleapis.com/workloadIdentityPools.undelete
iam.roles.create
iam.roles.delete
iam.roles.list
iam.roles.undelete
iam.roles.update
iam.serviceAccounts.getIamPolicy
iam.serviceAccounts.setIamPolicy
iam.workloadIdentityPoolProviders.get
iam.workloadIdentityPools.delete
1429
storage.buckets.get
storage.buckets.setIamPolicy
Procedure

NOTE

$ chmod 775 ccoctl
Verification
$ ccoctl --help
1430
Usage:
ccoctl [command]
Available Commands:
Flags:
9.4.8.2.2. Creating GCP resources with the Cloud Credential Operator utility
You can use the ccoctl gcp create-all command to automate the creation of GCP resources.
NOTE
Prerequisites
You must have:
Procedure

--included \ 1
1431
NOTE
command:
$ ccoctl gcp create-all \

--name=<name> \ 1
--region=<gcp_region> \ 2
--project=<gcp_project_id> \ 3
--credentials-requests-dir=<path_to_credentials_requests_directory> 4
1 Specify the user-defined name for all created GCP resources used for tracking.
2 Specify the GCP region in which cloud resources will be created.
3 Specify the GCP project ID in which cloud resources will be created.
4 Specify the directory containing the files of CredentialsRequest manifests to create GCP
service accounts.
NOTE
preview parameter.
Verification
Example output
openshift-cloud-controller-manager-gcp-ccm-cloud-credentials-credentials.yaml
openshift-cloud-credential-operator-cloud-credential-operator-gcp-ro-creds-credentials.yaml
openshift-cluster-api-capg-manager-bootstrap-credentials-credentials.yaml
openshift-cluster-csi-drivers-gcp-pd-cloud-credentials-credentials.yaml
openshift-machine-api-gcp-cloud-credentials-credentials.yaml
1432
You can verify that the IAM service accounts are created by querying GCP. For more
information, refer to GCP documentation on listing IAM service accounts.
Prerequisites
utility.
Procedure
compute.zones.list
dns.changes.create
dns.changes.get
apiVersion: v1
1433
# ...
command:
9.4.9. Using the GCP Marketplace offering

Using the GCP Marketplace offering lets you deploy an OpenShift Container Platform cluster, which is
billed on pay-per-use basis (hourly, per core) through GCP, while still being supported directly by
Red Hat.
By default, the installation program downloads and installs the Red Hat Enterprise Linux CoreOS
(RHCOS) image that is used to deploy compute machines. To deploy an OpenShift Container Platform
cluster using an RHCOS image from the GCP Marketplace, override the default behavior by modifying
the install-config.yaml file to reference the location of GCP Marketplace offer.
Prerequisites
Procedure
1. Edit the compute.platform.gcp.osImage parameters to specify the location of the GCP

Marketplace image:
Set the project parameter to redhat-marketplace-public
Set the name parameter to one of the following offers:
OpenShift Container Platform

redhat-coreos-ocp-413-x86-64-202305021736
OpenShift Platform Plus
redhat-coreos-opp-413-x86-64-202305021736
OpenShift Kubernetes Engine
redhat-coreos-oke-413-x86-64-202305021736
1434
Sample install-config.yaml file that specifies a GCP Marketplace image for compute
machines
apiVersion: v1
controlPlane:
# ...
compute:
platform:
gcp:
osImage:
project: redhat-marketplace-public
name: redhat-coreos-ocp-413-x86-64-202305021736
# ...

IMPORTANT
Prerequisites
cluster.
Procedure

deployment:

--log-level=info 2
1435

config.yaml file.
Verification
IMPORTANT
Example output
...
IMPORTANT
1436

Prerequisites
Procedure
$ oc whoami
Example output
system:admin

9.4.13. Next steps

1437
9.5. INSTALLING A CLUSTER ON GCP WITH NETWORK

CUSTOMIZATIONS
configuration on infrastructure that the installation program provisions on Google Cloud Platform
(GCP). By customizing your network configuration, your cluster can coexist with existing IP address
allocations in your environment and integrate with existing MTU and VXLAN configurations. To
customize the installation, you modify parameters in the install-config.yaml file before you install the
cluster.
processes.
users.

IMPORTANT

1438
authentication.
user.
IMPORTANT
NOTE
Procedure
NOTE
ecdsa algorithm.
NOTE
1439
NOTE

task:
Example output
Agent pid 31874
NOTE

Example output
Next steps
program.

for installation.
Prerequisites
Procedure
1440
IMPORTANT
IMPORTANT
components.

(GCP).
Prerequisites
cluster.
Procedure
command:
1441
NOTE

IMPORTANT

1442

System Per Second
(IOPS)[2]

8.6 and later
[3]
Optimizing storage
Platform.
A2
C2
C2D
C3
E2
M1
1443
N1
N2
N2D
Tau T2D
Tau T2A
compute:
name: worker
platform:
gcp:
replicas: 2
controlPlane:
architecture: amd64
name: master
platform:
gcp:
replicas: 3
1444
NOTE
Prerequisites
Procedure
controlPlane:
platform:
gcp:
secureBoot: Enabled
compute:
- platform:
gcp:
secureBoot: Enabled
platform:
gcp:
secureBoot: Enabled
each other.
NOTE
Prerequisites
1445
Procedure
controlPlane:
platform:
gcp:
compute:
- platform:
gcp:
platform:
gcp:
IMPORTANT
1446
apiVersion: v1
controlPlane: 3 4
name: master
platform:
gcp:
type: n2-standard-4
zones:
- us-central1-a
- us-central1-c
osDisk:
diskType: pd-ssd
diskSizeGB: 1024
encryptionKey: 6
kmsKey:
name: worker-key
location: global
tags: 7
osImage: 8
replicas: 3
compute: 9 10
name: worker
platform:
gcp:
type: n2-standard-4
zones:
- us-central1-a
- us-central1-c
osDisk:
diskSizeGB: 128
encryptionKey: 12
kmsKey:
name: worker-key
location: global
tags: 13
- compute-tag1
- compute-tag2
osImage: 14
replicas: 3
metadata:
1447
networking: 16
clusterNetwork:
- cidr: 10.128.0.0/14
hostPrefix: 23
machineNetwork:
- cidr: 10.0.0.0/16
serviceNetwork:
- 172.30.0.0/16
platform:
gcp:
tags: 20
- global-tag1
- global-tag2
osImage: 21
fips: false 23
1 15 18 19 22 Required. The installation program prompts you for this value.
default value.

IMPORTANT

multithreading.
1448
value.
IMPORTANT
architectures.
NOTE
process uses.

Prerequisites
NOTE
1449
NOTE
(169.254.169.254).
Procedure
apiVersion: v1
proxy:
must be http.

destinations.
NOTE
1450
NOTE
NOTE
created.

IMPORTANT

Procedure
Portal.
$ tar xvf <file>

$ echo $PATH
Verification
1451
$ oc <command>

Procedure
Portal.

C:\> path
Verification
C:\> oc <command>

Procedure
Portal.
NOTE

$ echo $PATH
1452
Verification
$ oc <command>

namespace.
Procedure
compute.zones.list
dns.changes.create
dns.changes.get
1453
apiVersion: v1
# ...
command:

--included \ 1
metadata:
...
spec:
providerSpec:
1454
predefinedRoles:
...
metadata:
...
spec:
providerSpec:
...
secretRef:
...
apiVersion: v1
kind: Secret
metadata:
data:
IMPORTANT
NOTE
1455
NOTE
Prerequisites

iam.roles.create
iam.roles.delete
iam.roles.list
iam.roles.undelete
iam.roles.update
1456
storage.buckets.get
Procedure

NOTE

$ chmod 775 ccoctl
Verification
$ ccoctl --help
1457
Usage:
ccoctl [command]
Available Commands:
Flags:
NOTE
Prerequisites
You must have:
Procedure

--included \ 1
1458
NOTE
command:

--name=<name> \ 1
service accounts.
NOTE
preview parameter.
Verification
Example output
1459
Prerequisites
utility.
Procedure
compute.zones.list
dns.changes.create
dns.changes.get
1460
apiVersion: v1
# ...
command:

Phase 1
NOTE

NIC resides in.
IMPORTANT
cluster.
1461
Phase 2

the cluster.
IMPORTANT

Prerequisites
Procedure
kind: Network
metadata:
name: cluster
spec:
kind: Network
metadata:
name: cluster
spec:
defaultNetwork:
1462
ipsecConfig:
mode: Full


clusterNetwork
serviceNetwork
defaultNetwork.type
spec:
clusterNetwork:
- cidr: 10.128.0.0/19
hostPrefix: 23
- cidr: 10.128.32.0/19
hostPrefix: 23
1463
spec:
serviceNetwork:
- 172.30.0.0/14

manifest file.
work


NOTE


network plugin.

1464
detected MTU.


value to 1400.

configuration.


NOTE

1465

t network
infrastructure
overlaps with the
100.64.0.0/16
IPv4 subnet, you
can specify a
different IP
address range for
internal use by
OVN-Kubernetes.
You must ensure
that the IP address
range does not
overlap with any
other subnet used
by your OpenShift
Container
Platform
address range
must be larger
than the maximum
number of nodes
that can be added
to the cluster. For
example, if the
clusterNetwork.
cidr value is
10.128.0.0/14
and the
clusterNetwork.
hostPrefix value
is /23, then the
maximum number
of nodes is 2^(23-
14)=512 .
This field cannot

be changed after
installation.
1466

t network
infrastructure
overlaps with the
fd98::/48 IPv6
subnet, you can
specify a different
IP address range
for internal use by
OVN-Kubernetes.
You must ensure
that the IP address
range does not
overlap with any
other subnet used
by your OpenShift
Container
Platform
address range
must be larger
than the maximum
number of nodes
that can be added
to the cluster.
This field cannot

be changed after
installation.

50000000 or 50 MB.
1467
libc
host.
udp:<host>:<port>
unix:<file>
null

1468


external hosts.

defaultNetwork:
type: OVNKubernetes
mtu: 1400
genevePort: 6081
ipsecConfig:
mode: Full
IMPORTANT

documentation.
NOTE
1469

kubeProxyConfig:
proxyArguments:
- 0s

IMPORTANT
Prerequisites
cluster.
Procedure

deployment:

--log-level=info 2
1470

config.yaml file.
Verification
IMPORTANT
Example output
...
IMPORTANT
1471

Prerequisites
Procedure
$ oc whoami
Example output
system:admin

9.5.15. Next steps

1472
9.6. INSTALLING A CLUSTER ON GCP IN A RESTRICTED NETWORK

In OpenShift Container Platform 4.15, you can install a cluster on Google Cloud Platform (GCP) in a
restricted network by creating an internal mirror of the installation release content on an existing Google
Virtual Private Cloud (VPC).
IMPORTANT
You can install an OpenShift Container Platform cluster by using mirrored installation
release content, but your cluster will require internet access to use the GCP APIs.
processes.
users.
IMPORTANT
You have an existing VPC in GCP. While installing a cluster in a restricted network that uses
installer-provisioned infrastructure, you cannot use the installer-provisioned VPC. You must use
a user-provisioned VPC that satisfies one of the following requirements:
Contains the mirror registry
Has firewall rules or a peering connection to access the mirror registry hosted elsewhere
While you might need to grant access to more sites, you must grant access to
*.googleapis.com and accounts.google.com.

1473
your restrictions.


authentication.
user.
IMPORTANT
NOTE
1474
Procedure
NOTE
ecdsa algorithm.
NOTE

task:
Example output
Agent pid 31874
NOTE

1475
Example output
Next steps
program.

(GCP).
Prerequisites
creation.
Procedure
command:
1476
NOTE


c. Define the network and subnets for the VPC to install the cluster in under the parent
platform.gcp field:
network: <existing_vpc>
controlPlaneSubnet: <control_plane_subnet>
computeSubnet: <compute_subnet>
1477
For platform.gcp.network, specify the name for the existing Google VPC. For
platform.gcp.controlPlaneSubnet and platform.gcp.computeSubnet, specify the existing
subnets to deploy the control plane machines and compute machines, respectively.
- mirrors:
- mirrors:
creation.
publish: Internal
IMPORTANT


System Per Second
(IOPS)[2]
1478

System Per Second
(IOPS)[2]

8.6 and later
[3]
Optimizing storage
Platform.
A2
C2
C2D
C3
E2
M1
N1
N2
N2D
1479
Tau T2D
Tau T2A
compute:
name: worker
platform:
gcp:
replicas: 2
controlPlane:
architecture: amd64
name: master
platform:
gcp:
replicas: 3
1480
NOTE
Prerequisites
Procedure
controlPlane:
platform:
gcp:
secureBoot: Enabled
compute:
- platform:
gcp:
secureBoot: Enabled
platform:
gcp:
secureBoot: Enabled
each other.
NOTE
Prerequisites
Procedure
1481
controlPlane:
platform:
gcp:
compute:
- platform:
gcp:
platform:
gcp:
IMPORTANT
apiVersion: v1
controlPlane: 3 4
1482
name: master
platform:
gcp:
type: n2-standard-4
zones:
- us-central1-a
- us-central1-c
osDisk:
diskType: pd-ssd
diskSizeGB: 1024
encryptionKey: 6
kmsKey:
name: worker-key
location: global
tags: 7
osImage: 8
replicas: 3
compute: 9 10
name: worker
platform:
gcp:
type: n2-standard-4
zones:
- us-central1-a
- us-central1-c
osDisk:
diskSizeGB: 128
encryptionKey: 12
kmsKey:
name: worker-key
location: global
tags: 13
- compute-tag1
- compute-tag2
osImage: 14
replicas: 3
metadata:
networking:
clusterNetwork:
- cidr: 10.128.0.0/14
hostPrefix: 23
machineNetwork:
1483
- cidr: 10.0.0.0/16
serviceNetwork:
- 172.30.0.0/16
platform:
gcp:
tags: 19
- global-tag1
- global-tag2
osImage: 20
network: existing_vpc 21
fips: false 25
- mirrors:
- mirrors:
value.

IMPORTANT
1484
IMPORTANT

multithreading.
value.
21 Specify the name of an existing VPC.
22 Specify the name of the existing subnet to deploy the control plane machines to. The subnet must
belong to the VPC that you specified.
23 Specify the name of the existing subnet to deploy the compute machines to. The subnet must
registry uses to serve content. For example, registry.example.com or
IMPORTANT
architectures.
1485
NOTE
process uses.
repository.
9.6.5.8. Create an Ingress Controller with global access on GCP
You can create an Ingress Controller that has global access to a Google Cloud Platform (GCP) cluster.
Global access is only available to Ingress Controllers using internal load balancers.
Prerequisites
You created the install-config.yaml and complete any modifications to it.
Procedure
Create an Ingress Controller with global access on a new GCP cluster.
1. Change to the directory that contains the installation program and create a manifest file:

shown:
Example output
1486

Sample clientAccess configuration to Global
metadata:
name: default
spec:
loadBalancer:
providerParameters:
gcp:
clientAccess: Global 1
type: GCP
scope: Internal 2
1 Set gcp.clientAccess to Global.
2 Global access is only available to Ingress Controllers using internal load balancers.
Prerequisites
NOTE
(169.254.169.254).
Procedure
1487
apiVersion: v1
proxy:
must be http.

destinations.
NOTE
NOTE
NOTE
1488
NOTE
created.

IMPORTANT

Procedure
Portal.
$ tar xvf <file>

$ echo $PATH
Verification
$ oc <command>

Procedure
Portal.
1489

C:\> path
Verification
C:\> oc <command>

Procedure
Portal.
NOTE

$ echo $PATH
Verification
$ oc <command>

1490
namespace.
Procedure
compute.zones.list
dns.changes.create
dns.changes.get
apiVersion: v1
# ...
command:
1491

--included \ 1
metadata:
...
spec:
providerSpec:
predefinedRoles:
...
1492
metadata:
...
spec:
providerSpec:
...
secretRef:
...
apiVersion: v1
kind: Secret
metadata:
data:
IMPORTANT
NOTE
Prerequisites
1493

iam.roles.create
iam.roles.delete
iam.roles.list
iam.roles.undelete
iam.roles.update
storage.buckets.get
1494
Procedure

NOTE

$ chmod 775 ccoctl
Verification
$ ccoctl --help
Usage:
ccoctl [command]
Available Commands:
1495
Flags:
NOTE
Prerequisites
You must have:
Procedure

--included \ 1
NOTE
1496
command:

--name=<name> \ 1
service accounts.
NOTE
preview parameter.
Verification
Example output
Prerequisites
1497
utility.
Procedure
compute.zones.list
dns.changes.create
dns.changes.get
apiVersion: v1
# ...
command:
1498

IMPORTANT
Prerequisites
cluster.
Procedure

deployment:

--log-level=info 2

config.yaml file.
1499
Verification
IMPORTANT
Example output
...
IMPORTANT

1500
Prerequisites
Procedure
$ oc whoami
Example output
system:admin

Procedure
OperatorHub object:

TIP

1501
9.6.12. Next steps

9.7. INSTALLING A CLUSTER ON GCP INTO AN EXISTING VPC

In OpenShift Container Platform version 4.15, you can install a cluster into an existing Virtual Private
Cloud (VPC) on Google Cloud Platform (GCP). The installation program provisions the rest of the
required infrastructure, which you can further customize. To customize the installation, you modify
processes.
users.

In OpenShift Container Platform 4.15, you can deploy a cluster into existing subnets in an existing Virtual
Private Cloud (VPC) in Google Cloud Platform (GCP). By deploying OpenShift Container Platform into
an existing GCP VPC, you might be able to avoid limit constraints in new accounts or more easily abide
by the operational constraints that your company’s guidelines set. If you cannot obtain the infrastructure
creation permissions that are required to create the VPC yourself, use this installation option. You must
configure networking for the subnets.
The union of the VPC CIDR block and the machine network CIDR must be non-empty. The subnets must
be within the machine network.
The installation program does not create the following components:
1502
NAT gateways
Subnets
Route tables
VPC network
NOTE
data:
You provide one subnet for control-plane machines and one subnet for compute machines.
The subnet’s CIDRs belong to the machine CIDR that you specified.
Some individuals can create different resource in your clouds than others. For example, you might be
able to create application-specific items, like instances, buckets, and load balancers, but not
networking-related components such as VPCs, subnets, or ingress rules.

1503
IMPORTANT

authentication.
user.
IMPORTANT
NOTE
Procedure
NOTE
1504
NOTE
ecdsa algorithm.
NOTE

task:
Example output
Agent pid 31874
NOTE

Example output
Next steps
1505
program.

for installation.
Prerequisites
Procedure
IMPORTANT
IMPORTANT
components.

(GCP).
Prerequisites
1506
cluster.
Procedure
command:
NOTE

1507
IMPORTANT


System Per Second
(IOPS)[2]

8.6 and later
[3]
Optimizing storage
1508
Platform.
A2
C2
C2D
C3
E2
M1
N1
N2
N2D
Tau T2D
Tau T2A
1509
compute:
name: worker
platform:
gcp:
replicas: 2
controlPlane:
architecture: amd64
name: master
platform:
gcp:
replicas: 3
NOTE
Prerequisites
Procedure
controlPlane:
platform:
gcp:
secureBoot: Enabled
compute:
- platform:
gcp:
secureBoot: Enabled
platform:
gcp:
1510
secureBoot: Enabled
each other.
NOTE
Prerequisites
Procedure
controlPlane:
platform:
gcp:
compute:
- platform:
gcp:
1511
platform:
gcp:
IMPORTANT
apiVersion: v1
controlPlane: 3 4
name: master
platform:
gcp:
type: n2-standard-4
zones:
- us-central1-a
- us-central1-c
osDisk:
diskType: pd-ssd
diskSizeGB: 1024
encryptionKey: 6
kmsKey:
name: worker-key
location: global
tags: 7
osImage: 8
replicas: 3
compute: 9 10
name: worker
platform:
gcp:
type: n2-standard-4
zones:
- us-central1-a
1512
- us-central1-c
osDisk:
diskSizeGB: 128
encryptionKey: 12
kmsKey:
name: worker-key
location: global
tags: 13
- compute-tag1
- compute-tag2
osImage: 14
replicas: 3
metadata:
networking:
clusterNetwork:
- cidr: 10.128.0.0/14
hostPrefix: 23
machineNetwork:
- cidr: 10.0.0.0/16
serviceNetwork:
- 172.30.0.0/16
platform:
gcp:
tags: 19
- global-tag1
- global-tag2
osImage: 20
fips: false 25
1 15 17 18 24Required. The installation program prompts you for this value.
1513

IMPORTANT

multithreading.
value.
IMPORTANT
1514
IMPORTANT
NOTE
process uses.
Prerequisites
Procedure

shown:
1515
Example output

metadata:
name: default
spec:
loadBalancer:
providerParameters:
gcp:
type: GCP
scope: Internal 2

Prerequisites
NOTE
1516
NOTE
(169.254.169.254).
Procedure
apiVersion: v1
proxy:
must be http.

destinations.
NOTE
1517
NOTE
NOTE
created.

IMPORTANT

Procedure
Portal.
$ tar xvf <file>

$ echo $PATH
Verification
1518
$ oc <command>

Procedure
Portal.

C:\> path
Verification
C:\> oc <command>

Procedure
Portal.
NOTE

$ echo $PATH
Verification
1519
Verification
$ oc <command>

namespace.
Procedure
compute.zones.list
dns.changes.create
dns.changes.get
1520
apiVersion: v1
# ...
command:

--included \ 1
metadata:
...
spec:
providerSpec:
1521
predefinedRoles:
...
metadata:
...
spec:
providerSpec:
...
secretRef:
...
apiVersion: v1
kind: Secret
metadata:
data:
IMPORTANT
NOTE
1522
NOTE
Prerequisites

iam.roles.create
iam.roles.delete
iam.roles.list
iam.roles.undelete
iam.roles.update
1523
storage.buckets.get
Procedure

NOTE

$ chmod 775 ccoctl
Verification
$ ccoctl --help
1524
Usage:
ccoctl [command]
Available Commands:
Flags:
NOTE
Prerequisites
You must have:
Procedure

--included \ 1
1525
NOTE
command:

--name=<name> \ 1
service accounts.
NOTE
preview parameter.
Verification
Example output
1526
Prerequisites
utility.
Procedure
compute.zones.list
dns.changes.create
dns.changes.get
1527
apiVersion: v1
# ...
command:

IMPORTANT
Prerequisites
cluster.
Procedure

1528
deployment:

--log-level=info 2

config.yaml file.
Verification
IMPORTANT
Example output
...
IMPORTANT
1529
IMPORTANT

Prerequisites
Procedure
$ oc whoami
Example output
system:admin
1530
9.7.13. Next steps

9.8. INSTALLING A CLUSTER ON GCP INTO A SHARED VPC

In OpenShift Container Platform version 4.15, you can install a cluster into a shared Virtual Private Cloud
(VPC) on Google Cloud Platform (GCP). In this installation method, the cluster is configured to use a
VPC from a different GCP project. A shared VPC enables an organization to connect resources from
multiple projects to a common VPC network. You can communicate within the organization securely and
efficiently by using internal IP addresses from that network. For more information about shared VPC,
see Shared VPC overview in the GCP documentation .
The installation program provisions the rest of the required infrastructure, which you can further
customize. To customize the installation, you modify parameters in the install-config.yaml file before
you install the cluster.
processes.
users.
You have a GCP host project which contains a shared VPC network.
You configured a GCP project to host the cluster. This project, known as the service project,
must be attached to the host project. For more information, see Attaching service projects in
the GCP documentation.
You have a GCP service account that has the required GCP permissions in both the host and
service projects.

1531
IMPORTANT

authentication.
user.
IMPORTANT
NOTE
Procedure
1532
NOTE
ecdsa algorithm.
NOTE

task:
Example output
Agent pid 31874
NOTE

Example output
Next steps
1533
program.

for installation.
Prerequisites
Procedure
IMPORTANT
IMPORTANT
components.
9.8.5. Creating the installation files for GCP

To install OpenShift Container Platform on Google Cloud Platform (GCP) into a shared VPC, you must
generate the install-config.yaml file and modify it so that the cluster uses the correct VPC networks,
DNS zones, and project names.
1534
Prerequisites
recovery.
for your cluster.
Procedure
IMPORTANT
NOTE
IMPORTANT
NOTE
1535
NOTE
Prerequisites
Procedure
controlPlane:
platform:
gcp:
secureBoot: Enabled
compute:
- platform:
gcp:
secureBoot: Enabled
platform:
gcp:
secureBoot: Enabled
each other.
NOTE
Prerequisites
Procedure
1536
controlPlane:
platform:
gcp:
compute:
- platform:
gcp:
platform:
gcp:
9.8.5.4. Sample customized install-config.yaml file for shared VPC installation
There are several configuration parameters which are required to install OpenShift Container Platform
on GCP using a shared VPC. The following is a sample install-config.yaml file which demonstrates
these fields.
IMPORTANT
This sample YAML file is provided for reference only. You must modify this file with the
correct values for your environment and cluster.
apiVersion: v1
credentialsMode: Passthrough 1
metadata:
1537
name: cluster_name
platform:
gcp:
computeSubnet: shared-vpc-subnet-1 2
controlPlaneSubnet: shared-vpc-subnet-2 3
network: shared-vpc 4
networkProjectID: host-project-name 5
projectID: service-project-name 6
region: us-east1
tags: 7
- global-tag1
controlPlane:
name: master
platform:
gcp:
tags: 8
type: n2-standard-4
zones:
- us-central1-a
- us-central1-c
replicas: 3
compute:
- name: worker
platform:
gcp:
tags: 9
- compute-tag1
type: n2-standard-4
zones:
- us-central1-a
- us-central1-c
replicas: 3
networking:
clusterNetwork:
- cidr: 10.128.0.0/14
hostPrefix: 23
machineNetwork:
- cidr: 10.0.0.0/16
1 credentialsMode must be set to Passthrough or Manual. See the "Prerequisites" section for the
required GCP permissions that your service account must have.
2 The name of the subnet in the shared VPC for compute machines to use.
3 The name of the subnet in the shared VPC for control plane machines to use.
4 The name of the shared VPC.
5 The name of the host project where the shared VPC exists.
6 The name of the GCP project where you want to install the cluster.
1538
7 8 9 Optional. One or more network tags to apply to compute machines, control plane machines, or
all machines.
Prerequisites
NOTE
(169.254.169.254).
Procedure
apiVersion: v1
proxy:
must be http.
1539
NOTE
NOTE
NOTE
created.

IMPORTANT

Procedure
Portal.
1540
$ tar xvf <file>

$ echo $PATH
Verification
$ oc <command>

Procedure
Portal.

C:\> path
Verification
C:\> oc <command>

Procedure
Portal.
1541
NOTE

$ echo $PATH
Verification
$ oc <command>

namespace.
Procedure
compute.zones.list
dns.changes.create
dns.changes.get
1542
apiVersion: v1
# ...
command:

--included \ 1
1543
metadata:
...
spec:
providerSpec:
predefinedRoles:
...
metadata:
...
spec:
providerSpec:
...
secretRef:
...
apiVersion: v1
kind: Secret
metadata:
data:
IMPORTANT
1544
NOTE
Prerequisites

iam.roles.create
iam.roles.delete
iam.roles.list
iam.roles.undelete
iam.roles.update
1545
storage.buckets.get
Procedure

NOTE

1546
$ chmod 775 ccoctl
Verification
$ ccoctl --help
Usage:
ccoctl [command]
Available Commands:
Flags:
NOTE
Prerequisites
You must have:
Procedure
1547

--included \ 1
NOTE
command:

--name=<name> \ 1
service accounts.
NOTE
preview parameter.
Verification
1548
Example output
Prerequisites
utility.
Procedure
compute.zones.list
dns.changes.create
dns.changes.get
1549
apiVersion: v1
# ...
command:

IMPORTANT
Prerequisites
cluster.
1550
Procedure

deployment:

--log-level=info 2

config.yaml file.
Verification
IMPORTANT
Example output
...
1551
IMPORTANT

Prerequisites
Procedure
$ oc whoami
Example output
system:admin
1552
9.8.11. Next steps

9.9. INSTALLING A PRIVATE CLUSTER ON GCP

In OpenShift Container Platform version 4.15, you can install a private cluster into an existing VPC on
Google Cloud Platform (GCP). The installation program provisions the rest of the required
processes.
users.

internet.
IMPORTANT
1553
9.9.2.1. Private clusters in GCP
To create a private cluster on Google Cloud Platform (GCP), you must provide an existing private VPC
only internal traffic.
The cluster still requires access to internet to access the GCP APIs.
Public subnets
Public network load balancers, which support public ingress
A public DNS zone that matches the baseDomain for the cluster
The installation program does use the baseDomain that you specify to create a private DNS zone and
the required records for the cluster. The cluster is configured so that the Operators do not create public
records for the cluster and all cluster machines are placed in the private subnets that you specify.
Because it is not possible to limit access to external load balancers based on source tags, the private
cluster uses only internal load balancers to allow access to internal instances.
The internal load balancer relies on instance groups rather than the target pools that the network load
balancers use. The installation program creates instance groups for each zone, even if there is no
instance in that group.
The cluster IP address is internal only.
One forwarding rule manages both the Kubernetes API and machine config server ports.
The backend service is comprised of each zone’s instance group and, while it exists, the
bootstrap instance group.
The firewall uses a single rule that is based on only internal source ranges.
No health check for the Machine config server, /healthz, runs because of a difference in load balancer
1554
functionality. Two internal load balancers cannot share a single IP address, but two network load
balancers can share a single external IP address. Instead, the health of an instance is determined entirely
by the /readyz check on port 6443.

In OpenShift Container Platform 4.15, you can deploy a cluster into an existing VPC in Google Cloud
Platform (GCP). If you do, you must also use existing subnets within the VPC and routing rules.
By deploying OpenShift Container Platform into an existing GCP VPC, you might be able to avoid limit
constraints in new accounts or more easily abide by the operational constraints that your company’s
guidelines set. This is a good option to use if you cannot obtain the infrastructure creation permissions
that are required to create the VPC yourself.
The installation program will no longer create the following components:
VPC
Subnets
Cloud router
Cloud NAT
NAT IP addresses
the cluster to use. The installation program cannot subdivide network ranges for the cluster to use, set
route tables for the subnets, or set VPC options like DHCP, so you must do so before you install the
cluster.
Your VPC and subnets must meet the following characteristics:
The VPC must be in the same GCP project that you deploy the OpenShift Container Platform
cluster to.
To allow access to the internet from the control plane and compute machines, you must
configure cloud NAT on the subnets to allow egress to it. These machines do not have a public
address. Even if you do not require access to the internet, you must allow egress to the VPC
network to obtain the installation program and images. Because multiple cloud NATs cannot be
configured on the shared subnets, the installation program cannot configure it.
data:
All the subnets that you specify exist and belong to the VPC that you specified.
The subnet CIDRs belong to the machine CIDR.
You must provide a subnet to deploy the cluster control plane and compute machines to. You
can use the same subnet for both machine types.
If you destroy a cluster that uses an existing VPC, the VPC is not deleted.
1555
items, like instances, buckets, and load balancers, but not networking-related components such as VPCs,
subnets, or Ingress rules.
The GCP credentials that you use when you create your cluster do not need the networking permissions
resources that the machines within the cluster require, such as load balancers, security groups, storage,
and nodes.
preserved by firewall rules that reference the machines in your cluster by the cluster’s infrastructure ID.
Only traffic within the cluster is allowed.
If you deploy multiple clusters to the same VPC, the following components might share access between
clusters:
The API, which is globally available with an external publishing strategy or available throughout
the network in an internal publishing strategy
Debugging tools, such as ports on VM instances that are open to the machine CIDR for SSH and
ICMP access

IMPORTANT
1556
authentication.
user.
IMPORTANT
NOTE
Procedure
NOTE
ecdsa algorithm.
1557
NOTE

task:
Example output
Agent pid 31874
NOTE

Example output
Next steps
program.

for installation.
Prerequisites
Procedure
1558
IMPORTANT
IMPORTANT
components.

Prerequisites
recovery.
for your cluster.
Procedure
IMPORTANT
1559
IMPORTANT
NOTE
IMPORTANT

System Per Second
(IOPS)[2]

8.6 and later
[3]
1560
Optimizing storage
Platform.
A2
C2
C2D
C3
E2
M1
N1
N2
N2D
Tau T2D
Tau T2A
1561
compute:
name: worker
platform:
gcp:
replicas: 2
controlPlane:
architecture: amd64
name: master
platform:
gcp:
replicas: 3
NOTE
Prerequisites
Procedure
1562
controlPlane:
platform:
gcp:
secureBoot: Enabled
compute:
- platform:
gcp:
secureBoot: Enabled
platform:
gcp:
secureBoot: Enabled
each other.
NOTE
Prerequisites
Procedure
controlPlane:
platform:
gcp:
1563
compute:
- platform:
gcp:
platform:
gcp:
IMPORTANT
apiVersion: v1
controlPlane: 3 4
name: master
platform:
gcp:
type: n2-standard-4
zones:
- us-central1-a
- us-central1-c
osDisk:
diskType: pd-ssd
diskSizeGB: 1024
encryptionKey: 6
kmsKey:
name: worker-key
location: global
1564
tags: 7
osImage: 8
replicas: 3
compute: 9 10
name: worker
platform:
gcp:
type: n2-standard-4
zones:
- us-central1-a
- us-central1-c
osDisk:
diskSizeGB: 128
encryptionKey: 12
kmsKey:
name: worker-key
location: global
tags: 13
- compute-tag1
- compute-tag2
osImage: 14
replicas: 3
metadata:
networking:
clusterNetwork:
- cidr: 10.128.0.0/14
hostPrefix: 23
machineNetwork:
- cidr: 10.0.0.0/16
serviceNetwork:
- 172.30.0.0/16
platform:
gcp:
tags: 19
- global-tag1
- global-tag2
osImage: 20
1565
fips: false 25
1 15 17 18 24Required. The installation program prompts you for this value.
value.

IMPORTANT

multithreading.
16
The cluster network plugin to install. The default value OVNKubernetes is the only supported
1566
The cluster network plugin to install. The default value OVNKubernetes is the only supported
value.
IMPORTANT
NOTE
process uses.
Prerequisites
Procedure
1567

shown:
Example output

metadata:
name: default
spec:
loadBalancer:
providerParameters:
gcp:
type: GCP
scope: Internal 2

1568
Prerequisites
NOTE
(169.254.169.254).
Procedure
apiVersion: v1
proxy:
must be http.

destinations.
1569
NOTE
NOTE
NOTE
created.

IMPORTANT

Procedure
Portal.
1570
$ tar xvf <file>

$ echo $PATH
Verification
$ oc <command>

Procedure
Portal.

C:\> path
Verification
C:\> oc <command>

Procedure
Portal.
1571
NOTE

$ echo $PATH
Verification
$ oc <command>

project
namespace.
Procedure
compute.zones.list
dns.changes.create
dns.changes.get
1572
apiVersion: v1
# ...
command:

--included \ 1
1573
metadata:
...
spec:
providerSpec:
predefinedRoles:
...
metadata:
...
spec:
providerSpec:
...
secretRef:
...
apiVersion: v1
kind: Secret
metadata:
data:
IMPORTANT
1574
IMPORTANT
NOTE
Prerequisites

iam.roles.create
iam.roles.delete
iam.roles.list
iam.roles.undelete
iam.roles.update
1575
storage.buckets.get
Procedure

NOTE

1576
$ chmod 775 ccoctl
Verification
$ ccoctl --help
Usage:
ccoctl [command]
Available Commands:
Flags:
NOTE
Prerequisites
You must have:
Procedure
1577

--included \ 1
NOTE
command:

--name=<name> \ 1
service accounts.
NOTE
preview parameter.
1578
Verification
Example output
Prerequisites
utility.
Procedure
compute.zones.list
dns.changes.create
dns.changes.get
1579
apiVersion: v1
# ...
command:

IMPORTANT
Prerequisites
1580
cluster.
Procedure

deployment:

--log-level=info 2

config.yaml file.
Verification
IMPORTANT
Example output
...
1581

IMPORTANT

Prerequisites
Procedure
$ oc whoami
Example output
system:admin
1582

9.9.14. Next steps

9.10. INSTALLING A CLUSTER ON USER-PROVISIONED

INFRASTRUCTURE IN GCP BY USING DEPLOYMENT MANAGER
TEMPLATES
that uses infrastructure that you provide.
The steps for performing a user-provided infrastructure install are outlined here. Several Deployment
Manager templates are provided to assist in completing these steps or to help model your own. You are
also free to create the required resources through other methods.
IMPORTANT

Deployment Manager templates are provided to assist in completing these steps or to
help model your own. You are also free to create the required resources through other
processes.
users.
1583
NOTE
9.10.2. Certificate signing requests management

and approving them.

IMPORTANT
9.10.4. Configuring your GCP project

9.10.4.1. Creating a GCP project
1584
Procedure
IMPORTANT
9.10.4.2. Enabling API services in GCP
Prerequisites
Procedure
GCP documentation.

(IAM) API
Cloud Deployment Manager V2 API deploymentmanager.googleapis.com
1585
9.10.4.3. Configuring DNS for GCP
Procedure
NOTE
Google Domains.
9.10.4.4. GCP account limits
1586
Platform cluster.

bootstrap
Firewall rules Networking Global 11 1
Networks Networking Global 1 0
Routers Networking Global 1 0
Routes Networking Global 2 0
Target pools Networking Global 2 0
NOTE
asia-east2
asia-northeast2
asia-south1
1587
europe-north1
europe-west2
europe-west3
europe-west6
southamerica-east1
us-west2
9.10.4.5. Creating a service account in GCP
Prerequisites
Procedure
NOTE
create the cluster.
NOTE
1588
NOTE
following lists:
Compute Admin
Role Administrator
Security Admin
Storage Admin
DNS Administrator
Required roles for user-provisioned GCP infrastructure
Deployment Manager Editor
use:
Account Roles
1589
Account Roles
roles/storage.admin
roles/storage.admin
9.10.4.7. Required GCP permissions for user-provisioned infrastructure
custom roles with the necessary permissions. The following permissions are required for the user-
1590
compute.routers.get
compute.routes.list
dns.changes.create
dns.changes.get
1591
dns.resourceRecordSets.update
compute.disks.get
compute.disks.list
1592
storage.buckets.get
storage.objects.get
1593
compute.zones.get
compute.zones.list
iam.roles.get
Example 9.61. Required Images permissions for installation
compute.images.create
compute.images.delete
compute.images.get
compute.images.list
1594
compute.routes.list
dns.changes.create
1595
compute.disks.list
1596
compute.images.list
Example 9.71. Required permissions to get Region related information
compute.regions.get
Example 9.72. Required Deployment Manager permissions
deploymentmanager.deployments.create
deploymentmanager.deployments.delete
deploymentmanager.deployments.get
deploymentmanager.deployments.list
deploymentmanager.manifests.get
deploymentmanager.operations.get
deploymentmanager.resources.list
Optimizing storage
9.10.4.8. Supported GCP regions
regions:
1597
NOTE
1598
9.10.4.9. Installing and configuring CLI tools for GCP
To install OpenShift Container Platform on Google Cloud Platform (GCP) using user-provisioned
infrastructure, you must install and configure the CLI tools for GCP.
Prerequisites
You created a service account and granted it the required permissions.
Procedure
1. Install the following binaries in $PATH:
gcloud
gsutil
See Install the latest Cloud SDK version in the GCP documentation.
2. Authenticate using the gcloud tool with your configured service account.
See Authorizing with a service account in the GCP documentation.

Hosts Description
control plane.
IMPORTANT
1599
IMPORTANT
machines.

System Per Second
(IOPS)[2]

8.6 and later
[3]
Optimizing storage
1600
Platform.
A2
C2
C2D
C3
E2
M1
N1
N2
N2D
Tau T2D
Tau T2A

1601
installation.
IMPORTANT
Procedure
Example output

1602


Example output
...
variant: openshift
version: 4.15.0
metadata:
labels:
storage:
disks:
partitions:
- label: var
number: 5
filesystems:
path: /var
format: xfs
1603
NOTE
name.

partition.yaml

(GCP).
Prerequisites
cluster.
Procedure
command:
1604
NOTE

NOTE

more information, see "Installing a three-node cluster on GCP".
IMPORTANT

1605
NOTE
Prerequisites
Procedure
controlPlane:
platform:
gcp:
secureBoot: Enabled
compute:
- platform:
gcp:
secureBoot: Enabled
platform:
gcp:
secureBoot: Enabled
each other.
NOTE
Prerequisites
1606
Procedure
controlPlane:
platform:
gcp:
compute:
- platform:
gcp:
platform:
gcp:
Prerequisites
1607
NOTE
(169.254.169.254).
Procedure
apiVersion: v1
proxy:
must be http.

destinations.
NOTE
1608
NOTE
NOTE
NOTE
created.
machines.
IMPORTANT
Prerequisites
Procedure
1609
Procedure
machines.
machine-set.yaml
4. Optional: If you do not want the cluster to provision compute machines, remove the Kubernetes
manifest files that define the worker machines:
IMPORTANT

these machines.

WARNING
IMPORTANT
1610

machines:
kind: DNS
metadata:
name: cluster
spec:
privateZone: 1
publicZone: 2
status: {}
.
├── auth
1611
Optional: Adding the ingress DNS records
9.10.7. Exporting common variables
9.10.7.1. Extracting the infrastructure name
cluster in Google Cloud Platform (GCP). The infrastructure name is also used to locate the appropriate
GCP resources during an OpenShift Container Platform installation. The provided Deployment Manager
Prerequisites
your cluster.
Procedure
following command:
Example output
openshift-vw9j6 1
9.10.7.2. Exporting common variables for Deployment Manager templates
You must export a common set of variables that are used with the provided Deployment Manager
templates used to assist in completing a user-provided infrastructure install on Google Cloud Platform
(GCP).
NOTE
Specific Deployment Manager templates can also require additional exported variables,
which are detailed in their related procedures.
Prerequisites
cluster.
1612
Install the jq package.
Procedure
1. Export the following common variables to be used by the provided Deployment Manager
templates:
$ export BASE_DOMAIN='<base_domain>'
$ export BASE_DOMAIN_ZONE_NAME='<base_domain_zone_name>'
$ export NETWORK_CIDR='10.0.0.0/16'
$ export MASTER_SUBNET_CIDR='10.0.0.0/17'
$ export WORKER_SUBNET_CIDR='10.0.128.0/17'
$ export CLUSTER_NAME=`jq -r .clusterName <installation_directory>/metadata.json`
$ export INFRA_ID=`jq -r .infraID <installation_directory>/metadata.json`
$ export PROJECT_NAME=`jq -r .gcp.projectID <installation_directory>/metadata.json`
$ export REGION=`jq -r .gcp.region <installation_directory>/metadata.json`
9.10.8. Creating a VPC in GCP

You must create a VPC in Google Cloud Platform (GCP) for your OpenShift Container Platform cluster
to use. You can customize the VPC to meet your requirements. One way to create the VPC is to modify
the provided Deployment Manager template.
NOTE
If you do not use the provided Deployment Manager template to create your GCP
Prerequisites
Configure a GCP account.
Procedure
1. Copy the template from the Deployment Manager template for the VPCsection of this topic
and save it as 01_vpc.py on your computer. This template describes the VPC that your cluster
requires.
2. Create a 01_vpc.yaml resource definition file:
$ cat <<EOF >01_vpc.yaml

imports:
1613
- path: 01_vpc.py
resources:
- name: cluster-vpc
type: 01_vpc.py
properties:
infra_id: '${INFRA_ID}' 1
region: '${REGION}' 2
master_subnet_cidr: '${MASTER_SUBNET_CIDR}' 3
worker_subnet_cidr: '${WORKER_SUBNET_CIDR}' 4
EOF
1 infra_id is the INFRA_ID infrastructure name from the extraction step.
2 region is the region to deploy the cluster into, for example us-central1.
3 master_subnet_cidr is the CIDR for the master subnet, for example 10.0.0.0/17.
4 worker_subnet_cidr is the CIDR for the worker subnet, for example 10.0.128.0/17.
3. Create the deployment by using the gcloud CLI:
$ gcloud deployment-manager deployments create ${INFRA_ID}-vpc --config 01_vpc.yaml
9.10.8.1. Deployment Manager template for the VPC
You can use the following Deployment Manager template to deploy the VPC that you need for your
OpenShift Container Platform cluster:
Example 9.75. 01_vpc.py Deployment Manager template
def GenerateConfig(context):
resources = [{
'name': context.properties['infra_id'] + '-network',
'type': 'compute.v1.network',
'properties': {
'region': context.properties['region'],
'autoCreateSubnetworks': False
}
}, {
'name': context.properties['infra_id'] + '-master-subnet',
'type': 'compute.v1.subnetwork',
'properties': {
'network': '$(ref.' + context.properties['infra_id'] + '-network.selfLink)',
'ipCidrRange': context.properties['master_subnet_cidr']
}
}, {
'name': context.properties['infra_id'] + '-worker-subnet',
'properties': {
1614
'ipCidrRange': context.properties['worker_subnet_cidr']
}
}, {
'name': context.properties['infra_id'] + '-router',
'type': 'compute.v1.router',
'properties': {
'nats': [{
'name': context.properties['infra_id'] + '-nat-master',
'natIpAllocateOption': 'AUTO_ONLY',
'minPortsPerVm': 7168,
'sourceSubnetworkIpRangesToNat': 'LIST_OF_SUBNETWORKS',
'subnetworks': [{
'name': '$(ref.' + context.properties['infra_id'] + '-master-subnet.selfLink)',
'sourceIpRangesToNat': ['ALL_IP_RANGES']
}]
}, {
'name': context.properties['infra_id'] + '-nat-worker',
'subnetworks': [{
'name': '$(ref.' + context.properties['infra_id'] + '-worker-subnet.selfLink)',
}]
}]
}
}]
return {'resources': resources}

9.10.9.1. Setting the cluster node hostnames through DHCP
On Red Hat Enterprise Linux CoreOS (RHCOS) machines, the hostname is set through
NetworkManager. By default, the machines obtain their hostname through DHCP. If the hostname is not
provided by DHCP, set statically through kernel arguments, or another method, it is obtained through a
reverse DNS lookup. Reverse DNS lookup occurs after the network has been initialized on a node and
can take time to resolve. Other system services can start prior to this and detect the hostname as
localhost or similar. You can avoid this by using DHCP to provide the hostname for each cluster node.
Additionally, setting the hostnames through DHCP can bypass any manual DNS record name
configuration errors in environments that have a DNS split-horizon implementation.
1615
IMPORTANT
Hat.
TCP 1936 Metrics
UDP 4789 VXLAN
6081 Geneve
9100- 9101.
1616
9.10.10. Creating load balancers in GCP

You must configure load balancers in Google Cloud Platform (GCP) for your OpenShift Container
Platform cluster to use. One way to create these components is to modify the provided Deployment
Manager template.
NOTE
Prerequisites
Create and configure a VPC and associated subnets in GCP.
Procedure
1. Copy the template from the Deployment Manager template for the internal load balancer
section of this topic and save it as 02_lb_int.py on your computer. This template describes the
internal load balancing objects that your cluster requires.
2. For an external cluster, also copy the template from the Deployment Manager template for
the external load balancer section of this topic and save it as 02_lb_ext.py on your computer.
This template describes the external load balancing objects that your cluster requires.
3. Export the variables that the deployment template uses:
a. Export the cluster network location:
$ export CLUSTER_NETWORK=(`gcloud compute networks describe ${INFRA_ID}-

network --format json | jq -r .selfLink`)
b. Export the control plane subnet location:
$ export CONTROL_SUBNET=(`gcloud compute networks subnets describe

${INFRA_ID}-master-subnet --region=${REGION} --format json | jq -r .selfLink`)
c. Export the three zones that the cluster uses:
$ export ZONE_0=(`gcloud compute regions describe ${REGION} --format=json | jq -r

.zones[0] | cut -d "/" -f9`)

.zones[1] | cut -d "/" -f9`)

.zones[2] | cut -d "/" -f9`)
1617
4. Create a 02_infra.yaml resource definition file:
$ cat <<EOF >02_infra.yaml

imports:
- path: 02_lb_ext.py
- path: 02_lb_int.py 1
resources:
- name: cluster-lb-ext 2
type: 02_lb_ext.py
properties:
- name: cluster-lb-int
type: 02_lb_int.py
properties:
cluster_network: '${CLUSTER_NETWORK}'
control_subnet: '${CONTROL_SUBNET}' 5
infra_id: '${INFRA_ID}'
region: '${REGION}'
zones: 6
- '${ZONE_0}'
- '${ZONE_1}'
- '${ZONE_2}'
EOF
1 2 Required only when deploying an external cluster.
5 control_subnet is the URI to the control subnet.
6 zones are the zones to deploy the control plane instances into, like us-east1-b, us-east1-c,
and us-east1-d.
$ gcloud deployment-manager deployments create ${INFRA_ID}-infra --config 02_infra.yaml
6. Export the cluster IP address:
$ export CLUSTER_IP=(`gcloud compute addresses describe ${INFRA_ID}-cluster-ip --

region=${REGION} --format json | jq -r .address`)
7. For an external cluster, also export the cluster public IP address:
$ export CLUSTER_PUBLIC_IP=(`gcloud compute addresses describe ${INFRA_ID}-cluster-

public-ip --region=${REGION} --format json | jq -r .address`)
9.10.10.1. Deployment Manager template for the external load balancer
You can use the following Deployment Manager template to deploy the external load balancer that you
1618
need for your OpenShift Container Platform cluster:
Example 9.76. 02_lb_ext.py Deployment Manager template
resources = [{
'name': context.properties['infra_id'] + '-cluster-public-ip',
'type': 'compute.v1.address',
'properties': {
'region': context.properties['region']
}
}, {
# Refer to docs/dev/kube-apiserver-health-check.md on how to correctly setup health check
probe for kube-apiserver
'name': context.properties['infra_id'] + '-api-http-health-check',
'type': 'compute.v1.httpHealthCheck',
'properties': {
'port': 6080,
'requestPath': '/readyz'
}
}, {
'name': context.properties['infra_id'] + '-api-target-pool',
'type': 'compute.v1.targetPool',
'properties': {
'healthChecks': ['$(ref.' + context.properties['infra_id'] + '-api-http-health-check.selfLink)'],
'instances': []
}
}, {
'name': context.properties['infra_id'] + '-api-forwarding-rule',
'type': 'compute.v1.forwardingRule',
'properties': {
'IPAddress': '$(ref.' + context.properties['infra_id'] + '-cluster-public-ip.selfLink)',
'target': '$(ref.' + context.properties['infra_id'] + '-api-target-pool.selfLink)',
'portRange': '6443'
}
}]
9.10.10.2. Deployment Manager template for the internal load balancer
You can use the following Deployment Manager template to deploy the internal load balancer that you
Example 9.77. 02_lb_int.py Deployment Manager template
backends = []
for zone in context.properties['zones']:
1619
backends.append({
'group': '$(ref.' + context.properties['infra_id'] + '-master-' + zone + '-ig' + '.selfLink)'
})
resources = [{
'name': context.properties['infra_id'] + '-cluster-ip',
'properties': {
'addressType': 'INTERNAL',
'subnetwork': context.properties['control_subnet']
}
}, {
'name': context.properties['infra_id'] + '-api-internal-health-check',
'type': 'compute.v1.healthCheck',
'properties': {
'httpsHealthCheck': {
'port': 6443,
},
'type': "HTTPS"
}
}, {
'name': context.properties['infra_id'] + '-api-internal-backend-service',
'type': 'compute.v1.regionBackendService',
'properties': {
'backends': backends,
'healthChecks': ['$(ref.' + context.properties['infra_id'] + '-api-internal-health-
check.selfLink)'],
'loadBalancingScheme': 'INTERNAL',
'protocol': 'TCP',
'timeoutSec': 120
}
}, {
'name': context.properties['infra_id'] + '-api-internal-forwarding-rule',
'properties': {
'backendService': '$(ref.' + context.properties['infra_id'] + '-api-internal-backend-
service.selfLink)',
'IPAddress': '$(ref.' + context.properties['infra_id'] + '-cluster-ip.selfLink)',
'ports': ['6443','22623'],
}
}]

resources.append({
'name': context.properties['infra_id'] + '-master-' + zone + '-ig',
'type': 'compute.v1.instanceGroup',
'properties': {
'namedPorts': [
1620
{
'name': 'ignition',
'port': 22623
}, {
'name': 'https',
'port': 6443
}
],
'network': context.properties['cluster_network'],
'zone': zone
}
})
You will need this template in addition to the 02_lb_ext.py template when you create an external
cluster.
9.10.11. Creating a private DNS zone in GCP

You must configure a private DNS zone in Google Cloud Platform (GCP) for your OpenShift Container
Platform cluster to use. One way to create this component is to modify the provided Deployment
Manager template.
NOTE
Prerequisites
Procedure
1. Copy the template from the Deployment Manager template for the private DNS section of
this topic and save it as 02_dns.py on your computer. This template describes the private DNS
objects that your cluster requires.
2. Create a 02_dns.yaml resource definition file:
$ cat <<EOF >02_dns.yaml

imports:
- path: 02_dns.py
resources:
- name: cluster-dns
type: 02_dns.py
1621
properties:
cluster_domain: '${CLUSTER_NAME}.${BASE_DOMAIN}' 2
cluster_network: '${CLUSTER_NETWORK}' 3
EOF
2 cluster_domain is the domain for the cluster, for example openshift.example.com.
3 cluster_network is the selfLink URL to the cluster network.
$ gcloud deployment-manager deployments create ${INFRA_ID}-dns --config 02_dns.yaml
4. The templates do not create DNS entries due to limitations of Deployment Manager, so you
must create them manually:
a. Add the internal DNS entries:
$ if [ -f transaction.yaml ]; then rm transaction.yaml; fi

$ gcloud dns record-sets transaction start --zone ${INFRA_ID}-private-zone
$ gcloud dns record-sets transaction add ${CLUSTER_IP} --name
api.${CLUSTER_NAME}.${BASE_DOMAIN}. --ttl 60 --type A --zone ${INFRA_ID}-
private-zone
$ gcloud dns record-sets transaction add ${CLUSTER_IP} --name api-
int.${CLUSTER_NAME}.${BASE_DOMAIN}. --ttl 60 --type A --zone ${INFRA_ID}-
private-zone
$ gcloud dns record-sets transaction execute --zone ${INFRA_ID}-private-zone
b. For an external cluster, also add the external DNS entries:

$ gcloud dns record-sets transaction start --zone ${BASE_DOMAIN_ZONE_NAME}
$ gcloud dns record-sets transaction add ${CLUSTER_PUBLIC_IP} --name
api.${CLUSTER_NAME}.${BASE_DOMAIN}. --ttl 60 --type A --zone
${BASE_DOMAIN_ZONE_NAME}
$ gcloud dns record-sets transaction execute --zone ${BASE_DOMAIN_ZONE_NAME}
9.10.11.1. Deployment Manager template for the private DNS
You can use the following Deployment Manager template to deploy the private DNS that you need for
your OpenShift Container Platform cluster:
Example 9.78. 02_dns.py Deployment Manager template
resources = [{
'name': context.properties['infra_id'] + '-private-zone',
'type': 'dns.v1.managedZone',
'properties': {
1622
'description': '',
'dnsName': context.properties['cluster_domain'] + '.',
'visibility': 'private',
'privateVisibilityConfig': {
'networks': [{
'networkUrl': context.properties['cluster_network']
}]
}
}
}]
9.10.12. Creating firewall rules in GCP

You must create firewall rules in Google Cloud Platform (GCP) for your OpenShift Container Platform
cluster to use. One way to create these components is to modify the provided Deployment Manager
template.
NOTE
Prerequisites
Procedure
1. Copy the template from the Deployment Manager template for firewall rules section of this
topic and save it as 03_firewall.py on your computer. This template describes the security
groups that your cluster requires.
2. Create a 03_firewall.yaml resource definition file:
$ cat <<EOF >03_firewall.yaml

imports:
- path: 03_firewall.py
resources:
- name: cluster-firewall
type: 03_firewall.py
properties:
allowed_external_cidr: '0.0.0.0/0' 1
1623
network_cidr: '${NETWORK_CIDR}' 4
EOF
1 allowed_external_cidr is the CIDR range that can access the cluster API and SSH to the
bootstrap host. For an internal cluster, set this value to ${NETWORK_CIDR}.
4 network_cidr is the CIDR of the VPC network, for example 10.0.0.0/16.
$ gcloud deployment-manager deployments create ${INFRA_ID}-firewall --config

03_firewall.yaml
9.10.12.1. Deployment Manager template for firewall rules
You can use the following Deployment Manager template to deploy the firewall rues that you need for
Example 9.79. 03_firewall.py Deployment Manager template
resources = [{
'name': context.properties['infra_id'] + '-bootstrap-in-ssh',
'type': 'compute.v1.firewall',
'properties': {
'allowed': [{
'IPProtocol': 'tcp',
'ports': ['22']
}],
'sourceRanges': [context.properties['allowed_external_cidr']],
'targetTags': [context.properties['infra_id'] + '-bootstrap']
}
}, {
'name': context.properties['infra_id'] + '-api',
'properties': {
'allowed': [{
'ports': ['6443']
}],
'targetTags': [context.properties['infra_id'] + '-master']
}
}, {
'name': context.properties['infra_id'] + '-health-checks',
1624
'properties': {
'allowed': [{
'ports': ['6080', '6443', '22624']
}],
'sourceRanges': ['35.191.0.0/16', '130.211.0.0/22', '209.85.152.0/22', '209.85.204.0/22'],
}
}, {
'name': context.properties['infra_id'] + '-etcd',
'properties': {
'allowed': [{
'ports': ['2379-2380']
}],
'sourceTags': [context.properties['infra_id'] + '-master'],
}
}, {
'name': context.properties['infra_id'] + '-control-plane',
'properties': {
'allowed': [{
'ports': ['10257']
},{
'ports': ['10259']
},{
'ports': ['22623']
}],
'sourceTags': [
context.properties['infra_id'] + '-master',
context.properties['infra_id'] + '-worker'
],
}
}, {
'name': context.properties['infra_id'] + '-internal-network',
'properties': {
'allowed': [{
'IPProtocol': 'icmp'
},{
'ports': ['22']
}],
'sourceRanges': [context.properties['network_cidr']],
'targetTags': [
1625
]
}
}, {
'name': context.properties['infra_id'] + '-internal-cluster',
'properties': {
'allowed': [{
'IPProtocol': 'udp',
'ports': ['4789', '6081']
},{
'ports': ['500', '4500']
},{
'IPProtocol': 'esp',
},{
'ports': ['9000-9999']
},{
'ports': ['9000-9999']
},{
'ports': ['10250']
},{
'ports': ['30000-32767']
},{
'ports': ['30000-32767']
}],
'sourceTags': [
],
'targetTags': [
]
}
}]
9.10.13. Creating IAM roles in GCP

You must create IAM roles in Google Cloud Platform (GCP) for your OpenShift Container Platform
template.
NOTE
1626
NOTE
Prerequisites
Procedure
1. Copy the template from the Deployment Manager template for IAM rolessection of this topic
and save it as 03_iam.py on your computer. This template describes the IAM roles that your
cluster requires.
2. Create a 03_iam.yaml resource definition file:
$ cat <<EOF >03_iam.yaml

imports:
- path: 03_iam.py
resources:
- name: cluster-iam
type: 03_iam.py
properties:
EOF
$ gcloud deployment-manager deployments create ${INFRA_ID}-iam --config 03_iam.yaml
4. Export the variable for the master service account:
$ export MASTER_SERVICE_ACCOUNT=(`gcloud iam service-accounts list --filter

"email~^${INFRA_ID}-m@${PROJECT_NAME}." --format json | jq -r '.[0].email'`)
5. Export the variable for the worker service account:
$ export WORKER_SERVICE_ACCOUNT=(`gcloud iam service-accounts list --filter

"email~^${INFRA_ID}-w@${PROJECT_NAME}." --format json | jq -r '.[0].email'`)
6. Export the variable for the subnet that hosts the compute machines:
$ export COMPUTE_SUBNET=(`gcloud compute networks subnets describe ${INFRA_ID}-

worker-subnet --region=${REGION} --format json | jq -r .selfLink`)
1627
7. The templates do not create the policy bindings due to limitations of Deployment Manager, so
you must create them manually:
$ gcloud projects add-iam-policy-binding ${PROJECT_NAME} --member

"serviceAccount:${MASTER_SERVICE_ACCOUNT}" --role "roles/compute.instanceAdmin"
"serviceAccount:${MASTER_SERVICE_ACCOUNT}" --role "roles/compute.networkAdmin"
"serviceAccount:${MASTER_SERVICE_ACCOUNT}" --role "roles/compute.securityAdmin"
"serviceAccount:${MASTER_SERVICE_ACCOUNT}" --role "roles/iam.serviceAccountUser"
"serviceAccount:${MASTER_SERVICE_ACCOUNT}" --role "roles/storage.admin"

"serviceAccount:${WORKER_SERVICE_ACCOUNT}" --role "roles/compute.viewer"
"serviceAccount:${WORKER_SERVICE_ACCOUNT}" --role "roles/storage.admin"
8. Create a service account key and store it locally for later use:
$ gcloud iam service-accounts keys create service-account-key.json --iam-

account=${MASTER_SERVICE_ACCOUNT}
9.10.13.1. Deployment Manager template for IAM roles
You can use the following Deployment Manager template to deploy the IAM roles that you need for your
Example 9.80. 03_iam.py Deployment Manager template
resources = [{
'name': context.properties['infra_id'] + '-master-node-sa',
'type': 'iam.v1.serviceAccount',
'properties': {
'accountId': context.properties['infra_id'] + '-m',
'displayName': context.properties['infra_id'] + '-master-node'
}
}, {
'name': context.properties['infra_id'] + '-worker-node-sa',
'properties': {
'accountId': context.properties['infra_id'] + '-w',
'displayName': context.properties['infra_id'] + '-worker-node'
}
}]
9.10.14. Creating the RHCOS cluster image for the GCP infrastructure
1628
You must use a valid Red Hat Enterprise Linux CoreOS (RHCOS) image for Google Cloud Platform
(GCP) for your OpenShift Container Platform nodes.
Procedure
1. Obtain the RHCOS image from the RHCOS image mirror page.
IMPORTANT
Platform. You must download an image with the highest version that is less than
or equal to the OpenShift Container Platform version that you install. Use the
image version that matches your OpenShift Container Platform version if it is
available.
The file name contains the OpenShift Container Platform version number in the format rhcos-
<version>-<arch>-gcp.<arch>.tar.gz.
2. Create the Google storage bucket:
$ gsutil mb gs://<bucket_name>
3. Upload the RHCOS image to the Google storage bucket:
$ gsutil cp <downloaded_image_file_path>/rhcos-<version>-x86_64-gcp.x86_64.tar.gz
gs://<bucket_name>
4. Export the uploaded RHCOS image location as a variable:
$ export IMAGE_SOURCE=gs://<bucket_name>/rhcos-<version>-x86_64-gcp.x86_64.tar.gz
5. Create the cluster image:
$ gcloud compute images create "${INFRA_ID}-rhcos-image" \

--source-uri="${IMAGE_SOURCE}"
9.10.15. Creating the bootstrap machine in GCP

You must create the bootstrap machine in Google Cloud Platform (GCP) to use during OpenShift
Container Platform cluster initialization. One way to create this machine is to modify the provided
Deployment Manager template.
NOTE
If you do not use the provided Deployment Manager template to create your bootstrap
machine, you must review the provided information and manually create the
Prerequisites
1629
Create and configure networking and load balancers in GCP.
Ensure pyOpenSSL is installed.
Procedure
1. Copy the template from the Deployment Manager template for the bootstrap machine
section of this topic and save it as 04_bootstrap.py on your computer. This template describes
2. Export the location of the Red Hat Enterprise Linux CoreOS (RHCOS) image that the
installation program requires:
$ export CLUSTER_IMAGE=(`gcloud compute images describe ${INFRA_ID}-rhcos-image --

format json | jq -r .selfLink`)
3. Create a bucket and upload the bootstrap.ign file:
$ gsutil mb gs://${INFRA_ID}-bootstrap-ignition
$ gsutil cp <installation_directory>/bootstrap.ign gs://${INFRA_ID}-bootstrap-ignition/
4. Create a signed URL for the bootstrap instance to use to access the Ignition config. Export the
URL from the output as a variable:
$ export BOOTSTRAP_IGN=`gsutil signurl -d 1h service-account-key.json gs://${INFRA_ID}-

bootstrap-ignition/bootstrap.ign | grep "^gs:" | awk '{print $5}'`
5. Create a 04_bootstrap.yaml resource definition file:
$ cat <<EOF >04_bootstrap.yaml

imports:
- path: 04_bootstrap.py
resources:
- name: cluster-bootstrap
type: 04_bootstrap.py
properties:
zone: '${ZONE_0}' 3
image: '${CLUSTER_IMAGE}' 6
machine_type: 'n1-standard-4' 7
root_volume_size: '128' 8
1630
bootstrap_ign: '${BOOTSTRAP_IGN}' 9
EOF
3 zone is the zone to deploy the bootstrap instance into, for example us-central1-b.
5 control_subnet is the selfLink URL to the control subnet.
6 image is the selfLink URL to the RHCOS image.
7 machine_type is the machine type of the instance, for example n1-standard-4.
8 root_volume_size is the boot disk size for the bootstrap machine.
9 bootstrap_ign is the URL output when creating a signed URL.
$ gcloud deployment-manager deployments create ${INFRA_ID}-bootstrap --config

04_bootstrap.yaml
7. The templates do not manage load balancer membership due to limitations of Deployment
Manager, so you must add the bootstrap machine manually.
a. Add the bootstrap instance to the internal load balancer instance group:
$ gcloud compute instance-groups unmanaged add-instances \

${INFRA_ID}-bootstrap-ig --zone=${ZONE_0} --instances=${INFRA_ID}-bootstrap
b. Add the bootstrap instance group to the internal load balancer backend service:
$ gcloud compute backend-services add-backend \

${INFRA_ID}-api-internal-backend-service --region=${REGION} --instance-
group=${INFRA_ID}-bootstrap-ig --instance-group-zone=${ZONE_0}
9.10.15.1. Deployment Manager template for the bootstrap machine
You can use the following Deployment Manager template to deploy the bootstrap machine that you
Example 9.81. 04_bootstrap.py Deployment Manager template
resources = [{
'name': context.properties['infra_id'] + '-bootstrap-public-ip',
'properties': {
1631
}
}, {
'name': context.properties['infra_id'] + '-bootstrap',
'type': 'compute.v1.instance',
'properties': {
'disks': [{
'autoDelete': True,
'boot': True,
'initializeParams': {
'diskSizeGb': context.properties['root_volume_size'],
'sourceImage': context.properties['image']
}
}],
'machineType': 'zones/' + context.properties['zone'] + '/machineTypes/' +
context.properties['machine_type'],
'metadata': {
'items': [{
'key': 'user-data',
'value': '{"ignition":{"config":{"replace":{"source":"' + context.properties['bootstrap_ign']
+ '"}},"version":"3.2.0"}}',
}]
},
'networkInterfaces': [{
'subnetwork': context.properties['control_subnet'],
'accessConfigs': [{
'natIP': '$(ref.' + context.properties['infra_id'] + '-bootstrap-public-ip.address)'
}]
}],
'tags': {
'items': [
context.properties['infra_id'] + '-bootstrap'
]
},
'zone': context.properties['zone']
}
}, {
'name': context.properties['infra_id'] + '-bootstrap-ig',
'properties': {
'namedPorts': [
{
'name': 'ignition',
'port': 22623
}, {
'name': 'https',
'port': 6443
}
],
}
}]
1632
9.10.16. Creating the control plane machines in GCP

You must create the control plane machines in Google Cloud Platform (GCP) for your cluster to use.
One way to create these machines is to modify the provided Deployment Manager template.
NOTE
If you do not use the provided Deployment Manager template to create your control
plane machines, you must review the provided information and manually create the
Prerequisites
Procedure
1. Copy the template from the Deployment Manager template for control plane machines
section of this topic and save it as 05_control_plane.py on your computer. This template
describes the control plane machines that your cluster requires.
2. Export the following variable required by the resource definition:
$ export MASTER_IGNITION=`cat <installation_directory>/master.ign`
3. Create a 05_control_plane.yaml resource definition file:
$ cat <<EOF >05_control_plane.yaml

imports:
- path: 05_control_plane.py
resources:
- name: cluster-control-plane
type: 05_control_plane.py
properties:
zones: 2
- '${ZONE_0}'
- '${ZONE_1}'
- '${ZONE_2}'
1633
root_volume_size: '128'
service_account_email: '${MASTER_SERVICE_ACCOUNT}' 6
ignition: '${MASTER_IGNITION}' 7
EOF
2 zones are the zones to deploy the control plane instances into, for example us-central1-a,
us-central1-b, and us-central1-c.
6 service_account_email is the email address for the master service account that you
created.
7 ignition is the contents of the master.ign file.
$ gcloud deployment-manager deployments create ${INFRA_ID}-control-plane --config

05_control_plane.yaml
Manager, so you must add the control plane machines manually.
Run the following commands to add the control plane machines to the appropriate instance
groups:
$ gcloud compute instance-groups unmanaged add-instances ${INFRA_ID}-master-

${ZONE_0}-ig --zone=${ZONE_0} --instances=${INFRA_ID}-master-0


For an external cluster, you must also run the following commands to add the control plane
machines to the target pools:
$ gcloud compute target-pools add-instances ${INFRA_ID}-api-target-pool --instances-

zone="${ZONE_0}" --instances=${INFRA_ID}-master-0

1634

9.10.16.1. Deployment Manager template for control plane machines
You can use the following Deployment Manager template to deploy the control plane machines that you
Example 9.82. 05_control_plane.py Deployment Manager template
resources = [{
'name': context.properties['infra_id'] + '-master-0',
'properties': {
'disks': [{
'autoDelete': True,
'boot': True,
'diskType': 'zones/' + context.properties['zones'][0] + '/diskTypes/pd-ssd',
}
}],
'machineType': 'zones/' + context.properties['zones'][0] + '/machineTypes/' +
'metadata': {
'items': [{
'key': 'user-data',
'value': context.properties['ignition']
}]
},
}],
'serviceAccounts': [{
'email': context.properties['service_account_email'],
'scopes': ['https://www.googleapis.com/auth/cloud-platform']
}],
'tags': {
'items': [
]
},
'zone': context.properties['zones'][0]
}
}, {
'properties': {
'disks': [{
'autoDelete': True,
'boot': True,
1635
}
}],
'metadata': {
'items': [{
'key': 'user-data',
}]
},
}],
}],
'tags': {
'items': [
]
},
}
}, {
'properties': {
'disks': [{
'autoDelete': True,
'boot': True,
}
}],
'metadata': {
'items': [{
'key': 'user-data',
}]
},
}],
}],
'tags': {
1636
'items': [
]
},
}
}]
9.10.17. Wait for bootstrap completion and remove bootstrap resources in GCP
After you create all of the required infrastructure in Google Cloud Platform (GCP), wait for the
bootstrap process to complete on the machines that you provisioned by using the Ignition config files
that you generated with the installation program.
Prerequisites
Procedure

--log-level info 2
$ gcloud compute backend-services remove-backend ${INFRA_ID}-api-internal-backend-

service --region=${REGION} --instance-group=${INFRA_ID}-bootstrap-ig --instance-group-
zone=${ZONE_0}
1637
$ gsutil rm gs://${INFRA_ID}-bootstrap-ignition/bootstrap.ign
$ gsutil rb gs://${INFRA_ID}-bootstrap-ignition
$ gcloud deployment-manager deployments delete ${INFRA_ID}-bootstrap
9.10.18. Creating additional worker machines in GCP

You can create worker machines in Google Cloud Platform (GCP) for your cluster to use by launching
NOTE
In this example, you manually launch one instance by using the Deployment Manager template.
Additional instances can be launched by including additional resources of type 06_worker.py in the file.
NOTE
If you do not use the provided Deployment Manager template to create your worker
machines, you must review the provided information and manually create the
Prerequisites
Procedure
1. Copy the template from the Deployment Manager template for worker machines section of
this topic and save it as 06_worker.py on your computer. This template describes the worker
2. Export the variables that the resource definition uses.
a. Export the subnet that hosts the compute machines:
1638
$ export COMPUTE_SUBNET=(`gcloud compute networks subnets describe

${INFRA_ID}-worker-subnet --region=${REGION} --format json | jq -r .selfLink`)
b. Export the email address for your service account:

c. Export the location of the compute machine Ignition config file:
$ export WORKER_IGNITION=`cat <installation_directory>/worker.ign`
3. Create a 06_worker.yaml resource definition file:
$ cat <<EOF >06_worker.yaml

imports:
- path: 06_worker.py
resources:
- name: 'worker-0' 1
type: 06_worker.py
properties:
zone: '${ZONE_0}' 3
compute_subnet: '${COMPUTE_SUBNET}' 4
service_account_email: '${WORKER_SERVICE_ACCOUNT}' 7
ignition: '${WORKER_IGNITION}' 8
- name: 'worker-1'
type: 06_worker.py
properties:
zone: '${ZONE_1}' 10
EOF
1 name is the name of the worker machine, for example worker-0.
2 9 infra_id is the INFRA_ID infrastructure name from the extraction step.
3 10 zone is the zone to deploy the worker machine into, for example us-central1-a.
4 11 compute_subnet is the selfLink URL to the compute subnet.
5 12 image is the selfLink URL to the RHCOS image. 1
1639
6 13 machine_type is the machine type of the instance, for example n1-standard-4.
7 14 service_account_email is the email address for the worker service account that you
created.
8 15 ignition is the contents of the worker.ign file.
4. Optional: If you want to launch additional instances, include additional resources of type
06_worker.py in your 06_worker.yaml resource definition file.
$ gcloud deployment-manager deployments create ${INFRA_ID}-worker --config

06_worker.yaml
1. To use a GCP Marketplace image, specify the offer to use:
OpenShift Container Platform:

https://www.googleapis.com/compute/v1/projects/redhat-marketplace-
public/global/images/redhat-coreos-ocp-413-x86-64-202305021736
OpenShift Platform Plus: https://www.googleapis.com/compute/v1/projects/redhat-

marketplace-public/global/images/redhat-coreos-opp-413-x86-64-202305021736
OpenShift Kubernetes Engine:

public/global/images/redhat-coreos-oke-413-x86-64-202305021736
9.10.18.1. Deployment Manager template for worker machines
You can use the following Deployment Manager template to deploy the worker machines that you need
Example 9.83. 06_worker.py Deployment Manager template
resources = [{
'name': context.properties['infra_id'] + '-' + context.env['name'],
'properties': {
'disks': [{
'autoDelete': True,
'boot': True,
}
}],
'metadata': {
'items': [{
'key': 'user-data',
1640
}]
},
'subnetwork': context.properties['compute_subnet']
}],
}],
'tags': {
'items': [
context.properties['infra_id'] + '-worker',
]
},
}
}]

IMPORTANT

Procedure
Portal.
$ tar xvf <file>

$ echo $PATH
1641
Verification
$ oc <command>

Procedure
Portal.

C:\> path
Verification
C:\> oc <command>

Procedure
Portal.
NOTE

$ echo $PATH
1642
Verification
$ oc <command>

Prerequisites
Procedure
$ oc whoami
Example output
system:admin

Prerequisites
Procedure
$ oc get nodes
Example output
1643
Example output

NOTE
$ oc get csr
Example output

...
list.
NOTE
NOTE
1644
NOTE
of the node.

NOTE
$ oc get csr
Example output

Pending
Pending
...
1645

$ oc get nodes
Example output

NOTE
9.10.22. Optional: Adding the ingress DNS records

If you removed the DNS zone configuration when creating Kubernetes manifests and generating Ignition
configs, you must manually create DNS records that point at the ingress load balancer. You can create
either a wildcard *.apps.{baseDomain}. or specific records. You can use A, CNAME, and other records
per your requirements.
Prerequisites
Remove the DNS Zone configuration when creating Kubernetes manifests and generating
Ignition configs.
Create the worker machines.
1646
Procedure
1. Wait for the Ingress router to create a load balancer and populate the EXTERNAL-IP field:
Example output

80:32288/TCP,443:31215/TCP 98
2. Add the A record to your zones:
To use A records:
i. Export the variable for the router IP address:
$ export ROUTER_IP=òc -n openshift-ingress get service router-default --no-

ii. Add the A record to the private zones:

$ gcloud dns record-sets transaction add ${ROUTER_IP} --name
\*.apps.${CLUSTER_NAME}.${BASE_DOMAIN}. --ttl 300 --type A --zone
${INFRA_ID}-private-zone
iii. For an external cluster, also add the A record to the public zones:

$ gcloud dns record-sets transaction execute --zone
To add explicit domains instead of using a wildcard, create entries for each of the cluster’s
current routes:

Example output
oauth-openshift.apps.your.cluster.domain.example.com
console-openshift-console.apps.your.cluster.domain.example.com
downloads-openshift-console.apps.your.cluster.domain.example.com
alertmanager-main-openshift-monitoring.apps.your.cluster.domain.example.com
prometheus-k8s-openshift-monitoring.apps.your.cluster.domain.example.com
1647
9.10.23. Completing a GCP installation on user-provisioned infrastructure

After you start the OpenShift Container Platform installation on Google Cloud Platform (GCP) user-
Prerequisites
GCP infrastructure.
Procedure
1. Complete the cluster installation:
Example output
IMPORTANT
installation.
2. Observe the running state of your cluster.
a. Run the following command to view the current cluster version and status:
$ oc get clusterversion
Example output
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS

version False True 24m Working towards 4.5.4: 99% complete
1648
b. Run the following command to view the Operators managed on the control plane by the
Cluster Version Operator (CVO):
$ oc get clusteroperators
Example output

SINCE
authentication 4.5.4 True False False 7m56s
etcd 4.5.4 False False False 25s
service-catalog-apiserver 4.5.4 True False False 23m
service-catalog-controller-manager 4.5.4 True False False 23m
c. Run the following command to view your cluster pods:
$ oc get pods --all-namespaces
Example output
NAMESPACE NAME
READY STATUS RESTARTS AGE
kube-system etcd-member-ip-10-0-3-111.us-east-
2.compute.internal 1/1 Running 0 35m
1649
openshift-apiserver-operator openshift-apiserver-operator-6d6674f4f4-
h7t2t 1/1 Running 1 37m
openshift-apiserver apiserver-fm48r
1/1 Running 0 30m
openshift-apiserver apiserver-fxkvv
1/1 Running 0 29m
openshift-apiserver apiserver-q85nm
1/1 Running 0 29m
...
openshift-service-ca-operator openshift-service-ca-operator-66ff6dc6cd-
9r257 1/1 Running 0 37m
openshift-service-ca apiservice-cabundle-injector-695b6bcbc-cl5hm
1/1 Running 0 35m
openshift-service-ca configmap-cabundle-injector-8498544d7-
25qn6 1/1 Running 0 35m
openshift-service-ca service-serving-cert-signer-6445fc9c6-wqdqn
1/1 Running 0 35m
openshift-service-catalog-apiserver-operator openshift-service-catalog-apiserver-
operator-549f44668b-b5q2w 1/1 Running 0 32m
openshift-service-catalog-controller-manager-operator openshift-service-catalog-
controller-manager-operator-b78cr2lnm 1/1 Running 0 31m
When the current cluster version is AVAILABLE, the installation is complete.

9.10.25. Next steps

Configure Global Access for an Ingress Controller on GCP .
9.11. INSTALLING A CLUSTER INTO A SHARED VPC ON GCP USING

DEPLOYMENT MANAGER TEMPLATES
In OpenShift Container Platform version 4.15, you can install a cluster into a shared Virtual Private Cloud
(VPC) on Google Cloud Platform (GCP) that uses infrastructure that you provide. In this context, a
cluster installed into a shared VPC is a cluster that is configured to use a VPC from a project different
from where the cluster is being deployed.
A shared VPC enables an organization to connect resources from multiple projects to a common VPC
1650
A shared VPC enables an organization to connect resources from multiple projects to a common VPC
network. You can communicate within the organization securely and efficiently by using internal IPs from
that network. For more information about shared VPC, see Shared VPC overview in the GCP
documentation.
The steps for performing a user-provided infrastructure installation into a shared VPC are outlined here.
Several Deployment Manager templates are provided to assist in completing these steps or to help
model your own. You are also free to create the required resources through other methods.
IMPORTANT

processes.
users.
NOTE
9.11.2. Certificate signing requests management

and approving them.

1651
IMPORTANT
9.11.4. Configuring the GCP project that hosts your cluster

Procedure
IMPORTANT
Prerequisites
Procedure
GCP documentation.

1652

(IAM) API
Cloud Deployment Manager V2 API deploymentmanager.googleapis.com
Platform cluster.

bootstrap
1653

bootstrap
NOTE
asia-east2
asia-northeast2
asia-south1
europe-north1
europe-west2
europe-west3
europe-west6
1654
southamerica-east1
us-west2
Prerequisites
Procedure
NOTE
create the cluster.
NOTE
9.11.4.4.1. Required GCP roles
following lists:
1655
Compute Admin
Role Administrator
Security Admin
Storage Admin
DNS Administrator
use:
Account Roles
roles/storage.admin
roles/storage.admin
regions:
1656
1657
NOTE
Prerequisites
Procedure
gcloud
gsutil

1658
Hosts Description
control plane.
IMPORTANT
machines.

System Per Second
(IOPS)[2]

8.6 and later
[3]
1659
Optimizing storage
Platform.
A2
C2
C2D
C3
E2
M1
N1
N2
N2D
Tau T2D
1660
9.11.6. Configuring the GCP project that hosts your shared VPC network
If you use a shared Virtual Private Cloud (VPC) to host your OpenShift Container Platform cluster in
Google Cloud Platform (GCP), you must configure the project that hosts it.
NOTE
If you already have a project that hosts the shared VPC network, review this section to
ensure that the project meets all of the requirements to install an OpenShift Container
Platform cluster.
Procedure
1. Create a project to host the shared VPC for your OpenShift Container Platform cluster. See
Creating and Managing Projects in the GCP documentation.
2. Create a service account in the project that hosts your shared VPC. See Creating a service
account in the GCP documentation.
NOTE
The service account for the project that hosts the shared VPC network requires
the following roles:
Compute Network User
Compute Security Admin
DNS Administrator
Security Admin
Network Management Admin
a dedicated public hosted zone in the project that hosts the shared VPC that you install the cluster into.
This zone must be authoritative for the domain. The DNS service provides cluster DNS resolution and
name lookup for external connections to the cluster.
1661
Procedure
NOTE
Google Domains.
9.11.6.2. Creating a VPC in GCP
NOTE
Prerequisites
Procedure
requires.
2. Export the following variables required by the resource definition:
1662
a. Export the control plane CIDR:
b. Export the compute CIDR:
c. Export the region to deploy the VPC network and cluster to:
$ export REGION='<region>'
3. Export the variable for the ID of the project that hosts the shared VPC:
$ export HOST_PROJECT=<host_project>
4. Export the variable for the email of the service account that belongs to host project:
$ export HOST_PROJECT_ACCOUNT=<host_service_account_email>

imports:
- path: 01_vpc.py
resources:
- name: cluster-vpc
type: 01_vpc.py
properties:
infra_id: '<prefix>' 1
EOF
1 infra_id is the prefix of the network name.
$ gcloud deployment-manager deployments create <vpc_deployment_name> --config

01_vpc.yaml --project ${HOST_PROJECT} --account ${HOST_PROJECT_ACCOUNT} 1
1 For <vpc_deployment_name>, specify the name of the VPC to deploy.
1663
7. Export the VPC variable that other components require:
a. Export the name of the host project network:
$ export HOST_PROJECT_NETWORK=<vpc_network>
b. Export the name of the host project control plane subnet:
$ export HOST_PROJECT_CONTROL_SUBNET=<control_plane_subnet>
c. Export the name of the host project compute subnet:
$ export HOST_PROJECT_COMPUTE_SUBNET=<compute_subnet>
8. Set up the shared VPC. See Setting up Shared VPC in the GCP documentation.
9.11.6.2.1. Deployment Manager template for the VPC
resources = [{
'properties': {
}
}, {
'properties': {
}
}, {
'properties': {
}
}, {
'properties': {
1664
'nats': [{
'subnetworks': [{
}]
}, {
'subnetworks': [{
}]
}]
}
}]

Prerequisites
recovery.
for your cluster.
Procedure
IMPORTANT
1665
IMPORTANT
NOTE
IMPORTANT
NOTE
Prerequisites
Procedure
controlPlane:
platform:
gcp:
secureBoot: Enabled
1666
compute:
- platform:
gcp:
secureBoot: Enabled
platform:
gcp:
secureBoot: Enabled
each other.
NOTE
Prerequisites
Procedure
controlPlane:
platform:
gcp:
1667
compute:
- platform:
gcp:
platform:
gcp:
IMPORTANT
apiVersion: v1
controlPlane: 2
name: master
platform:
gcp:
type: n2-standard-4
zones:
- us-central1-a
- us-central1-c
tags: 5
replicas: 3
compute: 6
name: worker
platform:
gcp:
type: n2-standard-4
zones:
- us-central1-a
- us-central1-c
tags: 8
1668
- compute-tag1
- compute-tag2
replicas: 0
metadata:
name: test-cluster
networking:
clusterNetwork:
- cidr: 10.128.0.0/14
hostPrefix: 23
machineNetwork:
- cidr: 10.0.0.0/16
serviceNetwork:
- 172.30.0.0/16
platform:
gcp:
tags: 10
- global-tag1
- global-tag2
fips: false 13
1 Specify the public DNS on the host project.
value.
3 7 The controlPlane section is a single mapping, but the compute section is a sequence of mappings.

IMPORTANT

multithreading.
platform.gcp.defaultMachinePlatform.tags parameter applies to both control plane and
1669
value.
11 Specify the main project where the VM instances reside.
12 Specify the region that your VPC network is in.
IMPORTANT
NOTE
process uses.
private cluster, which cannot be accessed from the internet. The default value is External. To use a
shared VPC in a cluster that uses infrastructure that you provision, you must set publish to
Internal. The installation program will no longer be able to access the public DNS zone for the base
domain in the host project.
Prerequisites
NOTE
1670
NOTE
(169.254.169.254).
Procedure
apiVersion: v1
proxy:
must be http.

destinations.
NOTE
1671
NOTE
NOTE
created.
machines.
IMPORTANT
Prerequisites
Procedure
1672
machines.
machine-set.yaml
these machines.

machines:
6. Remove the privateZone sections from the <installation_directory>/manifests/cluster-dns-

02-config.yml DNS configuration file:
kind: DNS
metadata:
name: cluster
spec:
privateZone: 1
status: {}
1 Remove this section completely.
7. Configure the cloud provider for your VPC.
a. Open the <installation_directory>/manifests/cloud-provider-config.yaml file.
1673
a. Open the <installation_directory>/manifests/cloud-provider-config.yaml file.
b. Add the network-project-id parameter and set its value to the ID of project that hosts the
shared VPC network.
c. Add the network-name parameter and set its value to the name of the shared VPC network
that hosts the OpenShift Container Platform cluster.
d. Replace the value of the subnetwork-name parameter with the value of the shared VPC
subnet that hosts your compute machines.
The contents of the <installation_directory>/manifests/cloud-provider-config.yaml

resemble the following example:
config: |+
[global]
project-id = example-project
regional = true
multizone = true
node-tags = opensh-ptzzx-master
node-tags = opensh-ptzzx-worker
node-instance-prefix = opensh-ptzzx
external-instance-groups-prefix = opensh-ptzzx
network-project-id = example-shared-vpc
network-name = example-network
subnetwork-name = example-worker-subnet
8. If you deploy a cluster that is not on a private network, open the

<installation_directory>/manifests/cluster-ingress-default-ingresscontroller.yaml file and
replace the value of the scope parameter with External. The contents of the file resemble the
following example:
metadata:
name: default
spec:
loadBalancer:
scope: External
status:
availableReplicas: 0
domain: ''
selector: ''
1674
.
├── auth
Prerequisites
your cluster.
Procedure
following command:
Example output
openshift-vw9j6 1
1675
(GCP).
NOTE
Prerequisites
cluster.
Procedure
templates:
$ export BASE_DOMAIN='<base_domain>' 1
$ export BASE_DOMAIN_ZONE_NAME='<base_domain_zone_name>' 2
1 2 Supply the values for the host project.
3 For <installation_directory>, specify the path to the directory that you stored the installation files
in.

1676
IMPORTANT
Hat.
TCP 1936 Metrics
UDP 4789 VXLAN
6081 Geneve
9100- 9101.
1677

Manager template.
NOTE
Prerequisites
Procedure
$ export CLUSTER_NETWORK=(`gcloud compute networks describe

${HOST_PROJECT_NETWORK} --project ${HOST_PROJECT} --account
${HOST_PROJECT_ACCOUNT} --format json | jq -r .selfLink`)

${HOST_PROJECT_CONTROL_SUBNET} --region=${REGION} --project
${HOST_PROJECT} --account ${HOST_PROJECT_ACCOUNT} --format json | jq -r
.selfLink`)
1678

.zones[0] | cut -d "/" -f9`)

.zones[1] | cut -d "/" -f9`)

.zones[2] | cut -d "/" -f9`)

imports:
resources:
type: 02_lb_ext.py
properties:
type: 02_lb_int.py
properties:
region: '${REGION}'
zones: 6
- '${ZONE_0}'
- '${ZONE_1}'
- '${ZONE_2}'
EOF
and us-east1-d.
1679


resources = [{
'properties': {
}
}, {
'properties': {
'port': 6080,
}
}, {
'properties': {
'instances': []
}
}, {
'properties': {
'portRange': '6443'
}
}]
1680
backends = []
backends.append({
})
resources = [{
'properties': {
}
}, {
'properties': {
'port': 6443,
},
'type': "HTTPS"
}
}, {
'properties': {
check.selfLink)'],
'protocol': 'TCP',
'timeoutSec': 120
}
}, {
'properties': {
service.selfLink)',
'ports': ['6443','22623'],
1681
}
}]

resources.append({
'properties': {
'namedPorts': [
{
'name': 'ignition',
'port': 22623
}, {
'name': 'https',
'port': 6443
}
],
'zone': zone
}
})
cluster.

Manager template.
NOTE
Prerequisites
Procedure
1682

imports:
- path: 02_dns.py
resources:
- name: cluster-dns
type: 02_dns.py
properties:
EOF
$ gcloud deployment-manager deployments create ${INFRA_ID}-dns --config 02_dns.yaml --

project ${HOST_PROJECT} --account ${HOST_PROJECT_ACCOUNT}

$ gcloud dns record-sets transaction start --zone ${INFRA_ID}-private-zone --project
${HOST_PROJECT} --account ${HOST_PROJECT_ACCOUNT}
private-zone --project ${HOST_PROJECT} --account ${HOST_PROJECT_ACCOUNT}
private-zone --project ${HOST_PROJECT} --account ${HOST_PROJECT_ACCOUNT}
$ gcloud dns record-sets transaction execute --zone ${INFRA_ID}-private-zone --project

$ gcloud --account=${HOST_PROJECT_ACCOUNT} --project=${HOST_PROJECT} dns
record-sets transaction start --zone ${BASE_DOMAIN_ZONE_NAME}
record-sets transaction add ${CLUSTER_PUBLIC_IP} --name
1683

record-sets transaction execute --zone ${BASE_DOMAIN_ZONE_NAME}
resources = [{
'properties': {
'description': '',
'networks': [{
}]
}
}
}]

template.
NOTE
Prerequisites
1684
Procedure

imports:
resources:
properties:
EOF

03_firewall.yaml --project ${HOST_PROJECT} --account ${HOST_PROJECT_ACCOUNT}
resources = [{
'properties': {
'allowed': [{
'ports': ['22']
}],
1685
}
}, {
'properties': {
'allowed': [{
'ports': ['6443']
}],
}
}, {
'properties': {
'allowed': [{
'ports': ['6080', '6443', '22624']
}],
'sourceRanges': ['35.191.0.0/16', '130.211.0.0/22', '209.85.152.0/22', '209.85.204.0/22'],
}
}, {
'properties': {
'allowed': [{
'ports': ['2379-2380']
}],
}
}, {
'properties': {
'allowed': [{
'ports': ['10257']
},{
'ports': ['10259']
},{
'ports': ['22623']
}],
'sourceTags': [
1686
],
}
}, {
'properties': {
'allowed': [{
},{
'ports': ['22']
}],
'targetTags': [
]
}
}, {
'properties': {
'allowed': [{
'ports': ['4789', '6081']
},{
'ports': ['500', '4500']
},{
},{
'ports': ['9000-9999']
},{
'ports': ['9000-9999']
},{
'ports': ['10250']
},{
'ports': ['30000-32767']
},{
'ports': ['30000-32767']
}],
'sourceTags': [
],
'targetTags': [
1687
]
}
}]

template.
NOTE
Prerequisites
Procedure
cluster requires.

imports:
- path: 03_iam.py
resources:
- name: cluster-iam
type: 03_iam.py
properties:
EOF
1688


6. Assign the permissions that the installation program requires to the service accounts for the
subnets that host the control plane and compute subnets:
a. Grant the networkViewer role of the project that hosts your shared VPC to the master
service account:
$ gcloud --account=${HOST_PROJECT_ACCOUNT} --project=${HOST_PROJECT}

projects add-iam-policy-binding ${HOST_PROJECT} --member
"serviceAccount:${MASTER_SERVICE_ACCOUNT}" --role
"roles/compute.networkViewer"
b. Grant the networkUser role to the master service account for the control plane subnet:

compute networks subnets add-iam-policy-binding
"${HOST_PROJECT_CONTROL_SUBNET}" --member
"serviceAccount:${MASTER_SERVICE_ACCOUNT}" --role "roles/compute.networkUser"
--region ${REGION}
c. Grant the networkUser role to the worker service account for the control plane subnet:

"${HOST_PROJECT_CONTROL_SUBNET}" --member
"serviceAccount:${WORKER_SERVICE_ACCOUNT}" --role
"roles/compute.networkUser" --region ${REGION}
d. Grant the networkUser role to the master service account for the compute subnet:

"${HOST_PROJECT_COMPUTE_SUBNET}" --member
"serviceAccount:${MASTER_SERVICE_ACCOUNT}" --role "roles/compute.networkUser"
--region ${REGION}
e. Grant the networkUser role to the worker service account for the compute subnet:

"${HOST_PROJECT_COMPUTE_SUBNET}" --member
"serviceAccount:${WORKER_SERVICE_ACCOUNT}" --role
"roles/compute.networkUser" --region ${REGION}
1689



resources = [{
'properties': {
}
}, {
'properties': {
}
}]
1690
Procedure
IMPORTANT
available.
gs://<bucket_name>


NOTE
Prerequisites
1691
Procedure



imports:
resources:
properties:
zone: '${ZONE_0}' 3
1692
EOF

04_bootstrap.yaml
7. Add the bootstrap instance to the internal load balancer instance group:
$ gcloud compute instance-groups unmanaged add-instances ${INFRA_ID}-bootstrap-ig --

zone=${ZONE_0} --instances=${INFRA_ID}-bootstrap
8. Add the bootstrap instance group to the internal load balancer backend service:
$ gcloud compute backend-services add-backend ${INFRA_ID}-api-internal-backend-service

--region=${REGION} --instance-group=${INFRA_ID}-bootstrap-ig --instance-group-
zone=${ZONE_0}
resources = [{
'properties': {
}
}, {
1693

'properties': {
'disks': [{
'autoDelete': True,
'boot': True,
}
}],
'metadata': {
'items': [{
'key': 'user-data',
+ '"}},"version":"3.2.0"}}',
}]
},
'accessConfigs': [{
}]
}],
'tags': {
'items': [
]
},
}
}, {
'properties': {
'namedPorts': [
{
'name': 'ignition',
'port': 22623
}, {
'name': 'https',
'port': 6443
}
],
}
}]
1694

NOTE
Prerequisites
Procedure

imports:
resources:
properties:
zones: 2
- '${ZONE_0}'
- '${ZONE_1}'
- '${ZONE_2}'
1695
EOF
created.

groups:






1696
resources = [{
'properties': {
'disks': [{
'autoDelete': True,
'boot': True,
}
}],
'metadata': {
'items': [{
'key': 'user-data',
}]
},
}],
}],
'tags': {
'items': [
]
},
}
}, {
'properties': {
'disks': [{
'autoDelete': True,
'boot': True,
1697
}
}],
'metadata': {
'items': [{
'key': 'user-data',
}]
},
}],
}],
'tags': {
'items': [
]
},
}
}, {
'properties': {
'disks': [{
'autoDelete': True,
'boot': True,
}
}],
'metadata': {
'items': [{
'key': 'user-data',
}]
},
}],
}],
'tags': {
'items': [
1698
]
},
}
}]
Prerequisites
Procedure

--log-level info 2

zone=${ZONE_0}
1699

NOTE
Prerequisites
Procedure

${HOST_PROJECT_COMPUTE_SUBNET} --region=${REGION} --project
${HOST_PROJECT} --account ${HOST_PROJECT_ACCOUNT} --format json | jq -r
.selfLink`)
1700


imports:
resources:
type: 06_worker.py
properties:
zone: '${ZONE_0}' 3
- name: 'worker-1'
type: 06_worker.py
properties:
EOF
created.
1701

06_worker.yaml



resources = [{
'properties': {
'disks': [{
'autoDelete': True,
'boot': True,
}
}],
'metadata': {
'items': [{
'key': 'user-data',
}]
},
}],
1702
}],
'tags': {
'items': [
]
},
}
}]

IMPORTANT

Procedure
Portal.
$ tar xvf <file>

$ echo $PATH
Verification
1703
$ oc <command>

Procedure
Portal.

C:\> path
Verification
C:\> oc <command>

Procedure
Portal.
NOTE

$ echo $PATH
Verification
1704
$ oc <command>

Prerequisites
Procedure
$ oc whoami
Example output
system:admin

Prerequisites
Procedure
$ oc get nodes
Example output
1705

NOTE
$ oc get csr
Example output

...
list.
NOTE
NOTE
1706
NOTE
of the node.

NOTE
$ oc get csr
Example output

Pending
Pending
...
1707

$ oc get nodes
Example output

NOTE
9.11.22. Adding the ingress DNS records

DNS zone configuration is removed when creating Kubernetes manifests and generating Ignition
configs. You must manually create DNS records that point at the ingress load balancer. You can create
Prerequisites
Ignition configs.
1708
Procedure
Example output

80:32288/TCP,443:31215/TCP 98
To use A records:


$ gcloud dns record-sets transaction start --zone ${INFRA_ID}-private-zone --project
${INFRA_ID}-private-zone --project ${HOST_PROJECT} --account
${HOST_PROJECT_ACCOUNT}
$ gcloud dns record-sets transaction execute --zone ${INFRA_ID}-private-zone --
project ${HOST_PROJECT} --account ${HOST_PROJECT_ACCOUNT}

$ gcloud dns record-sets transaction start --zone ${BASE_DOMAIN_ZONE_NAME} -
-project ${HOST_PROJECT} --account ${HOST_PROJECT_ACCOUNT}
${BASE_DOMAIN_ZONE_NAME} --project ${HOST_PROJECT} --account
${BASE_DOMAIN_ZONE_NAME} --project ${HOST_PROJECT} --account
current routes:

Example output
1709
9.11.23. Adding ingress firewall rules

The cluster requires several firewall rules. If you do not use a shared VPC, these rules are created by the
Ingress Controller via the GCP cloud provider. When you use a shared VPC, you can either create
cluster-wide firewall rules for all services now or create each rule based on events, when the cluster
requests access. By creating each rule when the cluster requests access, you know exactly which firewall
rules are required. By creating cluster-wide firewall rules, you can apply the same rule set across multiple
clusters.
If you choose to create each rule based on events, you must create firewall rules after you provision the
cluster and during the life of the cluster when the console notifies you that rules are missing. Events that
are similar to the following event are displayed, and you must add the firewall rules that are required:
$ oc get events -n openshift-ingress --field-selector="reason=LoadBalancerManualChange"
Example output
Firewall change required by security admin: `gcloud compute firewall-rules create k8s-fw-
a26e631036a3f46cba28f8df67266d55 --network example-network --description "
{\"kubernetes.io/service-name\":\"openshift-ingress/router-default\", \"kubernetes.io/service-
ip\":\"35.237.236.234\"}\" --allow tcp:443,tcp:80 --source-ranges 0.0.0.0/0 --target-tags exampl-fqzq7-
master,exampl-fqzq7-worker --project example-project`
If you encounter issues when creating these rule-based events, you can configure the cluster-wide
firewall rules while your cluster is running.
9.11.23.1. Creating cluster-wide firewall rules for a shared VPC in GCP
You can create cluster-wide firewall rules to allow the access that the OpenShift Container Platform
cluster requires.

WARNING
If you do not choose to create firewall rules based on cluster events, you must
create cluster-wide firewall rules.
Prerequisites
You exported the variables that the Deployment Manager templates require to deploy your
cluster.
You created the networking and load balancing components in GCP that your cluster requires.
Procedure
1710
Procedure
1. Add a single firewall rule to allow the Google Cloud Engine health checks to access all of the
services. This rule enables the ingress load balancers to determine the health status of their
instances.
$ gcloud compute firewall-rules create --allow='tcp:30000-32767,udp:30000-32767' --

network="${CLUSTER_NETWORK}" --source-
ranges='130.211.0.0/22,35.191.0.0/16,209.85.152.0/22,209.85.204.0/22' --target-
tags="${INFRA_ID}-master,${INFRA_ID}-worker" ${INFRA_ID}-ingress-hc --
account=${HOST_PROJECT_ACCOUNT} --project=${HOST_PROJECT}
2. Add a single firewall rule to allow access to all cluster services:
For an external cluster:
$ gcloud compute firewall-rules create --allow='tcp:80,tcp:443' --

network="${CLUSTER_NETWORK}" --source-ranges="0.0.0.0/0" --target-
tags="${INFRA_ID}-master,${INFRA_ID}-worker" ${INFRA_ID}-ingress --
For a private cluster:
$ gcloud compute firewall-rules create --allow='tcp:80,tcp:443' --

network="${CLUSTER_NETWORK}" --source-ranges=${NETWORK_CIDR} --target-
tags="${INFRA_ID}-master,${INFRA_ID}-worker" ${INFRA_ID}-ingress --
Because this rule only allows traffic on TCP ports 80 and 443, ensure that you add all the ports
that your services use.

Prerequisites
GCP infrastructure.
Procedure
Example output
1711
IMPORTANT
installation.
Example output

Example output

SINCE
1712

Example output
NAMESPACE NAME
1/1 Running 0 30m
1/1 Running 0 29m
1/1 Running 0 29m
...
9r257 1/1 Running 0 37m
1/1 Running 0 35m
1/1 Running 0 35m
1713

9.11.26. Next steps

9.12. INSTALLING A CLUSTER ON GCP IN A RESTRICTED NETWORK

that uses infrastructure that you provide and an internal mirror of the installation release content.
IMPORTANT
While you can install an OpenShift Container Platform cluster by using mirrored
installation release content, your cluster still requires internet access to use the GCP
APIs.
The steps for performing a user-provided infrastructure install are outlined here. Several Deployment
Manager templates are provided to assist in completing these steps or to help model your own. You are
also free to create the required resources through other methods.
IMPORTANT

processes.
users.
You created a registry on your mirror host and obtained the imageContentSources data for
1714
You created a registry on your mirror host and obtained the imageContentSources data for
your version of OpenShift Container Platform.
IMPORTANT
While you might need to grant access to more sites, you must grant access to
*.googleapis.com and accounts.google.com.

your restrictions.
IMPORTANT


1715
9.12.4. Configuring your GCP project

Procedure
IMPORTANT
Prerequisites
Procedure
GCP documentation.
1716

(IAM) API
Procedure
NOTE
Google Domains.
1717
Platform cluster.

bootstrap
NOTE
1718
NOTE
asia-east2
asia-northeast2
asia-south1
europe-north1
europe-west2
europe-west3
europe-west6
southamerica-east1
us-west2
Prerequisites
Procedure
NOTE
1719
NOTE
create the cluster.
NOTE
following lists:
Compute Admin
Role Administrator
Security Admin
Storage Admin
DNS Administrator
1720
use:
Account Roles
roles/storage.admin
roles/storage.admin
9.12.4.7. Required GCP permissions for user-provisioned infrastructure
custom roles with the necessary permissions. The following permissions are required for the user-
1721
compute.routers.get
compute.routes.list
1722
dns.changes.create
dns.changes.get
dns.resourceRecordSets.update
compute.disks.get
1723
compute.disks.list
storage.buckets.get
storage.objects.get
1724
compute.zones.get
compute.zones.list
iam.roles.get
Example 9.104. Required Images permissions for installation
compute.images.create
compute.images.get
compute.images.list
1725
compute.routes.list
dns.changes.create
1726
compute.disks.list
1727
compute.images.list
Example 9.114. Required permissions to get Region related information
compute.regions.get
Example 9.115. Required Deployment Manager permissions
deploymentmanager.deployments.create
deploymentmanager.deployments.delete
deploymentmanager.deployments.get
deploymentmanager.deployments.list
deploymentmanager.manifests.get
deploymentmanager.operations.get
deploymentmanager.resources.list
Optimizing storage
regions:
1728
1729
NOTE
Prerequisites
Procedure
gcloud
gsutil

Hosts Description
1730
Hosts Description
control plane.
IMPORTANT
machines.

System Per Second
(IOPS)[2]

8.6 and later
[3]
1731
Platform.
A2
C2
C2D
C3
E2
M1
N1
N2
N2D
Tau T2D

1732
installation.
IMPORTANT
Procedure
Example output

1733

Example output
...
variant: openshift
version: 4.15.0
metadata:
labels:
storage:
disks:
partitions:
- label: var
number: 5
filesystems:
path: /var
format: xfs
NOTE
1734
NOTE
name.

partition.yaml

(GCP).
Prerequisites
creation.
Procedure
command:
1735
NOTE


1736
platform.gcp field:
network: <existing_vpc>
controlPlaneSubnet: <control_plane_subnet>
computeSubnet: <compute_subnet>
For platform.gcp.network, specify the name for the existing Google VPC. For
platform.gcp.controlPlaneSubnet and platform.gcp.computeSubnet, specify the existing
subnets to deploy the control plane machines and compute machines, respectively.
- mirrors:
- mirrors:
creation.
publish: Internal
IMPORTANT

NOTE
1737
Prerequisites
Procedure
controlPlane:
platform:
gcp:
secureBoot: Enabled
compute:
- platform:
gcp:
secureBoot: Enabled
platform:
gcp:
secureBoot: Enabled
each other.
NOTE
Prerequisites
Procedure
controlPlane:
platform:
1738
gcp:
compute:
- platform:
gcp:
platform:
gcp:
Prerequisites
NOTE
1739
NOTE
(169.254.169.254).
Procedure
apiVersion: v1
proxy:
must be http.

destinations.
NOTE
1740
NOTE
NOTE
created.
machines.
IMPORTANT
Prerequisites
installation, these files are on your mirror host.
Procedure
1741
machines.
machine-set.yaml
4. Optional: If you do not want the cluster to provision compute machines, remove the Kubernetes
manifest files that define the worker machines:
IMPORTANT

these machines.

machines:
kind: DNS
metadata:
name: cluster
spec:
1742
privateZone: 1
publicZone: 2
status: {}
.
├── auth
Optional: Adding the ingress DNS records
Prerequisites
your cluster.
1743
Procedure
following command:
Example output
openshift-vw9j6 1
(GCP).
NOTE
Prerequisites
cluster.
Procedure
templates:
$ export BASE_DOMAIN='<base_domain>'
$ export BASE_DOMAIN_ZONE_NAME='<base_domain_zone_name>'
$ export REGION=`jq -r .gcp.region <installation_directory>/metadata.json`
1744
9.12.8. Creating a VPC in GCP

NOTE
Prerequisites
Procedure
requires.

imports:
- path: 01_vpc.py
resources:
- name: cluster-vpc
type: 01_vpc.py
properties:
EOF
1745
$ gcloud deployment-manager deployments create ${INFRA_ID}-vpc --config 01_vpc.yaml
9.12.8.1. Deployment Manager template for the VPC
resources = [{
'properties': {
}
}, {
'properties': {
}
}, {
'properties': {
}
}, {
'properties': {
'nats': [{
'subnetworks': [{
}]
}, {
'subnetworks': [{
1746

}]
}]
}
}]

TCP 1936 Metrics
UDP 4789 VXLAN
6081 Geneve
1747
9100- 9101.

Manager template.
NOTE
Prerequisites
Procedure
1748
$ export CLUSTER_NETWORK=(`gcloud compute networks describe ${INFRA_ID}-

network --format json | jq -r .selfLink`)

${INFRA_ID}-master-subnet --region=${REGION} --format json | jq -r .selfLink`)

.zones[0] | cut -d "/" -f9`)

.zones[1] | cut -d "/" -f9`)

.zones[2] | cut -d "/" -f9`)

imports:
resources:
type: 02_lb_ext.py
properties:
type: 02_lb_int.py
properties:
region: '${REGION}'
zones: 6
- '${ZONE_0}'
1749
- '${ZONE_1}'
- '${ZONE_2}'
EOF
and us-east1-d.


resources = [{
'properties': {
}
}, {
'properties': {
'port': 6080,
}
1750
}, {
'properties': {
'instances': []
}
}, {
'properties': {
'portRange': '6443'
}
}]
backends = []
backends.append({
})
resources = [{
'properties': {
}
}, {
'properties': {
'port': 6443,
},
1751
'type': "HTTPS"
}
}, {
'properties': {
check.selfLink)'],
'protocol': 'TCP',
'timeoutSec': 120
}
}, {
'properties': {
service.selfLink)',
'ports': ['6443','22623'],
}
}]

resources.append({
'properties': {
'namedPorts': [
{
'name': 'ignition',
'port': 22623
}, {
'name': 'https',
'port': 6443
}
],
'zone': zone
}
})
cluster.
1752
Manager template.
NOTE
Prerequisites
Procedure

imports:
- path: 02_dns.py
resources:
- name: cluster-dns
type: 02_dns.py
properties:
EOF
$ gcloud deployment-manager deployments create ${INFRA_ID}-dns --config 02_dns.yaml
1753

private-zone
private-zone

$ gcloud dns record-sets transaction add ${CLUSTER_PUBLIC_IP} --name
$ gcloud dns record-sets transaction execute --zone ${BASE_DOMAIN_ZONE_NAME}
resources = [{
'properties': {
'description': '',
'networks': [{
}]
}
}
}]

template.
NOTE
1754
NOTE
Prerequisites
Procedure

imports:
resources:
properties:
EOF

03_firewall.yaml
1755
resources = [{
'properties': {
'allowed': [{
'ports': ['22']
}],
}
}, {
'properties': {
'allowed': [{
'ports': ['6443']
}],
}
}, {
'properties': {
'allowed': [{
'ports': ['6080', '6443', '22624']
}],
'sourceRanges': ['35.191.0.0/16', '130.211.0.0/22', '209.85.152.0/22', '209.85.204.0/22'],
}
}, {
'properties': {
'allowed': [{
'ports': ['2379-2380']
}],
}
}, {
1756

'properties': {
'allowed': [{
'ports': ['10257']
},{
'ports': ['10259']
},{
'ports': ['22623']
}],
'sourceTags': [
],
}
}, {
'properties': {
'allowed': [{
},{
'ports': ['22']
}],
'targetTags': [
]
}
}, {
'properties': {
'allowed': [{
'ports': ['4789', '6081']
},{
'ports': ['500', '4500']
},{
},{
'ports': ['9000-9999']
},{
'ports': ['9000-9999']
1757
},{
'ports': ['10250']
},{
'ports': ['30000-32767']
},{
'ports': ['30000-32767']
}],
'sourceTags': [
],
'targetTags': [
]
}
}]

template.
NOTE
Prerequisites
Procedure
cluster requires.
1758
imports:
- path: 03_iam.py
resources:
- name: cluster-iam
type: 03_iam.py
properties:
EOF


6. Export the variable for the subnet that hosts the compute machines:
$ export COMPUTE_SUBNET=(`gcloud compute networks subnets describe ${INFRA_ID}-

worker-subnet --region=${REGION} --format json | jq -r .selfLink`)



1759
resources = [{
'properties': {
}
}, {
'properties': {
}
}]
Procedure
IMPORTANT
available.
1760
gs://<bucket_name>


NOTE
Prerequisites
Procedure

1761


imports:
resources:
properties:
zone: '${ZONE_0}' 3
EOF
1762

04_bootstrap.yaml
Manager, so you must add the bootstrap machine manually.
a. Add the bootstrap instance to the internal load balancer instance group:
$ gcloud compute instance-groups unmanaged add-instances \

${INFRA_ID}-bootstrap-ig --zone=${ZONE_0} --instances=${INFRA_ID}-bootstrap
b. Add the bootstrap instance group to the internal load balancer backend service:
$ gcloud compute backend-services add-backend \

${INFRA_ID}-api-internal-backend-service --region=${REGION} --instance-
group=${INFRA_ID}-bootstrap-ig --instance-group-zone=${ZONE_0}
resources = [{
'properties': {
}
}, {
'properties': {
'disks': [{
'autoDelete': True,
'boot': True,
}
}],
'metadata': {
'items': [{
'key': 'user-data',
+ '"}},"version":"3.2.0"}}',
}]
},
1763
'accessConfigs': [{
}]
}],
'tags': {
'items': [
]
},
}
}, {
'properties': {
'namedPorts': [
{
'name': 'ignition',
'port': 22623
}, {
'name': 'https',
'port': 6443
}
],
}
}]

NOTE
Prerequisites
1764
Procedure

imports:
resources:
properties:
zones: 2
- '${ZONE_0}'
- '${ZONE_1}'
- '${ZONE_2}'
EOF
created.
1765

groups:






resources = [{
'properties': {
'disks': [{
'autoDelete': True,
'boot': True,
1766
}
}],
'metadata': {
'items': [{
'key': 'user-data',
}]
},
}],
}],
'tags': {
'items': [
]
},
}
}, {
'properties': {
'disks': [{
'autoDelete': True,
'boot': True,
}
}],
'metadata': {
'items': [{
'key': 'user-data',
}]
},
}],
}],
'tags': {
'items': [
]
1767
},
}
}, {
'properties': {
'disks': [{
'autoDelete': True,
'boot': True,
}
}],
'metadata': {
'items': [{
'key': 'user-data',
}]
},
}],
}],
'tags': {
'items': [
]
},
}
}]
Prerequisites
1768
Procedure

--log-level info 2

zone=${ZONE_0}

NOTE
1769
Prerequisites
Procedure

${INFRA_ID}-worker-subnet --region=${REGION} --format json | jq -r .selfLink`)


imports:
resources:
type: 06_worker.py
properties:
zone: '${ZONE_0}' 3
1770
- name: 'worker-1'
type: 06_worker.py
properties:
EOF
created.

06_worker.yaml



1771
resources = [{
'properties': {
'disks': [{
'autoDelete': True,
'boot': True,
}
}],
'metadata': {
'items': [{
'key': 'user-data',
}]
},
}],
}],
'tags': {
'items': [
]
},
}
}]

Prerequisites
1772
Procedure
$ oc whoami
Example output
system:admin

Procedure
OperatorHub object:

TIP

Prerequisites
Procedure
1773
Procedure
$ oc get nodes
Example output

NOTE
$ oc get csr
Example output

...
list.
NOTE
NOTE
1774
NOTE
of the node.

NOTE
$ oc get csr
Example output

Pending
Pending
...
1775

$ oc get nodes
Example output

NOTE
9.12.22. Optional: Adding the ingress DNS records

If you removed the DNS zone configuration when creating Kubernetes manifests and generating Ignition
configs, you must manually create DNS records that point at the ingress load balancer. You can create
Prerequisites
Ignition configs.
1776
Procedure
Example output

80:32288/TCP,443:31215/TCP 98
To use A records:


${INFRA_ID}-private-zone

current routes:

Example output
1777

Prerequisites
GCP infrastructure.
Procedure
Example output
IMPORTANT
installation.
Example output

1778
Example output

SINCE
Example output
NAMESPACE NAME
1779
1/1 Running 0 30m
1/1 Running 0 29m
1/1 Running 0 29m
...
9r257 1/1 Running 0 37m
1/1 Running 0 35m
1/1 Running 0 35m

9.12.25. Next steps

9.13. INSTALLING A THREE-NODE CLUSTER ON GCP
1780
In OpenShift Container Platform version 4.15, you can install a three-node cluster on Google Cloud
Platform (GCP). A three-node cluster consists of three control plane machines, which also act as
compute machines. This type of cluster provides a smaller, more resource efficient cluster, for cluster
administrators and developers to use for testing, development, and production.

NOTE
Prerequisites
Procedure
apiVersion: v1
compute:
- name: worker
platform: {}
replicas: 0
# ...
user-provisioned infrastructure in GCP by using Deployment Manager templates".
kind: Scheduler
metadata:
1781
name: cluster
spec:
policy:
name: ""
status: {}
9.13.2. Next steps

Installing a cluster on GCP with customizations
Installing a cluster on user-provisioned infrastructure in GCP by using Deployment Manager

templates
9.14. INSTALLATION CONFIGURATION PARAMETERS FOR GCP

Before you deploy an OpenShift Container Platform cluster on Google Cloud Platform (GCP), you
provide parameters to customize your cluster and the platform that hosts it. When you create the
install-config.yaml file, you provide values for the required parameters through the command line. You
can then modify the install-config.yaml file to customize your cluster further.
9.14.1. Available installation configuration parameters for GCP

The following tables specify the required, optional, and GCP-specific installation configuration
NOTE

versions.
1782

combination of the
baseDomain and
<metadata.name>.

consumed.
name: subdomains of
{{.metadata.name}}.
{{.baseDomain}}.

alibabacloud, aws,
ibmcloud, nutanix,
1783

"auth":"b3Blb=",
}
}
}
NOTE

NOTE
You cannot modify

by the networking
object after
installation.

1784

networking:
networking:
- cidr: 10.128.0.0/14
hostPrefix: 23

An IPv4 network.

23 provides 510 (2^(32 - 23) - 2) pod
IP addresses.
serviceNetwork:
networking:
- 172.30.0.0/16

networking:
networking:
machineNetwork:
- cidr: 10.0.0.0/16

NOTE
the CIDR that the
in.
1785



et:


section.

1786

and arm64.

cores.
IMPORTANT
If you disable
simultaneous
that your capacity
the dramatically
decreased machine
performance.

name:
value.

1787


and arm64.

cores.
IMPORTANT
If you disable
simultaneous
that your capacity
the dramatically
decreased machine
performance.

name:

parameter value.
1788

replicas:

1789

IMPORTANT
To enable FIPS mode

must run the
from a Red Hat
Enterprise Linux
(RHEL) computer
in FIPS mode. For
more information
about configuring
FIPS mode on RHEL,
see Installing the
Enterprise Linux
(RHEL) or Red Hat
Enterprise Linux
CoreOS (RHCOS)
OpenShift Container
Platform core
components use the
RHEL cryptographic
libraries that have
been submitted to
NIST for FIPS 140-
only the x86_64,
ppc64le, and s390x
architectures.
NOTE

File storage, you
cannot enable FIPS
mode.

1790


ces:
mirrors:

value is External.

NOTE
For production
OpenShift Container
which you want to
SSH key that your
ssh-agent process
uses.
NOTE
If you are installing on GCP into a shared virtual private cloud (VPC),
credentialsMode must be set to Passthrough or Manual.
IMPORTANT

1791
9.14.1.4. Additional Google Cloud Platform (GCP) configuration parameters
Additional GCP configuration parameters are described in the following table:
Table 9.47. Additional GCP parameters
Param Description Values

eter
Optional. By default, the installation program downloads and String. The name of GCP
con installs the Red Hat Enterprise Linux CoreOS (RHCOS) image project where the image is
trol that is used to boot control plane machines. You can override located.
Pla the default behavior by specifying the location of a custom
ne: RHCOS image that the installation program is to use for control
plane machines only.
plat
for
m:
gcp
:
osI
ma
ge:
proj
ect:
The name of the custom RHCOS image that the installation String. The name of the
con program is to use to boot control plane machines. If you use RHCOS image.
trol controlPlane.platform.gcp.osImage.project, this field is
Pla required.
ne:
plat
for
m:
gcp
:
osI
ma
ge:
na
me:
1792

eter
co installs the RHCOS image that is used to boot compute project where the image is
mp machines. You can override the default behavior by specifying located.
ute: the location of a custom RHCOS image that the installation
program is to use for compute machines only.
plat
for
m:
gcp
:
osI
ma
ge:
proj
ect:
co program is to use to boot compute machines. If you use RHCOS image.
mp compute.platform.gcp.osImage.project , this field is
ute: required.
plat
for
m:
gcp
:
osI
ma
ge:
na
me:
The name of the existing Virtual Private Cloud (VPC) where you String.
plat want to deploy your cluster. If you want to deploy your cluster
for into a shared VPC, you must set
m: platform.gcp.networkProjectID with the name of the GCP
project that contains the shared VPC.
gcp
:
net
wor
k:
1793

eter
Optional. The name of the GCP project that contains the shared String.
plat VPC where you want to deploy your cluster.
for
m:
gcp
:
net
wor
kPr
oje
ctI
D:
The name of the GCP project where the installation program String.
plat installs the cluster.
for
m:
gcp
:
proj
ectI
D:
The name of the GCP region that hosts your cluster. Any valid region name, such as
plat us-central1 .
for
m:
gcp
:
regi
on:
1794

eter
The name of the existing subnet where you want to deploy your The subnet name.
plat control plane machines.
for
m:
gcp
:
con
trol
Pla
ne
Su
bne
t:
The name of the existing subnet where you want to deploy your The subnet name.
plat compute machines.
for
m:
gcp
:
co
mp
ute
Su
bne
t:
1795

eter
The availability zones where the installation program creates A list of valid GCP availability
plat machines. zones, such as us-central1-a,
for in a YAML sequence.
m:
IMPORTA
gcp
NT
:
When running
def your cluster
ault on GCP 64-
Ma bit ARM
infrastructure
chi
s, ensure that
ne you use a
Plat zone where
for Ampere Altra
m: Arm CPU’s
are available.
zon You can find
which zones
es: are
compatible
with 64-bit
ARM
processors in
the "GCP
availability
zones" link.
1796

eter
The size of the disk in gigabytes (GB). Any size between 16 GB and
plat 65536 GB.
for
m:
gcp
:
def
ault
Ma
chi
ne
Plat
for
m:
os
Dis
k:
disk
Siz
eG
B:
The GCP disk type. The default disk type for all
plat machines. Control plane
for nodes must use the pd-ssd
m: disk type. Compute nodes can
use the pd-ssd, pd-
gcp balanced , or pd-standard
: disk types.
def
ault
Ma
chi
ne
Plat
for
m:
os
Dis
k:
disk
Typ
e:
1797

eter
plat installs the RHCOS image that is used to boot control plane and project where the image is
for compute machines. You can override the default behavior by located.
m: specifying the location of a custom RHCOS image that the
installation program is to use for both types of machines.
gcp
:
def
ault
Ma
chi
ne
Plat
for
m:
osI
ma
ge:
proj
ect:
plat program is to use to boot control plane and compute machines. RHCOS image.
for If you use
m: platform.gcp.defaultMachinePlatform.osImage.project,
this field is required.
gcp
:
def
ault
Ma
chi
ne
Plat
for
m:
osI
ma
ge:
na
me:
1798

eter
Optional. Additional network tags to add to the control plane One or more strings, for
plat and compute machines. example network-tag1.
for
m:
gcp
:
def
ault
Ma
chi
ne
Plat
for
m:
tag
s:
The GCP machine type for control plane and compute machines. The GCP machine type, for
plat example n1-standard-4 .
for
m:
gcp
:
def
ault
Ma
chi
ne
Plat
for
m:
typ
e:
1799

eter
The name of the customer managed encryption key to be used The encryption key name.
plat for machine disk encryption.
for
m:
gcp
:
def
ault
Ma
chi
ne
Plat
for
m:
os
Dis
k:
enc
rypt
ion
Key
:
km
sKe
y:
na
me:
1800

eter
The name of the Key Management Service (KMS) key ring to The KMS key ring name.
plat which the KMS key belongs.
for
m:
gcp
:
def
ault
Ma
chi
ne
Plat
for
m:
os
Dis
k:
enc
rypt
ion
Key
:
km
sKe
y:
key
Rin
g:
1801

eter
The GCP location in which the KMS key ring exists. The GCP location.
plat
for
m:
gcp
:
def
ault
Ma
chi
ne
Plat
for
m:
os
Dis
k:
enc
rypt
ion
Key
:
km
sKe
y:
loc
atio
n:
1802

eter
The ID of the project in which the KMS key ring exists. This value The GCP project ID.
plat defaults to the value of the platform.gcp.projectID parameter
for if it is not set.
m:
gcp
:
def
ault
Ma
chi
ne
Plat
for
m:
os
Dis
k:
enc
rypt
ion
Key
:
km
sKe
y:
proj
ectI
D:
1803

eter
The GCP service account used for the encryption request for The GCP service account
plat control plane and compute machines. If absent, the Compute email, for example
for Engine default service account is used. For more information <service_account_name>
m: about GCP service accounts, see Google’s documentation on @<project_id>.iam.gservi
service accounts. ceaccount.com.
gcp
:
def
ault
Ma
chi
ne
Plat
for
m:
os
Dis
k:
enc
rypt
ion
Key
:
km
sKe
ySe
rvic
eAc
cou
nt:
1804

eter
Whether to enable Shielded VM secure boot for all machines in Enabled or Disabled. The
plat the cluster. Shielded VMs have additional security protocols such default value is Disabled.
for as secure boot, firmware and integrity monitoring, and rootkit
m: protection. For more information on Shielded VMs, see Google’s
documentation on Shielded VMs.
gcp
:
def
ault
Ma
chi
ne
Plat
for
m:
sec
ure
Bo
ot:
Whether to use Confidential VMs for all machines in the cluster. Enabled or Disabled. The
plat Confidential VMs provide encryption for data during processing. default value is Disabled.
for For more information on Confidential computing, see Google’s
m: documentation on Confidential computing.
gcp
:
def
ault
Ma
chi
ne
Plat
for
m:
con
fide
ntia
lCo
mp
ute:
1805

eter
Specifies the behavior of all VMs during a host maintenance Terminate or Migrate. The
plat event, such as a software or hardware update. For Confidential default value is Migrate.
for VMs, this parameter must be set to Terminate . Confidential
m: VMs do not support live VM migration.
gcp
:
def
ault
Ma
chi
ne
Plat
for
m:
on
Ho
stM
aint
ena
nce
:
1806

eter
con for control plane machine disk encryption.
trol
Pla
ne:
plat
for
m:
gcp
:
os
Dis
k:
enc
rypt
ion
Key
:
km
sKe
y:
na
me:
1807

eter
For control plane machines, the name of the KMS key ring to The KMS key ring name.
con which the KMS key belongs.
trol
Pla
ne:
plat
for
m:
gcp
:
os
Dis
k:
enc
rypt
ion
Key
:
km
sKe
y:
key
Rin
g:
1808

eter
For control plane machines, the GCP location in which the key The GCP location for the key
con ring exists. For more information about KMS locations, see ring.
trol Google’s documentation on Cloud KMS locations.
Pla
ne:
plat
for
m:
gcp
:
os
Dis
k:
enc
rypt
ion
Key
:
km
sKe
y:
loc
atio
n:
1809

eter
For control plane machines, the ID of the project in which the The GCP project ID.
con KMS key ring exists. This value defaults to the VM project ID if
trol not set.
Pla
ne:
plat
for
m:
gcp
:
os
Dis
k:
enc
rypt
ion
Key
:
km
sKe
y:
proj
ectI
D:
1810

eter
con control plane machines. If absent, the Compute Engine default email, for example
trol service account is used. For more information about GCP service <service_account_name>
Pla accounts, see Google’s documentation on service accounts. @<project_id>.iam.gservi
ne: ceaccount.com.
plat
for
m:
gcp
:
os
Dis
k:
enc
rypt
ion
Key
:
km
sKe
ySe
rvic
eAc
cou
nt:
1811

eter
The size of the disk in gigabytes (GB). This value applies to Any integer between 16 and
con control plane machines. 65536.
trol
Pla
ne:
plat
for
m:
gcp
:
os
Dis
k:
disk
Siz
eG
B:
The GCP disk type for control plane machines. Control plane machines must
con use the pd-ssd disk type,
trol which is the default.
Pla
ne:
plat
for
m:
gcp
:
os
Dis
k:
disk
Typ
e:
1812

eter
Optional. Additional network tags to add to the control plane One or more strings, for
con machines. If set, this parameter overrides the example control-plane-
trol platform.gcp.defaultMachinePlatform.tags parameter for tag1 .
Pla control plane machines.
ne:
plat
for
m:
gcp
:
tag
s:
The GCP machine type for control plane machines. If set, this The GCP machine type, for
con parameter overrides the example n1-standard-4 .
trol platform.gcp.defaultMachinePlatform.type parameter.
Pla
ne:
plat
for
m:
gcp
:
typ
e:
1813

eter
con control plane machines. zones, such as us-central1-a,
trol in a YAML sequence.
Pla
ne: IMPORTA
NT
plat
for When running
m: your cluster
on GCP 64-
gcp bit ARM
infrastructure
:
s, ensure that
you use a
zon zone where
es: Ampere Altra
Arm CPU’s
are available.
You can find
which zones
are
compatible
with 64-bit
ARM
processors in
the "GCP
availability
zones" link.
1814

eter
Whether to enable Shielded VM secure boot for control plane Enabled or Disabled. The
con machines. Shielded VMs have additional security protocols such default value is Disabled.
trol as secure boot, firmware and integrity monitoring, and rootkit
Pla protection. For more information on Shielded VMs, see Google’s
ne: documentation on Shielded VMs.
plat
for
m:
gcp
:
sec
ure
Bo
ot:
Whether to enable Confidential VMs for control plane machines. Enabled or Disabled. The
con Confidential VMs provide encryption for data while it is being default value is Disabled.
trol processed. For more information on Confidential VMs, see
Pla Google’s documentation on Confidential Computing.
ne:
plat
for
m:
gcp
:
con
fide
ntia
lCo
mp
ute:
1815

eter
Specifies the behavior of control plane VMs during a host Terminate or Migrate. The
con maintenance event, such as a software or hardware update. For default value is Migrate.
trol Confidential VMs, this parameter must be set to Terminate .
Pla Confidential VMs do not support live VM migration.
ne:
plat
for
m:
gcp
:
on
Ho
stM
aint
ena
nce
:
1816

eter
co for compute machine disk encryption.
mp
ute:
plat
for
m:
gcp
:
os
Dis
k:
enc
rypt
ion
Key
:
km
sKe
y:
na
me:
1817

eter
For compute machines, the name of the KMS key ring to which The KMS key ring name.
co the KMS key belongs.
mp
ute:
plat
for
m:
gcp
:
os
Dis
k:
enc
rypt
ion
Key
:
km
sKe
y:
key
Rin
g:
1818

eter
For compute machines, the GCP location in which the key ring The GCP location for the key
co exists. For more information about KMS locations, see Google’s ring.
mp documentation on Cloud KMS locations.
ute:
plat
for
m:
gcp
:
os
Dis
k:
enc
rypt
ion
Key
:
km
sKe
y:
loc
atio
n:
1819

eter
For compute machines, the ID of the project in which the KMS The GCP project ID.
co key ring exists. This value defaults to the VM project ID if not set.
mp
ute:
plat
for
m:
gcp
:
os
Dis
k:
enc
rypt
ion
Key
:
km
sKe
y:
proj
ectI
D:
1820

eter
co compute machines. If this value is not set, the Compute Engine email, for example
mp default service account is used. For more information about <service_account_name>
ute: GCP service accounts, see Google’s documentation on service @<project_id>.iam.gservi
accounts. ceaccount.com.
plat
for
m:
gcp
:
os
Dis
k:
enc
rypt
ion
Key
:
km
sKe
ySe
rvic
eAc
cou
nt:
1821

eter
The size of the disk in gigabytes (GB). This value applies to Any integer between 16 and
co compute machines. 65536.
mp
ute:
plat
for
m:
gcp
:
os
Dis
k:
disk
Siz
eG
B:
The GCP disk type for compute machines. pd-ssd, pd-standard , or

co pd-balanced. The default is
mp pd-ssd.
ute:
plat
for
m:
gcp
:
os
Dis
k:
disk
Typ
e:
1822

eter
Optional. Additional network tags to add to the compute One or more strings, for
co machines. If set, this parameter overrides the example compute-network-
mp platform.gcp.defaultMachinePlatform.tags parameter for tag1 .
ute: compute machines.
plat
for
m:
gcp
:
tag
s:
The GCP machine type for compute machines. If set, this The GCP machine type, for
co parameter overrides the example n1-standard-4 .
mp platform.gcp.defaultMachinePlatform.type parameter.
ute:
plat
for
m:
gcp
:
typ
e:
1823

eter
co compute machines. zones, such as us-central1-a,
mp in a YAML sequence.
ute:
IMPORTA
plat
NT
for
m: When running
your cluster
gcp on GCP 64-
: bit ARM
infrastructure
s, ensure that
zon you use a
es: zone where
Ampere Altra
Arm CPU’s
are available.
You can find
which zones
are
compatible
with 64-bit
ARM
processors in
the "GCP
availability
zones" link.
Whether to enable Shielded VM secure boot for compute Enabled or Disabled. The
co machines. Shielded VMs have additional security protocols such default value is Disabled.
mp as secure boot, firmware and integrity monitoring, and rootkit
ute: protection. For more information on Shielded VMs, see Google’s
documentation on Shielded VMs.
plat
for
m:
gcp
:
sec
ure
Bo
ot:
1824

eter
Whether to enable Confidential VMs for compute machines. Enabled or Disabled. The
co Confidential VMs provide encryption for data while it is being default value is Disabled.
mp processed. For more information on Confidential VMs, see
ute: Google’s documentation on Confidential Computing.
plat
for
m:
gcp
:
con
fide
ntia
lCo
mp
ute:
Specifies the behavior of compute VMs during a host Terminate or Migrate. The
co maintenance event, such as a software or hardware update. For default value is Migrate.
mp Confidential VMs, this parameter must be set to Terminate .
ute: Confidential VMs do not support live VM migration.
plat
for
m:
gcp
:
on
Ho
stM
aint
ena
nce
:
9.15. UNINSTALLING A CLUSTER ON GCP

You can remove a cluster that you deployed to Google Cloud Platform (GCP).

NOTE
1825
NOTE
that the installer did not create or that the installer is unable to access. For example,
some Google Cloud resources require IAM permissions in shared VPC host projects, or
there might be unused health checks that must be deleted .
Prerequisites
Procedure

NOTE
9.15.2. Deleting Google Cloud Platform resources with the Cloud Credential
Operator utility
outside the cluster, you can use the CCO utility (ccoctl) to remove the Google Cloud Platform (GCP)
Prerequisites
Uninstall an OpenShift Container Platform cluster on GCP that uses short-term credentials.
Procedure
1826

--included \ 1
3. Delete the GCP resources that ccoctl created by running the following command:
$ ccoctl gcp delete \

--name=<name> \ 1
--force-delete-custom-roles 3
2 <gcp_project_id> is the GCP project ID in which to delete cloud resources.
3 Optional: This parameter deletes the custom roles that the ccoctl utility creates during
installation. GCP does not permanently delete custom roles immediately. For more
information, see GCP documentation about deleting a custom role .
Verification
To verify that the resources are deleted, query GCP. For more information, refer to GCP
documentation.
1827
CHAPTER 10. INSTALLING ON IBM CLOUD
10.1. PREPARING TO INSTALL ON IBM CLOUD

The installation workflows documented in this section are for IBM Cloud® infrastructure environments.
IBM Cloud® classic is not supported at this time. For more information about the difference between
classic and VPC infrastructures, see the IBM® documentation.
processes.
users.
10.1.2. Requirements for installing OpenShift Container Platform on IBM Cloud

Before installing OpenShift Container Platform on IBM Cloud®, you must create a service account and
configure an IBM Cloud® account. See Configuring an IBM Cloud® account for details about creating an
account, enabling API services, configuring DNS, IBM Cloud® account limits, and supported IBM Cloud®
regions.
You must manually manage your cloud credentials when installing a cluster to IBM Cloud®. Do this by
configuring the Cloud Credential Operator (CCO) for manual mode before you install the cluster. For
more information, see Configuring IAM for IBM Cloud® .
10.1.3. Choosing a method to install OpenShift Container Platform on IBM Cloud

You can install OpenShift Container Platform on IBM Cloud® using installer-provisioned infrastructure.
This process involves using an installation program to provision the underlying infrastructure for your
cluster. Installing OpenShift Container Platform on IBM Cloud® using user-provisioned infrastructure is
not supported at this time.
See Installation process for more information about installer-provisioned installation processes.
You can install a cluster on IBM Cloud® infrastructure that is provisioned by the OpenShift Container
Platform installation program by using one of the following methods:
Installing a customized cluster on IBM Cloud®: You can install a customized cluster on IBM
Cloud® infrastructure that the installation program provisions. The installation program allows
for some customization to be applied at the installation stage. Many other customization options
are available post-installation.
Installing a cluster on IBM Cloud® with network customizations: You can customize your
OpenShift Container Platform network configuration during installation, so that your cluster can
coexist with your existing IP address allocations and adhere to your network requirements.
Installing a cluster on IBM Cloud® into an existing VPC: You can install OpenShift Container
Platform on an existing IBM Cloud®. You can use this installation method if you have constraints
set by the guidelines of your company, such as limits when creating new accounts or
infrastructure.
1828
Virtual Private Cloud (VPC). You can use this method to deploy OpenShift Container Platform
on an internal network that is not visible to the internet.
Installing a cluster on IBM Cloud VPC in a restricted network : You can install OpenShift
Container Platform on IBM Cloud VPC on installer-provisioned infrastructure by using an
internal mirror of the installation release content. You can use this method to install a cluster
that does not require an active internet connection to obtain the software components.
10.1.4. Next steps

Configuring an IBM Cloud® account
10.2. CONFIGURING AN IBM CLOUD ACCOUNT

Before you can install OpenShift Container Platform, you must configure an IBM Cloud® account.
You have an IBM Cloud® account with a subscription. You cannot install OpenShift Container
Platform on a free or trial IBM Cloud® account.
10.2.2. Quotas and limits on IBM Cloud

The OpenShift Container Platform cluster uses a number of IBM Cloud® components, and the default
quotas and limits affect your ability to install OpenShift Container Platform clusters. If you use certain
cluster configurations, deploy your cluster in certain regions, or run multiple clusters from your account,
you might need to request additional resources for your IBM Cloud® account.
For a comprehensive list of the default IBM Cloud® quotas and service limits, see IBM Cloud®'s
documentation for Quotas and service limits .
Virtual Private Cloud (VPC)

Each OpenShift Container Platform cluster creates its own VPC. The default quota of VPCs per region
is 10 and will allow 10 clusters. To have more than 10 clusters in a single region, you must increase this
quota.
Application load balancer

By default, each cluster creates three application load balancers (ALBs):
Internal load balancer for the master API server
External load balancer for the master API server
Load balancer for the router
You can create additional LoadBalancer service objects to create additional ALBs. The default quota
of VPC ALBs are 50 per region. To have more than 50 ALBs, you must increase this quota.
VPC ALBs are supported. Classic ALBs are not supported for IBM Cloud®.
Floating IP address
By default, the installation program distributes control plane and compute machines across all
availability zones within a region to provision the cluster in a highly available configuration. In each
availability zone, a public gateway is created and requires a separate floating IP address.
1829
The default quota for a floating IP address is 20 addresses per availability zone. The default cluster
configuration yields three floating IP addresses:
Two floating IP addresses in the us-east-1 primary zone. The IP address associated with the
bootstrap node is removed after installation.
One floating IP address in the us-east-2 secondary zone.
One floating IP address in the us-east-3 secondary zone.
IBM Cloud® can support up to 19 clusters per region in an account. If you plan to have more than 19
default clusters, you must increase this quota.
Virtual Server Instances (VSI)

By default, a cluster creates VSIs using bx2-4x16 profiles, which includes the following resources by
default:
4 vCPUs
16 GB RAM
The following nodes are created:
One bx2-4x16 bootstrap machine, which is removed after the installation is complete
Three bx2-4x16 control plane nodes
Three bx2-4x16 compute nodes
For more information, see IBM Cloud®'s documentation on supported profiles.
Table 10.1. VSI component quotas and limits
VSI component Default IBM Default cluster configuration Maximum number

Cloud® quota of clusters
vCPU 200 vCPUs per 28 vCPUs, or 24 vCPUs after bootstrap 8 per region
region removal
RAM 1600 GB per 112 GB, or 96 GB after bootstrap removal 16 per region
region
Storage 18 TB per region 1050 GB, or 900 GB after bootstrap 19 per region
removal
If you plan to exceed the resources stated in the table, you must increase your IBM Cloud® account
quota.
Block Storage Volumes

For each VPC machine, a block storage device is attached for its boot volume. The default cluster
configuration creates seven VPC machines, resulting in seven block storage volumes. Additional
Kubernetes persistent volume claims (PVCs) of the IBM Cloud® storage class create additional block
storage volumes. The default quota of VPC block storage volumes are 300 per region. To have more
than 300 volumes, you must increase this quota.
1830
10.2.3. Configuring DNS resolution

How you configure DNS resolution depends on the type of OpenShift Container Platform cluster you
are installing:
If you are installing a public cluster, you use IBM Cloud Internet Services (CIS).
If you are installing a private cluster, you use IBM Cloud® DNS Services (DNS Services)
10.2.3.1. Using IBM Cloud Internet Services for DNS resolution
The installation program uses IBM Cloud® Internet Services (CIS) to configure cluster DNS resolution
and provide name lookup for a public cluster.
NOTE
This offering does not support IPv6, so dual stack or IPv6 environments are not possible.
You must create a domain zone in CIS in the same account as your cluster. You must also ensure the
zone is authoritative for the domain. You can do this using a root domain or subdomain.
Prerequisites
You have installed the IBM Cloud® CLI .
You have an existing domain and registrar. For more information, see the IBM® documentation.
Procedure
1. Create a CIS instance to use with your cluster:
a. Install the CIS plugin:
$ ibmcloud plugin install cis
b. Create the CIS instance:
$ ibmcloud cis instance-create <instance_name> standard 1
1 At a minimum, a Standard plan is required for CIS to manage the cluster subdomain
and its DNS records.
2. Connect an existing domain to your CIS instance:
a. Set the context instance for CIS:
$ ibmcloud cis instance-set <instance_name> 1
1 The instance cloud resource name.
b. Add the domain for CIS:
1831
$ ibmcloud cis domain-add <domain_name> 1
1 The fully qualified domain name. You can use either the root domain or subdomain
value as the domain name, depending on which you plan to configure.
NOTE
A root domain uses the form openshiftcorp.com. A subdomain uses the

form clusters.openshiftcorp.com.
3. Open the CIS web console , navigate to the Overview page, and note your CIS name servers.
These name servers will be used in the next step.
4. Configure the name servers for your domains or subdomains at the domain’s registrar or DNS
provider. For more information, see the IBM Cloud® documentation.
10.2.3.2. Using IBM Cloud DNS Services for DNS resolution
The installation program uses IBM Cloud® DNS Services to configure cluster DNS resolution and
provide name lookup for a private cluster.
You configure DNS resolution by creating a DNS services instance for the cluster, and then adding a
DNS zone to the DNS Services instance. Ensure that the zone is authoritative for the domain. You can
do this using a root domain or subdomain.
NOTE
IBM Cloud® does not support IPv6, so dual stack or IPv6 environments are not possible.
Prerequisites
You have installed the IBM Cloud® CLI .
You have an existing domain and registrar. For more information, see the IBM® documentation.
Procedure
1. Create a DNS Services instance to use with your cluster:
a. Install the DNS Services plugin by running the following command:
$ ibmcloud plugin install cloud-dns-services
b. Create the DNS Services instance by running the following command:
$ ibmcloud dns instance-create <instance-name> standard-dns 1
1 At a minimum, a Standard plan is required for DNS Services to manage the cluster
subdomain and its DNS records.
2. Create a DNS zone for the DNS Services instance:
1832
a. Set the target operating DNS Services instance by running the following command:
$ ibmcloud dns instance-target <instance-name>
b. Add the DNS zone to the DNS Services instance by running the following command:
$ ibmcloud dns zone-create <zone-name> 1
1 The fully qualified zone name. You can use either the root domain or subdomain value
as the zone name, depending on which you plan to configure. A root domain uses the
form openshiftcorp.com. A subdomain uses the form clusters.openshiftcorp.com.
3. Record the name of the DNS zone you have created. As part of the installation process, you
must update the install-config.yaml file before deploying the cluster. Use the name of the DNS
zone as the value for the baseDomain parameter.
NOTE
You do not have to manage permitted networks or configure an "A" DNS resource
record. As required, the installation program configures these resources automatically.
10.2.4. IBM Cloud IAM Policies and API Key

To install OpenShift Container Platform into your IBM Cloud® account, the installation program requires
an IAM API key, which provides authentication and authorization to access IBM Cloud® service APIs. You
can use an existing IAM API key that contains the required policies or create a new one.
For an IBM Cloud® IAM overview, see the IBM Cloud® documentation.
10.2.4.1. Required access policies
You must assign the required access policies to your IBM Cloud® account.
Table 10.2. Required access policies
Servic Service Access policy Platform access Service access

e type scope
Accoun IAM Identity All resources or a Editor, Operator, Service ID creator

t Service subset of Viewer,
manag resources [1] Administrator
ement
Accoun Identity and All resources Editor, Operator,

t Access Viewer,
manag Management Administrator
ement
[2]
1833
Servic Service Access policy Platform access Service access

e type scope
Accoun Resource group All resource groups Administrator

t only in the account
manag
ement
IAM Cloud Object All resources or a Editor, Operator, Reader, Writer, Manager,
service Storage subset of Viewer, Content Reader, Object
s resources [1] Administrator Reader, Object Writer
IAM Internet Services All resources or a Editor, Operator, Reader, Writer, Manager
service subset of Viewer,
s resources [1] Administrator
IAM DNS Services All resources or a Editor, Operator, Reader, Writer, Manager
service subset of Viewer,
IAM VPC Infrastructure All resources or a Editor, Operator, Reader, Writer, Manager
service Services subset of Viewer,
1. The policy access scope should be set based on how granular you want to assign access. The
scope can be set to All resources or Resources based on selected attributes.
2. Optional: This access policy is only required if you want the installation program to create a
resource group. For more information about resource groups, see the IBM® documentation.
10.2.4.2. Access policy assignment
In IBM Cloud® IAM, access policies can be attached to different subjects:
Access group (Recommended)
Service ID
User
The recommended method is to define IAM access policies in an access group. This helps organize all
the access required for OpenShift Container Platform and enables you to onboard users and service IDs
to this group. You can also assign access to users and service IDs directly, if desired.
10.2.4.3. Creating an API key
You must create a user API key or a service ID API key for your IBM Cloud® account.
Prerequisites
You have assigned the required access policies to your IBM Cloud® account.
1834
You have attached you IAM access policies to an access group, or other appropriate resource.
Procedure
Create an API key, depending on how you defined your IAM access policies.
For example, if you assigned your access policies to a user, you must create a user API key. If you
assigned your access policies to a service ID, you must create a service ID API key . If your access
policies are assigned to an access group, you can use either API key type. For more information
on IBM Cloud® API keys, see Understanding API keys .
10.2.5. Supported IBM Cloud regions

You can deploy an OpenShift Container Platform cluster to the following regions:
au-syd (Sydney, Australia)
br-sao (Sao Paulo, Brazil)
ca-tor (Toronto, Canada)
eu-de (Frankfurt, Germany)
eu-gb (London, United Kingdom)
eu-es (Madrid, Spain)
jp-osa (Osaka, Japan)
jp-tok (Tokyo, Japan)
us-east (Washington DC, United States)
us-south (Dallas, United States)
NOTE
Deploying your cluster in the eu-es (Madrid, Spain) region is not supported for OpenShift
Container Platform 4.14.6 and earlier versions.
10.2.6. Next steps

Configuring IAM for IBM Cloud®
10.3. CONFIGURING IAM FOR IBM CLOUD

In environments where the cloud identity and access management (IAM) APIs are not reachable, you
must put the Cloud Credential Operator (CCO) into manual mode before you install the cluster.

resource definitions (CRDs). You can configure the CCO to suit the security requirements of your
organization by setting different values for the credentialsMode parameter in the install-config.yaml
file.
Storing an administrator-level credential secret in the cluster kube-system project is not supported for
1835
Storing an administrator-level credential secret in the cluster kube-system project is not supported for
IBM Cloud®; therefore, you must set the credentialsMode parameter for the CCO to Manual when
installing OpenShift Container Platform and manage your cloud credentials manually.
Using manual mode allows each cluster component to have only the permissions it requires, without
storing an administrator-level credential in the cluster. You can also use this mode if your environment
does not have connectivity to the cloud provider public IAM endpoint. However, you must manually
reconcile permissions with new release images for every upgrade. You must also manually supply
credentials for every component that requests them.

NOTE
Prerequisites
Procedure

NOTE

1836
$ chmod 775 ccoctl
Verification
$ ccoctl --help
Usage:
ccoctl [command]
Available Commands:
Flags:
Rotating API keys for IBM Cloud®
10.3.3. Next steps

Installing a cluster on IBM Cloud® with customizations

10.4. USER-MANAGED ENCRYPTION FOR IBM CLOUD

By default, provider-managed encryption is used to secure the following when you deploy an OpenShift
Container Platform cluster:
The root (boot) volume of control plane and compute machines
Persistent volumes (data volumes) that are provisioned after the cluster is deployed
You can override the default behavior by specifying an IBM® Key Protect for IBM Cloud® (Key Protect)
root key as part of the installation process.
1837
When you bring our own root key, you modify the installation configuration file (install-config.yaml) to
specify the Cloud Resource Name (CRN) of the root key by using the encryptionKey parameter.
You can specify that:
The same root key be used be used for all cluster machines. You do so by specifying the key as
part of the cluster’s default machine configuration.
When specified as part of the default machine configuration, all managed storage classes are
updated with this key. As such, data volumes that are provisioned after the installation are also
encrypted using this key.
Separate root keys be used for the control plane and compute machine pools.
For more information about the encryptionKey parameter, see Additional IBM Cloud configuration
parameters.
NOTE
Make sure you have integrated Key Protect with your IBM Cloud Block Storage service.
For more information, see the Key Protect documentation.
10.4.1. Next steps

Installing a cluster on IBM Cloud with customizations
Installing a cluster on IBM Cloud with network customizations
Installing a cluster on IBM Cloud into an existing VPC
Installing a private cluster on IBM Cloud
10.5. INSTALLING A CLUSTER ON IBM CLOUD WITH

CUSTOMIZATIONS
the installation program provisions on IBM Cloud®. To customize the installation, you modify parameters
in the install-config.yaml file before you install the cluster.
processes.
users.
You configured an IBM Cloud® account to host the cluster.
You configured the ccoctl utility before you installed the cluster. For more information, see
Configuring IAM for IBM Cloud® .
1838

IMPORTANT

authentication.
user.
IMPORTANT
NOTE
Procedure
1839
NOTE
ecdsa algorithm.
NOTE

task:
Example output
Agent pid 31874
NOTE

1840
Example output
Next steps
program.

for installation.
Prerequisites
Procedure
IMPORTANT
IMPORTANT
components.
1841
10.5.5. Exporting the API key

You must set the API key you created as a global variable; the installation program ingests the variable
during startup to set the API key.
Prerequisites
You have created either a user API key or service ID API key for your IBM Cloud® account.
Procedure
Export your API key for your account as a global variable:
$ export IC_API_KEY=<api_key>
IMPORTANT
You must set the variable name exactly as specified; the installation program expects the
variable name to be present during startup.

You can customize the OpenShift Container Platform cluster you install on IBM Cloud®.
Prerequisites
cluster.
Procedure
command:
1842
NOTE

ii. Select ibmcloud as the platform to target.
v. Enter a descriptive name for your cluster.
IMPORTANT

Installation configuration parameters for IBM Cloud®
Machine Operating vCPU Virtual RAM Storage Input/Output

System Per Second
(IOPS)
Compute RHCOS 2 8 GB 100 GB 300
1843
Optimizing storage
10.5.6.2. Sample customized install-config.yaml file for IBM Cloud
IMPORTANT
config.yaml file by using the installation program and then modify it.
apiVersion: v1
controlPlane: 2 3
name: master
platform:
ibmcloud: {}
replicas: 3
compute: 5 6
name: worker
platform:
ibmcloud: {}
replicas: 3
metadata:
networking:
clusterNetwork:
- cidr: 10.128.0.0/14
hostPrefix: 23
machineNetwork:
- cidr: 10.0.0.0/16
serviceNetwork:
- 172.30.0.0/16
platform:
ibmcloud:
region: us-south 10
publish: External
fips: false 12
value.
The controlPlane section is a single mapping, but the compute section is a sequence of
1844
4 7 Enables or disables simultaneous multithreading, also known as Hyper-Threading. By default,

IMPORTANT

multithreading.
value.
12 Enables or disables FIPS mode. By default, FIPS mode is not enabled. If FIPS mode is enabled, the
Red Hat Enterprise Linux CoreOS (RHCOS) machines that OpenShift Container Platform runs on
bypass the default Kubernetes cryptography suite and use the cryptography modules that are
IMPORTANT
13 Optional: provide the sshKey value that you use to access the machines in your cluster.
NOTE
process uses.
Prerequisites
1845
NOTE
(169.254.169.254).
Procedure
apiVersion: v1
proxy:
must be http.

destinations.
1846
NOTE
NOTE
NOTE
created.
10.5.7. Manually creating IAM

Installing the cluster requires that the Cloud Credential Operator (CCO) operate in manual mode. While
the installation program configures the CCO for manual mode, you must specify the identity and access
management secrets for you cloud provider.
You can use the Cloud Credential Operator (CCO) utility (ccoctl) to create the required IBM Cloud®
resources.
Prerequisites
You have configured the ccoctl binary.
Procedure
1. Edit the install-config.yaml configuration file so that it contains the credentialsMode

parameter set to Manual.
Example install-config.yaml configuration file
apiVersion: v1
compute:
1 This line is added to set the credentialsMode parameter to Manual.
1847
2. To generate the manifests, run the following command from the directory that contains the
3. From the directory that contains the installation program, set a $RELEASE_IMAGE variable
with the release image from your installation file by running the following command:

--included \ 1
metadata:
labels:
name: openshift-image-registry-ibmcos
spec:
secretRef:
providerSpec:
kind: IBMCloudProviderSpec
policies:
- attributes:
- name: serviceName
value: cloud-object-storage
roles:
- crn:v1:bluemix:public:iam::::role:Viewer
- crn:v1:bluemix:public:iam::::role:Operator
- crn:v1:bluemix:public:iam::::role:Editor
1848
- crn:v1:bluemix:public:iam::::serviceRole:Reader
- crn:v1:bluemix:public:iam::::serviceRole:Writer
- attributes:
- name: resourceType
value: resource-group
roles:
5. Create the service ID for each credential request, assign the policies defined, create an API key,
and generate the secret:
$ ccoctl ibmcloud create-service-id \

--credentials-requests-dir=<path_to_credential_requests_directory> \ 1
--name=<cluster_name> \ 2
--output-dir=<installation_directory> \ 3
--resource-group-name=<resource_group_name> 4
2 Specify the name of the OpenShift Container Platform cluster.
4 Optional: Specify the name of the resource group used for scoping the access policies.
NOTE
preview parameter.
If an incorrect resource group name is provided, the installation fails during the
bootstrap phase. To find the correct resource group name, run the following
command:
$ grep resourceGroupName <installation_directory>/manifests/cluster-

infrastructure-02-config.yml
Verification
Ensure that the appropriate secrets were generated in your cluster’s manifests directory.

IMPORTANT
1849
Prerequisites
cluster.
Procedure
deployment:

--log-level=info 2

config.yaml file.
Verification
IMPORTANT
Example output
...
IMPORTANT
1850
IMPORTANT

IMPORTANT

Procedure
Portal.
$ tar xvf <file>

$ echo $PATH
Verification
$ oc <command>
1851

Procedure
Portal.

C:\> path
Verification
C:\> oc <command>

Procedure
Portal.
NOTE

$ echo $PATH
Verification
1852
$ oc <command>

Prerequisites
Procedure
$ oc whoami
Example output
system:admin

10.5.12. Next steps
1853
10.6. INSTALLING A CLUSTER ON IBM CLOUD WITH NETWORK

CUSTOMIZATIONS
configuration on infrastructure that the installation program provisions on IBM Cloud®. By customizing
your network configuration, your cluster can coexist with existing IP address allocations in your
environment and integrate with existing MTU and VXLAN configurations. To customize the installation,
you modify parameters in the install-config.yaml file before you install the cluster.
processes.
users.

IMPORTANT
1854

authentication.
user.
IMPORTANT
NOTE
Procedure
NOTE
ecdsa algorithm.
1855
NOTE

task:
Example output
Agent pid 31874
NOTE

Example output
Next steps
program.

for installation.
Prerequisites
Procedure
1856
IMPORTANT
IMPORTANT
components.

Prerequisites
Procedure
IMPORTANT
1857

Prerequisites
cluster.
Procedure
command:
NOTE

1858
IMPORTANT


System Per Second
(IOPS)
Optimizing storage
IMPORTANT
apiVersion: v1
controlPlane: 2 3
name: master
platform:
ibmcloud: {}
1859
replicas: 3
compute: 5 6
name: worker
platform:
ibmcloud: {}
replicas: 3
metadata:
networking: 9
clusterNetwork:
- cidr: 10.128.0.0/14
hostPrefix: 23
machineNetwork:
- cidr: 10.0.0.0/16
serviceNetwork:
- 172.30.0.0/16
platform:
ibmcloud:
region: us-south 11
publish: External
fips: false 13
default value.

IMPORTANT

multithreading.
value.
1860
IMPORTANT
NOTE
process uses.
Prerequisites
NOTE
(169.254.169.254).
Procedure
apiVersion: v1
1861
proxy:
must be http.

destinations.
NOTE
NOTE
NOTE
1862
NOTE
created.

resources.
Prerequisites
Procedure

apiVersion: v1
compute:

1863
--included \ 1
metadata:
labels:
spec:
secretRef:
providerSpec:
policies:
- attributes:
- name: serviceName
roles:
- attributes:
roles:

1864
NOTE
preview parameter.
command:

Verification

Phase 1
NOTE

NIC resides in.
IMPORTANT
1865
IMPORTANT
cluster.
Phase 2

the cluster.
IMPORTANT

Prerequisites
Procedure
kind: Network
metadata:
name: cluster
spec:
1866
kind: Network
metadata:
name: cluster
spec:
defaultNetwork:
ipsecConfig:
mode: Full


clusterNetwork
serviceNetwork
defaultNetwork.type
1867
spec:
clusterNetwork:
- cidr: 10.128.0.0/19
hostPrefix: 23
- cidr: 10.128.32.0/19
hostPrefix: 23
spec:
serviceNetwork:
- 172.30.0.0/14

manifest file.
work


NOTE

1868

network plugin.

detected MTU.


value to 1400.

configuration.

1869

NOTE


t network
infrastructure
overlaps with the
100.64.0.0/16
IPv4 subnet, you
can specify a
different IP
address range for
internal use by
OVN-Kubernetes.
You must ensure
that the IP address
range does not
overlap with any
other subnet used
by your OpenShift
Container
Platform
address range
must be larger
than the maximum
number of nodes
that can be added
to the cluster. For
example, if the
clusterNetwork.
cidr value is
10.128.0.0/14
and the
clusterNetwork.
hostPrefix value
is /23, then the
maximum number
of nodes is 2^(23-
14)=512 .
This field cannot

be changed after
installation.
1870

t network
infrastructure
overlaps with the
fd98::/48 IPv6
subnet, you can
specify a different
IP address range
for internal use by
OVN-Kubernetes.
You must ensure
that the IP address
range does not
overlap with any
other subnet used
by your OpenShift
Container
Platform
address range
must be larger
than the maximum
number of nodes
that can be added
to the cluster.
This field cannot

be changed after
installation.

50000000 or 50 MB.
1871
libc
host.
udp:<host>:<port>
unix:<file>
null

1872


external hosts.

defaultNetwork:
type: OVNKubernetes
mtu: 1400
genevePort: 6081
ipsecConfig:
mode: Full
IMPORTANT

documentation.
NOTE
1873

kubeProxyConfig:
proxyArguments:
- 0s

IMPORTANT
Prerequisites
cluster.
Procedure
deployment:

--log-level=info 2

config.yaml file.
Verification
1874
IMPORTANT
Example output
...
IMPORTANT

IMPORTANT

Procedure
1875
Portal.
$ tar xvf <file>

$ echo $PATH
Verification
$ oc <command>

Procedure
Portal.

C:\> path
Verification
C:\> oc <command>

Procedure
1876
Portal.
NOTE

$ echo $PATH
Verification
$ oc <command>

Prerequisites
Procedure
$ oc whoami
Example output
1877
system:admin

10.6.15. Next steps

10.7. INSTALLING A CLUSTER ON IBM CLOUD INTO AN EXISTING VPC

In OpenShift Container Platform version 4.15, you can install a cluster into an existing Virtual Private
Cloud (VPC) on IBM Cloud®. The installation program provisions the rest of the required infrastructure,
which you can then further customize. To customize the installation, you modify parameters in the
install-config.yaml file before you install the cluster.
processes.
users.

In OpenShift Container Platform 4.15, you can deploy a cluster into the subnets of an existing IBM®
Virtual Private Cloud (VPC). Deploying OpenShift Container Platform into an existing VPC can help you
avoid limit constraints in new accounts or more easily abide by the operational constraints that your
1878
company’s guidelines set. If you cannot obtain the infrastructure creation permissions that are required
to create the VPC yourself, use this installation option.
Because the installation program cannot know what other components are in your existing subnets, it
cannot choose subnet CIDRs and so forth. You must configure networking for the subnets to which you
will install the cluster.
You must correctly configure the existing VPC and its subnets before you install the cluster. The
installation program does not create the following components:
NAT gateways
Subnets
Route tables
VPC network
Subdivide network ranges for the cluster to use
Set route tables for the subnets
Set VPC options like DHCP
NOTE
The VPC and all of the subnets must be in an existing resource group. The cluster is deployed to the
existing VPC.
As part of the installation, specify the following in the install-config.yaml file:
The name of the existing resource group that contains the VPC and subnets
(networkResourceGroupName)
The name of the existing VPC (vpcName)
The subnets that were created for control plane machines and compute machines
(controlPlaneSubnets and computeSubnets)
NOTE
Additional installer-provisioned cluster resources are deployed to a separate resource

group (resourceGroupName). You can specify this resource group before installing the
cluster. If undefined, a new resource group is created for the cluster.
To ensure that the subnets that you provide are suitable, the installation program confirms the following:
1879
All of the subnets that you specify exist.
For each availability zone in the region, you specify:
One subnet for control plane machines.
One subnet for compute machines.
The machine CIDR that you specified contains the subnets for the compute machines and
control plane machines.
NOTE
Subnet IDs are not supported.
TCP port 22 ingress (SSH) is allowed to the entire network.

IMPORTANT
1880
authentication.
user.
IMPORTANT
NOTE
Procedure
NOTE
ecdsa algorithm.
1881
NOTE

task:
Example output
Agent pid 31874
NOTE

Example output
Next steps
program.

for installation.
Prerequisites
Procedure
1882
IMPORTANT
IMPORTANT
components.

Prerequisites
Procedure
IMPORTANT
1883

Prerequisites
cluster.
Procedure
command:
NOTE

1884
IMPORTANT


System Per Second
(IOPS)
Optimizing storage
IMPORTANT
apiVersion: v1
controlPlane: 2 3
name: master
platform:
ibmcloud: {}
1885
replicas: 3
compute: 5 6
name: worker
platform:
ibmcloud: {}
replicas: 3
metadata:
networking:
clusterNetwork:
- cidr: 10.128.0.0/14 9
hostPrefix: 23
machineNetwork:
- cidr: 10.0.0.0/16
serviceNetwork:
- 172.30.0.0/16
platform:
ibmcloud:
region: eu-gb 11
resourceGroupName: eu-gb-example-cluster-rg 12
networkResourceGroupName: eu-gb-example-existing-network-rg 13
vpcName: eu-gb-example-network-1 14
controlPlaneSubnets: 15
- eu-gb-example-network-1-cp-eu-gb-1
computeSubnets: 16
- eu-gb-example-network-1-compute-eu-gb-1
publish: External
fips: false 18
value.

IMPORTANT
1886
IMPORTANT

multithreading.
9 The machine CIDR must contain the subnets for the compute machines and control plane
machines.
value.
12 The name of an existing resource group. All installer-provisioned cluster resources are deployed to
this resource group. If undefined, a new resource group is created for the cluster.
13 Specify the name of the resource group that contains the existing virtual private cloud (VPC). The
existing VPC and subnets should be in this resource group. The cluster will be installed to this VPC.
15 Specify the name of the existing subnets to which to deploy the control plane machines. The
subnets must belong to the VPC that you specified. Specify a subnet for each availability zone in
the region.
16 Specify the name of the existing subnets to which to deploy the compute machines. The subnets
must belong to the VPC that you specified. Specify a subnet for each availability zone in the region.
IMPORTANT
architectures.
NOTE
process uses.
1887
Prerequisites
NOTE
(169.254.169.254).
Procedure
apiVersion: v1
proxy:
must be http.

destinations.
1888
NOTE
NOTE
NOTE
created.

resources.
Prerequisites
Procedure

1889
apiVersion: v1
compute:

--included \ 1
metadata:
labels:
spec:
secretRef:
1890
providerSpec:
policies:
- attributes:
- name: serviceName
roles:
- attributes:
roles:

NOTE
preview parameter.
command:

Verification
1891

IMPORTANT
Prerequisites
cluster.
Procedure
deployment:

--log-level=info 2

config.yaml file.
Verification
IMPORTANT
Example output
...
1892
IMPORTANT

IMPORTANT

Procedure
Portal.
$ tar xvf <file>

$ echo $PATH
1893
Verification
$ oc <command>

Procedure
Portal.

C:\> path
Verification
C:\> oc <command>

Procedure
Portal.
NOTE

$ echo $PATH
1894
Verification
$ oc <command>

Prerequisites
Procedure
$ oc whoami
Example output
system:admin

1895
10.7.13. Next steps

Optional: Opt out of remote health reporting .
10.8. INSTALLING A PRIVATE CLUSTER ON IBM CLOUD

In OpenShift Container Platform version 4.15, you can install a private cluster into an existing VPC. The
installation program provisions the rest of the required infrastructure, which you can further customize.
To customize the installation, you modify parameters in the install-config.yaml file before you install
the cluster.
processes.
users.

internet.
IMPORTANT
Create a DNS zone using IBM Cloud® DNS Services and specify it as the base domain of the
1896
Create a DNS zone using IBM Cloud® DNS Services and specify it as the base domain of the
cluster. For more information, see "Using IBM Cloud® DNS Services to configure DNS
resolution".
10.8.3. Private clusters in IBM Cloud

To create a private cluster on IBM Cloud®, you must provide an existing private VPC and subnets to host
the cluster. The installation program must also be able to resolve the DNS records that the cluster
traffic.
The cluster still requires access to internet to access the IBM Cloud® APIs.
Public subnets
Public network load balancers, which support public ingress
A public DNS zone that matches the baseDomain for the cluster
The installation program does use the baseDomain that you specify to create a private DNS zone and
the required records for the cluster. The cluster is configured so that the Operators do not create public
records for the cluster and all cluster machines are placed in the private subnets that you specify.
10.8.3.1. Limitations
Private clusters on IBM Cloud® are subject only to the limitations associated with the existing VPC that
was used for cluster deployment.

1897
NAT gateways
Subnets
Route tables
VPC network
NOTE
existing VPC.
NOTE

1898
NOTE

IMPORTANT

authentication.
1899
user.
IMPORTANT
NOTE
Procedure
NOTE
ecdsa algorithm.
NOTE
1900
NOTE

task:
Example output
Agent pid 31874
NOTE

Example output
Next steps
program.

Before you install OpenShift Container Platform, download the installation file on a bastion host on your
cloud network or a machine that has access to the to the network through a VPN.
For more information about private cluster installation requirements, see "Private clusters".
Prerequisites
You have a machine that runs Linux, for example Red Hat Enterprise Linux 8, with 500 MB of
local disk space.
Procedure
1901
IMPORTANT
IMPORTANT
components.

Prerequisites
Procedure
IMPORTANT

1902
Prerequisites
recovery.
for your cluster.
Procedure
IMPORTANT
NOTE
IMPORTANT
1903

System Per Second
(IOPS)
Optimizing storage
IMPORTANT
apiVersion: v1
controlPlane: 2 3
name: master
platform:
ibmcloud: {}
replicas: 3
compute: 5 6
name: worker
platform:
ibmcloud: {}
replicas: 3
metadata:
networking:
clusterNetwork:
- cidr: 10.128.0.0/14 9
hostPrefix: 23
machineNetwork:
- cidr: 10.0.0.0/16 10
serviceNetwork:
1904
- 172.30.0.0/16
platform:
ibmcloud:
region: eu-gb 12
resourceGroupName: eu-gb-example-cluster-rg 13
networkResourceGroupName: eu-gb-example-existing-network-rg 14
vpcName: eu-gb-example-network-1 15
computeSubnets: 17
fips: false 20
1 8 12 19 Required.
value.

IMPORTANT

multithreading.
machines.
10 The CIDR must contain the subnets defined in platform.ibmcloud.controlPlaneSubnets and

platform.ibmcloud.computeSubnets.
value.
1905
the region.
private cluster. The default value is External.
IMPORTANT
architectures.
NOTE
process uses.
Prerequisites
NOTE
1906
NOTE
(169.254.169.254).
Procedure
apiVersion: v1
proxy:
must be http.

destinations.
NOTE
1907
NOTE
NOTE
created.

resources.
Prerequisites
Procedure

apiVersion: v1
compute:
1908

--included \ 1
metadata:
labels:
spec:
secretRef:
providerSpec:
policies:
- attributes:
- name: serviceName
roles:
1909
- attributes:
roles:

NOTE
preview parameter.
command:

Verification

IMPORTANT
Prerequisites
1910
cluster.
Procedure
deployment:

--log-level=info 2

config.yaml file.
Verification
IMPORTANT
Example output
...
IMPORTANT
1911
IMPORTANT

IMPORTANT

Procedure
Portal.
$ tar xvf <file>

$ echo $PATH
Verification
$ oc <command>
1912

Procedure
Portal.

C:\> path
Verification
C:\> oc <command>

Procedure
Portal.
NOTE

$ echo $PATH
Verification
1913
$ oc <command>

Prerequisites
Procedure
$ oc whoami
Example output
system:admin

10.8.15. Next steps
1914
10.9. INSTALLING A CLUSTER ON IBM CLOUD IN A RESTRICTED

NETWORK
In OpenShift Container Platform 4.15, you can install a cluster in a restricted network by creating an
internal mirror of the installation release content that is accessible to an existing Virtual Private Cloud
(VPC) on IBM Cloud®.
processes.
You configured an IBM Cloud account to host the cluster.
You have a container image registry that is accessible to the internet and your restricted
network. The container image registry should mirror the contents of the OpenShift image
registry and contain the installation media. For more information, see Mirroring images for a
disconnected installation using the oc-mirror plugin.
You have an existing VPC on IBM Cloud® that meets the following requirements:
The VPC contains the mirror registry or has firewall rules or a peering connection to access
the mirror registry that is hosted elsewhere.
The VPC can access IBM Cloud® service endpoints using a public endpoint. If network
restrictions limit access to public service endpoints, evaluate those services for alternate
endpoints that might be available. For more information see Access to IBM service
endpoints.
You cannot use the VPC that the installation program provisions by default.
If you plan on configuring endpoint gateways to use IBM Cloud® Virtual Private Endpoints,
consider the following requirements:
Endpoint gateway support is currently limited to the us-east and us-south regions.
The VPC must allow traffic to and from the endpoint gateways. You can use the VPC’s
default security group, or a new security group, to allow traffic on port 443. For more
information, see Allowing endpoint gateway traffic.
Configuring IAM for IBM Cloud VPC.

1915
10.9.2.1. Required internet access and an installation host
You complete the installation using a bastion host or portable device that can access both the internet
and your closed network. You must use a host with internet access to:
Download the installation program, the OpenShift CLI (oc), and the CCO utility ( ccoctl).
Use the installation program to locate the Red Hat Enterprise Linux CoreOS (RHCOS) image
and create the installation configuration file.
Use oc to extract ccoctl from the CCO container image.
Use oc and ccoctl to configure IAM for IBM Cloud®.
10.9.2.2. Access to a mirror registry
OpenShift image registry and contains the installation media.
You can create this registry on a mirror host, which can access both the internet and your restricted
network, or by using other methods that meet your organization’s security restrictions.
For more information on mirroring images for a disconnected installation, see "Additional resources".
10.9.2.3. Access to IBM service endpoints
The installation program requires access to the following IBM Cloud® service endpoints:
Cloud Object Storage
DNS Services
Global Search
Global Tagging
Identity Services
Resource Controller
Resource Manager
VPC
NOTE
If you are specifying an IBM® Key Protect for IBM Cloud® root key as part of the
installation process, the service endpoint for Key Protect is also required.
By default, the public endpoint is used to access the service. If network restrictions limit access to public
service endpoints, you can override the default behavior.
Before deploying the cluster, you can update the installation configuration file (install-config.yaml) to
specify the URI of an alternate service endpoint. For more information on usage, see "Additional
resources".
1916
Mirroring images for a disconnected installation using the oc-mirror plugin
Additional IBM Cloud configuration parameters

NAT gateways
Subnets
Route tables
VPC network
NOTE
1917
existing VPC.
NOTE

NOTE
10.9.3.4. Allowing endpoint gateway traffic
If you are using IBM Cloud® Virtual Private endpoints, your Virtual Private Cloud (VPC) must be
configured to allow traffic to and from the endpoint gateways.
1918
A VPC’s default security group is configured to allow all outbound traffic to endpoint gateways.
Therefore, the simplest way to allow traffic between your VPC and endpoint gateways is to modify the
default security group to allow inbound traffic on port 443.
NOTE
If you choose to configure a new security group, the security group must be configured to
allow both inbound and outbound traffic.
Prerequisites
You have installed the IBM Cloud® Command Line Interface utility (ibmcloud).
Procedure
1. Obtain the identifier for the default security group by running the following command:
$ DEFAULT_SG=$(ibmcloud is vpc <your_vpc_name> --output JSON | jq -r

'.default_security_group.id')
2. Add a rule that allows inbound traffic on port 443 by running the following command:
$ ibmcloud is security-group-rule-add $DEFAULT_SG inbound tcp --remote 0.0.0.0/0 --port-

min 443 --port-max 443
NOTE
Be sure that your endpoint gateways are configured to use this security group.

authentication.
user.
IMPORTANT
NOTE
1919
NOTE
Procedure
NOTE
ecdsa algorithm.
NOTE

task:
Example output
Agent pid 31874
NOTE
1920
NOTE

Example output
Next steps
program.

Prerequisites
Procedure
IMPORTANT
10.9.6. Downloading the RHCOS cluster image

The installation program requires the Red Hat Enterprise Linux CoreOS (RHCOS) image to install the
cluster. While optional, downloading the Red Hat Enterprise Linux CoreOS (RHCOS) before deploying
removes the need for internet access when creating the cluster.
Use the installation program to locate and download the Red Hat Enterprise Linux CoreOS (RHCOS)
image.
Prerequisites
1921
The host running the installation program has internet access.
Procedure
$ ./openshift-install coreos print-stream-json
2. Use the output of the command to find the location of the IBM Cloud® image.
.Example output
----
"release": "415.92.202311241643-0",
"formats": {
"qcow2.gz": {
"disk": {
"location": "https://rhcos.mirror.openshift.com/art/storage/prod/streams/4.15-
9.2/builds/415.92.202311241643-0/x86_64/rhcos-415.92.202311241643-0-
ibmcloud.x86_64.qcow2.gz",
"sha256":
"6b562dee8431bec3b93adeac1cfefcd5e812d41e3b7d78d3e28319870ffc9eae",
"uncompressed-sha256":
"5a0f9479505e525a30367b6a6a6547c86a8f03136f453c1da035f3aa5daa8bc9"
----
3. Download and extract the image archive. Make the image available on the host that the
installation program uses to create the cluster.

Prerequisites
for your cluster.
You have the imageContentSourcePolicy.yaml file that was created when you mirrored your
registry.
Procedure
IMPORTANT
1922
IMPORTANT
NOTE
When customizing the sample template, be sure to provide the information that is required for
an installation in a restricted network:

platform.ibmcloud field:
vpcName: <existing_vpc>
controlPlaneSubnets: <control_plane_subnet>
computeSubnets: <compute_subnet>
For platform.ibmcloud.vpcName, specify the name for the existing IBM Cloud VPC. For
platform.ibmcloud.controlPlaneSubnets and platform.ibmcloud.computeSubnets,
specify the existing subnets to deploy the control plane machines and compute machines,
respectively.
1923
- mirrors:
- mirrors:
For these values, use the imageContentSourcePolicy.yaml file that was created when you
mirrored the registry.
e. If network restrictions limit the use of public endpoints to access the required IBM Cloud®
services, add the serviceEndpoints stanza to platform.ibmcloud to specify an alternate
service endpoint.
NOTE
You can specify only one alternate service endpoint for each service.
Example of using alternate services endpoints
# ...
serviceEndpoints:
- name: IAM
url: <iam_alternate_endpoint_url>
- name: VPC
url: <vpc_alternate_endpoint_url>
- name: ResourceController
url: <resource_controller_alternate_endpoint_url>
- name: ResourceManager
url: <resource_manager_alternate_endpoint_url>
- name: DNSServices
url: <dns_services_alternate_endpoint_url>
- name: COS
url: <cos_alternate_endpoint_url>
- name: GlobalSearch
url: <global_search_alternate_endpoint_url>
- name: GlobalTagging
url: <global_tagging_alternate_endpoint_url>
# ...
f. Optional: Set the publishing strategy to Internal:
publish: Internal
NOTE
If you use the default value of External, your network must be able to access
the public endpoint for IBM Cloud® Internet Services (CIS). CIS is not
enabled for Virtual Private Endpoints.
1924
IMPORTANT

System Per Second
(IOPS)
IMPORTANT
apiVersion: v1
controlPlane: 2 3
name: master
platform:
ibm-cloud: {}
replicas: 3
compute: 5 6
name: worker
1925
platform:
ibmcloud: {}
replicas: 3
metadata:
networking:
clusterNetwork:
- cidr: 10.128.0.0/14 9
hostPrefix: 23
machineNetwork:
- cidr: 10.0.0.0/16 10
serviceNetwork:
- 172.30.0.0/16
platform:
ibmcloud:
region: us-east 12
resourceGroupName: us-east-example-cluster-rg 13
- name: IAM
url: https://private.us-east.iam.cloud.ibm.com
- name: VPC
url: https://us-east.private.iaas.cloud.ibm.com/v1
- name: ResourceController
url: https://private.us-east.resource-controller.cloud.ibm.com
- name: ResourceManager
url: https://private.us-east.resource-controller.cloud.ibm.com
- name: DNSServices
url: https://api.private.dns-svcs.cloud.ibm.com/v1
- name: COS
url: https://s3.direct.us-east.cloud-object-storage.appdomain.cloud
- name: GlobalSearch
url: https://api.private.global-search-tagging.cloud.ibm.com
- name: GlobalTagging
url: https://tags.private.global-search-tagging.cloud.ibm.com
networkResourceGroupName: us-east-example-existing-network-rg 15
vpcName: us-east-example-network-1 16
- us-east-example-network-1-cp-us-east-1
computeSubnets: 18
- us-east-example-network-1-compute-us-east-1
fips: false 20
1926
- mirrors:
- mirrors:
1 8 12 Required.
value.

IMPORTANT

multithreading.
machines.
10 The CIDR must contain the subnets defined in platform.ibmcloud.controlPlaneSubnets and

platform.ibmcloud.computeSubnets.
value.
14 Based on the network restrictions of the VPC, specify alternate service endpoints as needed. This
overrides the default public endpoint for the service.
the region.
1927
registry uses to serve content. For example, registry.example.com or registry.example.com:5000.
IMPORTANT
The use of FIPS Validated or Modules in Process cryptographic libraries is only

supported on OpenShift Container Platform deployments on the x86_64
architecture.
23 Provide these values from the metadata.name: release-0 section of the

imageContentSourcePolicy.yaml file that was created when you mirrored the registry.
NOTE
process uses.
Prerequisites
NOTE
(169.254.169.254).
1928
Procedure
apiVersion: v1
proxy:
must be http.

destinations.
NOTE
NOTE
1929
NOTE
created.

IMPORTANT

Procedure
Portal.
$ tar xvf <file>

$ echo $PATH
Verification
$ oc <command>

Procedure
1930
Portal.

C:\> path
Verification
C:\> oc <command>

Procedure
Portal.
NOTE

$ echo $PATH
Verification
$ oc <command>
1931
resources.
Prerequisites
Procedure

apiVersion: v1
compute:

--included \ 1
1932
metadata:
labels:
spec:
secretRef:
providerSpec:
policies:
- attributes:
- name: serviceName
roles:
- attributes:
roles:

1933
NOTE
preview parameter.
command:

Verification

IMPORTANT
Prerequisites
cluster.
If the Red Hat Enterprise Linux CoreOS (RHCOS) image is available locally, the host running
the installation program does not require internet access.
Procedure
1. Export the OPENSHIFT_INSTALL_OS_IMAGE_OVERRIDE variable to specify the location of

the Red Hat Enterprise Linux CoreOS (RHCOS) image by running the following command:
$ export OPENSHIFT_INSTALL_OS_IMAGE_OVERRIDE="<path_to_image>/rhcos-
<image_version>-ibmcloud.x86_64.qcow2.gz"
deployment:

--log-level=info 2
1934

config.yaml file.
Verification
IMPORTANT
Example output
...
IMPORTANT

Prerequisites
1935
Prerequisites
Procedure
$ oc whoami
Example output
system:admin
10.9.12. Post installation

Complete the following steps to complete the configuration of your cluster.
Procedure
OperatorHub object:

TIP
10.9.12.2. Installing the policy resources into the cluster
1936
Mirroring the OpenShift Container Platform content using the oc-mirror OpenShift CLI (oc) plugin
creates resources, which include catalogSource-certified-operator-index.yaml and
imageContentSourcePolicy.yaml.
The ImageContentSourcePolicy resource associates the mirror registry with the source
registry and redirects image pull requests from the online registries to the mirror registry.
The CatalogSource resource is used by Operator Lifecycle Manager (OLM) to retrieve

information about the available Operators in the mirror registry, which lets users discover and
install Operators.
After you install the cluster, you must install these resources into the cluster.
Prerequisites
Procedure
2. Apply the YAML files from the results directory to the cluster:
$ oc apply -f ./oc-mirror-workspace/results-<id>/
Verification
1. Verify that the ImageContentSourcePolicy resources were successfully installed:
2. Verify that the CatalogSource resources were successfully installed:
$ oc get catalogsource --all-namespaces

10.9.14. Next steps

1937
Optional: Opt out of remote health reporting .
10.10. INSTALLATION CONFIGURATION PARAMETERS FOR IBM

CLOUD
Before you deploy an OpenShift Container Platform cluster on IBM Cloud®, you provide parameters to
10.10.1. Available installation configuration parameters for IBM Cloud

The following tables specify the required, optional, and IBM Cloud-specific installation configuration
NOTE

versions.

combination of the
baseDomain and
<metadata.name>.

consumed.
1938
name: subdomains of
{{.metadata.name}}.
{{.baseDomain}}.

alibabacloud, aws,
ibmcloud, nutanix,

"auth":"b3Blb=",
}
}
}
NOTE
1939

NOTE
You cannot modify

by the networking
object after
installation.


networking:
networking:
- cidr: 10.128.0.0/14
hostPrefix: 23

An IPv4 network.

23 provides 510 (2^(32 - 23) - 2) pod
IP addresses.
serviceNetwork:
networking:
serviceNetwork:
- 172.30.0.0/16
1940

networking:
networking:
machineNetwork:
- cidr: 10.0.0.0/16

NOTE
192.168.0.0/24. If you are deploying eNetwork to match
the cluster to an existing Virtual the CIDR that the
Private Cloud (VPC), the CIDR must
in.
contain the subnets defined in
platform.ibmcloud.controlPlaneS
ubnets and
platform.ibmcloud.computeSubn
ets.


1941

et:


section.


(the default).
1942

cores.
IMPORTANT
If you disable
simultaneous
that your capacity
the dramatically
decreased machine
performance.

name:
value.


1943

(the default).

cores.
IMPORTANT
If you disable
simultaneous
that your capacity
the dramatically
decreased machine
performance.

name:

parameter value.

replicas:

1944
fips: Enable or disable FIPS mode. The false or true

Parameter Description
default is false (disabled). If FIPS Values
IMPORTANT
To enable FIPS mode

must run the
from a Red Hat
Enterprise Linux
(RHEL) computer
in FIPS mode. For
more information
about configuring
FIPS mode on RHEL,
see Installing the
Enterprise Linux
(RHEL) or Red Hat
Enterprise Linux
CoreOS (RHCOS)
OpenShift Container
Platform core
components use the
RHEL cryptographic
libraries that have
been submitted to
NIST for FIPS 140-
only the x86_64,
ppc64le, and s390x
architectures.
NOTE

File storage, you
cannot enable FIPS
mode.
1945



ces:
mirrors:

value is External.

NOTE
For production
OpenShift Container
which you want to
SSH key that your
ssh-agent process
uses.
1946
10.10.1.4. Additional IBM Cloud configuration parameters
Additional IBM Cloud® configuration parameters are described in the following table:
Table 10.18. Additional IBM Cloud(R) parameters

eter
An IBM® Key Protect for IBM Cloud® (Key Protect) root key that The Cloud Resource Name
con should be used to encrypt the root (boot) volume of only control (CRN) of the root key.
trol plane machines.
Pla The CRN must be enclosed in
ne: quotes ("").
plat
for
m:
ibm
clo
ud:
boo
tVol
um
e:
enc
rypt
ion
Key
:
1947

eter
A Key Protect root key that should be used to encrypt the root The CRN of the root key.
co (boot) volume of only compute machines.
mp The CRN must be enclosed in
ute: quotes ("").
plat
for
m:
ibm
clo
ud:
boo
tVol
um
e:
enc
rypt
ion
Key
:
1948

eter
A Key Protect root key that should be used to encrypt the root The CRN of the root key.
plat (boot) volume of all of the cluster’s machines.
for The CRN must be enclosed in
m: When specified as part of the default machine configuration, all quotes ("").
managed storage classes are updated with this key. As such,
ibm data volumes that are provisioned after the installation are also
clo encrypted using this key.
ud:
def
ault
Ma
chi
ne
Plat
for
m:
boo
tvol
um
e:
enc
rypt
ion
Key
:
The name of an existing resource group. By default, an installer- String, for example
plat provisioned VPC and cluster resources are placed in this existing_resource_group.
for resource group. When not specified, the installation program
m: creates the resource group for the cluster. If you are deploying
the cluster into an existing VPC, the installer-provisioned cluster
ibm resources are placed in this resource group. When not specified,
clo the installation program creates the resource group for the
ud: cluster. The VPC resources that you have provisioned must exist
in a resource group that you specify using the
res networkResourceGroupName parameter. In either case, this
our resource group must only be used for a single cluster
ce installation, as the cluster components assume ownership of all
Gro of the resources in the resource group. [1]
up
Na
me:
1949

eter
A list of service endpoint names and URIs. A valid service endpoint name
plat and fully qualified URI.
for By default, the installation program and cluster components use
m: public service endpoints to access the required IBM Cloud® Valid names include:
services.
COS
ibm
If network restrictions limit access to public service endpoints,
clo DNSServices
you can specify an alternate service endpoint to override the
ud:
default behavior.
GlobalServices
ser
You can specify only one alternate service endpoint for each of GlobalTagging
vice
the following services:
En IAM
dpo Cloud Object Storage
ints KeyProtect
: DNS Services
ResourceControll
- Global Search er
na Global Tagging ResourceManager
me:
Identity Services VPC
url:
Key Protect
Resource Controller
Resource Manager
VPC
The name of an existing resource group. This resource contains String, for example
plat the existing VPC and subnets to which the cluster will be existing_network_resour
for deployed. This parameter is required when deploying the cluster ce_group.
m: to a VPC that you have provisioned.
ibm
clo
ud:
net
wor
kR
eso
urc
eGr
oup
Na
me:
1950

eter
The new dedicated host to create. If you specify a value for Valid IBM Cloud® dedicated
plat platform.ibmcloud.dedicatedHosts.name, this parameter host profile, such as cx2-
for is not required. host-152x304 . [2]
m:
ibm
clo
ud:
ded
icat
ed
Ho
sts:
pro
file:
An existing dedicated host. If you specify a value for String, for example my-
plat platform.ibmcloud.dedicatedHosts.profile, this parameter dedicated-host-name .
for is not required.
m:
ibm
clo
ud:
ded
icat
ed
Ho
sts:
na
me:
The instance type for all IBM Cloud® machines. Valid IBM Cloud® instance
plat type, such as bx2-8x32. [2]
for
m:
ibm
clo
ud:
typ
e:
1951

eter
The name of the existing VPC that you want to deploy your String.
plat cluster to.
for
m:
ibm
clo
ud:
vpc
Na
me:
The name(s) of the existing subnet(s) in your VPC that you want String array
plat to deploy your control plane machines to. Specify a subnet for
for each availability zone.
m:
ibm
clo
ud:
con
trol
Pla
ne
Su
bne
ts:
The name(s) of the existing subnet(s) in your VPC that you want String array
plat to deploy your compute machines to. Specify a subnet for each
for availability zone. Subnet IDs are not supported.
m:
ibm
clo
ud:
co
mp
ute
Su
bne
ts:
1. Whether you define an existing resource group, or if the installer creates one, determines how
the resource group is treated when the cluster is uninstalled. If you define a resource group, the
installer removes all of the installer-provisioned resources, but leaves the resource group alone;
1952
if a resource group is created as part of the installation, the installer removes all of the installer-
provisioned resources and the resource group.
2. To determine which profile best meets your needs, see Instance Profiles in the IBM®
documentation.
10.11. UNINSTALLING A CLUSTER ON IBM CLOUD

You can remove a cluster that you deployed to IBM Cloud®.

NOTE
Prerequisites
You have installed the IBM Cloud® CLI and installed or updated the VPC infrastructure service
plugin. For more information see "Prerequisites" in the IBM Cloud® CLI documentation .
Procedure
1. If the following conditions are met, this step is required:
The installer created a resource group as part of the installation process.
You or one of your applications created persistent volume claims (PVCs) after the cluster
was deployed.
In which case, the PVCs are not removed when uninstalling the cluster, which might prevent the
resource group from being successfully removed. To prevent a failure:
a. Log in to the IBM Cloud® using the CLI.
b. To list the PVCs, run the following command:
$ ibmcloud is volumes --resource-group-name <infrastructure_id>
For more information about listing volumes, see the IBM Cloud® CLI documentation .
c. To delete the PVCs, run the following command:
$ ibmcloud is volume-delete --force <volume_id>
1953
For more information about deleting volumes, see the IBM Cloud® CLI documentation .
2. Export the API key that was created as part of the installation process.
NOTE
You must set the variable name exactly as specified. The installation program
expects the variable name to be present to remove the service IDs that were
created when the cluster was installed.

NOTE
4. Remove the manual CCO credentials that were created for the cluster:
$ ccoctl ibmcloud delete-service-id \

--credentials-requests-dir <path_to_credential_requests_directory> \
--name <cluster_name>
NOTE
preview parameter.
1954
CHAPTER 11. INSTALLING ON NUTANIX
11.1. PREPARING TO INSTALL ON NUTANIX

Before you install an OpenShift Container Platform cluster, be sure that your Nutanix environment
meets the following requirements.
11.1.1. Nutanix version requirements

You must install the OpenShift Container Platform cluster to a Nutanix environment that meets the
following requirements.
Table 11.1. Version requirements for Nutanix virtual environments
Component Required version
Nutanix AOS 6.5.2.7 or later
Prism Central pc.2022.6 or later
11.1.2. Environment requirements

Before you install an OpenShift Container Platform cluster, review the following Nutanix AOS
environment requirements.
11.1.2.1. Required account privileges
The installation program requires access to a Nutanix account with the necessary permissions to deploy
the cluster and to maintain the daily operation of it. The following options are available to you:
You can use a local Prism Central user account with administrative privileges. Using a local
account is the quickest way to grant access to an account with the required permissions.
If your organization’s security policies require that you use a more restrictive set of permissions,
use the permissions that are listed in the following table to create a custom Cloud Native role in
Prism Central. You can then assign the role to a user account that is a member of a Prism
Central authentication directory.
Consider the following when managing this user account:
When assigning entities to the role, ensure that the user can access only the Prism Element and
subnet that are required to deploy the virtual machines.
Ensure that the user is a member of the project to which it needs to assign virtual machines.
For more information, see the Nutanix documentation about creating a Custom Cloud Native role ,
assigning a role , and adding a user to a project .
Example 11.1. Required permissions for creating a Custom Cloud Native role
1955
Nutanix Object When required Required permissions Description

in Nutanix API
Categories Always Create_Category_M Create, read, and

apping delete categories that
Create_Or_Update_ are assigned to the
Name_Category OpenShift Container
Create_Or_Update_ Platform machines.
Value_Category
Delete_Category_M
apping
Delete_Name_Categ
ory
Delete_Value_Categ
ory
View_Category_Ma
pping
View_Name_Catego
ry
View_Value_Catego
ry
Images Always Create_Image Create, read, and

Delete_Image delete the operating
View_Image system images used for
the OpenShift
Container Platform
machines.
Virtual Machines Always Create_Virtual_Mac Create, read, and

hine delete the OpenShift
Delete_Virtual_Mac Container Platform
hine machines.
View_Virtual_Machi
ne
Clusters Always View_Cluster View the Prism Element

clusters that host the
OpenShift Container
Platform machines.
Subnets Always View_Subnet View the subnets that

host the OpenShift
Container Platform
machines.
1956
Nutanix Object When required Required permissions Description

in Nutanix API
Projects If you will associate a View_Project View the projects

project with compute defined in Prism
machines, control plane Central and allow a
machines, or all project to be assigned
machines. to the OpenShift
Container Platform
machines.
11.1.2.2. Cluster limits
Available resources vary between clusters. The number of possible clusters within a Nutanix
environment is limited primarily by available storage space and any limitations associated with the
resources that the cluster creates, and resources that you require to deploy the cluster, such a IP
addresses and networks.
11.1.2.3. Cluster resources
A minimum of 800 GB of storage is required to use a standard cluster.
When you deploy a OpenShift Container Platform cluster that uses installer-provisioned infrastructure,
the installation program must be able to create several resources in your Nutanix instance. Although
these resources use 856 GB of storage, the bootstrap node is destroyed as part of the installation
process.
A standard OpenShift Container Platform installation creates the following resources:
1 label
Virtual machines:
1 disk image
1 temporary bootstrap node
3 control plane nodes
3 compute machines
11.1.2.4. Networking requirements
You must use either AHV IP Address Management (IPAM) or Dynamic Host Configuration Protocol
(DHCP) for the network and ensure that it is configured to provide persistent IP addresses to the
cluster machines. Additionally, create the following networking resources before you install the
IP addresses
DNS records
NOTE
1957
NOTE
It is recommended that each OpenShift Container Platform node in the cluster have
access to a Network Time Protocol (NTP) server that is discoverable via DHCP.
Installation is possible without an NTP server. However, an NTP server prevents errors
typically associated with asynchronous server clocks.
11.1.2.4.1. Required IP Addresses
An installer-provisioned installation requires two static virtual IP (VIP) addresses:
A VIP address for the API is required. This address is used to access the cluster API.
A VIP address for ingress is required. This address is used for cluster ingress traffic.
You specify these IP addresses when you install the OpenShift Container Platform cluster.
11.1.2.4.2. DNS records
You must create DNS records for two static IP addresses in the appropriate DNS server for the Nutanix
instance that hosts your OpenShift Container Platform cluster. In each record, <cluster_name> is the
cluster name and <base_domain> is the cluster base domain that you specify when you install the
cluster.
A complete DNS record takes the form: <component>.<cluster_name>.<base_domain>..
Table 11.2. Required DNS records
Compo Record Description

nent
API VIP api.<cluster_name>.<base_domain>. This DNS A/AAAA or CNAME

record must point to the load
balancer for the control plane
machines. This record must be
resolvable by both clients
external to the cluster and from
all the nodes within the cluster.
Ingress *.apps.<cluster_name>.<base_domain>. A wildcard DNS A/AAAA or

VIP CNAME record that points to the
load balancer that targets the
machines that run the Ingress
router pods, which are the worker
nodes by default. This record
must be resolvable by both
clients external to the cluster and
from all the nodes within the
cluster.
1958
resource definitions (CRDs). To install a cluster on Nutanix, you must set the CCO to manual mode as
part of the installation process.
NOTE
Prerequisites
Procedure

NOTE

$ chmod 775 ccoctl
Verification
$ ccoctl --help
1959
Usage:
ccoctl [command]
Available Commands:
Flags:
11.2. FAULT TOLERANT DEPLOYMENTS USING MULTIPLE PRISM

ELEMENTS
By default, the installation program installs control plane and compute machines into a single Nutanix
Prism Element (cluster). To improve the fault tolerance of your OpenShift Container Platform cluster,
you can specify that these machines be distributed across multiple Nutanix clusters by configuring
failure domains.
A failure domain represents an additional Prism Element instance that is available to OpenShift
Container Platform machine pools during and after installation.
11.2.1. Failure domain requirements

When planning to use failure domains, consider the following requirements:
All Nutanix Prism Element instances must be managed by the same instance of Prism Central. A
deployment that is comprised of multiple Prism Central instances is not supported.
The machines that make up the Prism Element clusters must reside on the same Ethernet
network for failure domains to be able to communicate with each other.
A subnet is required in each Prism Element that will be used as a failure domain in the OpenShift
Container Platform cluster. When defining these subnets, they must share the same IP address
prefix (CIDR) and should contain the virtual IP addresses that the OpenShift Container
Platform cluster uses.
11.2.2. Installation method and failure domain configuration

The OpenShift Container Platform installation method determines how and when you configure failure
domains:
If you deploy using installer-provisioned infrastructure, you can configure failure domains in the
1960
If you deploy using installer-provisioned infrastructure, you can configure failure domains in the
installation configuration file before deploying the cluster. For more information, see
Configuring failure domains.
You can also configure failure domains after the cluster is deployed. For more information
about configuring failure domains post-installation, see Adding failure domains to an existing
Nutanix cluster.
If you deploy using infrastructure that you manage (user-provisioned infrastructure) no

additional configuration is required. After the cluster is deployed, you can manually distribute
control plane and compute machines across failure domains.
11.3. INSTALLING A CLUSTER ON NUTANIX

In OpenShift Container Platform version 4.15, you can choose one of the following options to install a
cluster on your Nutanix instance:
Using installer-provisioned infrastructure: Use the procedures in the following sections to use
installer-provisioned infrastructure. Installer-provisioned infrastructure is ideal for installing in
connected or disconnected network environments. The installer-provisioned infrastructure includes an
installation program that provisions the underlying infrastructure for the cluster.
Using the Assisted Installer: The Assisted Installer hosted at console.redhat.com. The Assisted Installer
cannot be used in disconnected environments. The Assisted Installer does not provision the underlying
infrastructure for the cluster, so you must provision the infrastructure before the running the Assisted
Installer. Installing with the Assisted Installer also provides integration with Nutanix, enabling autoscaling.
See Installing an on-premise cluster using the Assisted Installer for additional details.
Using user-provisioned infrastructure: Complete the relevant steps outlined in the Installing a cluster
on any platform documentation.
You have reviewed details about the OpenShift Container Platform installation and update
processes.
The installation program requires access to port 9440 on Prism Central and Prism Element. You
verified that port 9440 is accessible.
If you use a firewall, you have met these prerequisites:
You confirmed that port 9440 is accessible. Control plane nodes must be able to reach
Prism Central and Prism Element on port 9440 for the installation to succeed.
You configured the firewall to grant access to the sites that OpenShift Container Platform
requires. This includes the use of Telemetry.
If your Nutanix environment is using the default self-signed SSL certificate, replace it with a
certificate that is signed by a CA. The installation program requires a valid CA-signed certificate
to access to the Prism Central API. For more information about replacing the self-signed
certificate, see the Nutanix AOS Security Guide .
If your Nutanix environment uses an internal CA to issue certificates, you must configure a
cluster-wide proxy as part of the installation process. For more information, see Configuring a
custom PKI.
IMPORTANT
1961
IMPORTANT
Use 2048-bit certificates. The installation fails if you use 4096-bit certificates
with Prism Central 2022.x.

IMPORTANT
11.3.3. Internet access for Prism Central

Prism Central requires internet access to obtain the Red Hat Enterprise Linux CoreOS (RHCOS) image
that is required to install the cluster. The RHCOS image for Nutanix is available at
rhcos.mirror.openshift.com.

authentication.
user.
IMPORTANT
NOTE
1962
NOTE
Procedure
NOTE
ecdsa algorithm.
NOTE

task:
Example output
Agent pid 31874
NOTE
1963
NOTE

Example output
Next steps
program.

for installation.
Prerequisites
Procedure
IMPORTANT
IMPORTANT
1964
IMPORTANT
components.
11.3.6. Adding Nutanix root CA certificates to your system trust

Because the installation program requires access to the Prism Central API, you must add your Nutanix
trusted root CA certificates to your system trust before you install an OpenShift Container Platform
cluster.
Procedure
1. From the Prism Central web console, download the Nutanix root CA certificates.
2. Extract the compressed file that contains the Nutanix root CA certificates.
3. Add the files for your operating system to the system trust. For example, on a Fedora operating
# cp certs/lin/* /etc/pki/ca-trust/source/anchors
4. Update your system trust. For example, on a Fedora operating system, run the following
command:
# update-ca-trust extract

You can customize the OpenShift Container Platform cluster you install on Nutanix.
Prerequisites
cluster.
You have verified that you have met the Nutanix networking requirements. For more
information, see "Preparing to install on Nutanix".
Procedure
1965
command:
NOTE

ii. Select nutanix as the platform to target.
iii. Enter the Prism Central domain name or IP address.
iv. Enter the port that is used to log into Prism Central.
v. Enter the credentials that are used to log into Prism Central.
The installation program connects to Prism Central.
vi. Select the Prism Element that will manage the OpenShift Container Platform cluster.
vii. Select the network subnet to use.
viii. Enter the virtual IP address that you configured for control plane API access.
ix. Enter the virtual IP address that you configured for cluster ingress.
x. Enter the base domain. This base domain must be the same one that you configured in
the DNS records.
xi. Enter a descriptive name for your cluster.

The cluster name you enter must match the cluster name you specified when
configuring the DNS records.
2. Optional: Update one or more of the default configuration parameters in the

1966

install.config.yaml file to customize the installation.
NOTE

parameter to 0. This ensures that cluster’s control planes are schedulable. For
more information, see "Installing a three-node cluster on Nutanix".
IMPORTANT

Installation configuration parameters for Nutanix
11.3.7.1. Sample customized install-config.yaml file for Nutanix
IMPORTANT
apiVersion: v1
compute: 2
name: worker
replicas: 3
platform:
nutanix: 4
cpus: 2
coresPerSocket: 2
memoryMiB: 8196
osDisk:
diskSizeGiB: 120
categories: 5
- key: <category_key_name>
value: <category_value>
controlPlane: 6
name: master
replicas: 3
platform:
1967
nutanix: 8
cpus: 4
coresPerSocket: 2
memoryMiB: 16384
osDisk:
diskSizeGiB: 120
categories: 9
metadata:
networking:
clusterNetwork:
- cidr: 10.128.0.0/14
hostPrefix: 23
machineNetwork:
- cidr: 10.0.0.0/16
serviceNetwork:
- 172.30.0.0/16
platform:
nutanix:
apiVIPs:
- 10.40.142.7 12
bootType: Legacy
categories: 13
project: 14
type: name
name: <project_name>
ingressVIPs:
- 10.40.142.8 15
prismCentral:
endpoint:
address: your.prismcentral.domainname 16
port: 9440 17
password: <password> 18
username: <username> 19
prismElements:
- endpoint:
address: your.prismelement.domainname
port: 9440
uuid: 0005b0f1-8f43-a0f2-02b7-3cecef193712
subnetUUIDs:
- c7938dc6-7659-453e-a688-e26020c68e43
clusterOSImage: http://example.com/images/rhcos-47.83.202103221318-0-nutanix.x86_64.qcow2
20
publish: External
1968

fips: false 22
1 10 12 15 16 17 18 19 21 Required. The installation program prompts you for this value.

IMPORTANT

accounts for the dramatically decreased machine performance.
4 8 Optional: Provide additional configuration for the machine pool parameters for the compute and
5 9 13 Optional: Provide one or more pairs of a prism category key and a prism category value. These
category key-value pairs must exist in Prism Central. You can provide separate categories to
compute machines, control plane machines, or all machines.
11 TThe cluster network plugin to install. The default value OVNKubernetes is the only supported
value.
14 Optional: Specify a project with which VMs are associated. Specify either name or uuid for the
project type, and then provide the corresponding UUID or project name. You can associate
projects to compute machines, control plane machines, or all machines.
20 Optional: By default, the installation program downloads and installs the Red Hat Enterprise Linux
CoreOS (RHCOS) image. If Prism Central does not have internet access, you can override the
default behavior by hosting the RHCOS image on any HTTP server and pointing the installation
program to the image.
IMPORTANT
architectures.
1969
23 Optional: You can provide the sshKey value that you use to access the machines in your cluster.
NOTE
process uses.
11.3.7.2. Configuring failure domains
Failure domains improve the fault tolerance of an OpenShift Container Platform cluster by distributing
control plane and compute machines across multiple Nutanix Prism Elements (clusters).
TIP
It is recommended that you configure three failure domains to ensure high-availability.
Prerequisites
You have an installation configuration file (install-config.yaml).
Procedure
1. Edit the install-config.yaml file and add the following stanza to configure the first failure
domain:
apiVersion: v1
compute:
# ...
platform:
nutanix:
failureDomains:
- name: <failure_domain_name>
prismElement:
name: <prism_element_name>
uuid: <prism_element_uuid>
subnetUUIDs:
- <network_uuid>
# ...
where:
<failure_domain_name>
Specifies a unique name for the failure domain. The name is limited to 64 or fewer
characters, which can include lower-case letters, digits, and a dash (-). The dash cannot be in
the leading or ending position of the name.
<prism_element_name>
Optional. Specifies the name of the Prism Element.
<prism_element_uuid>
Specifies the UUID of the Prism Element.
1970
<network_uuid>
Specifies the UUID of the Prism Element subnet object. The subnet’s IP address prefix
(CIDR) should contain the virtual IP addresses that the OpenShift Container Platform
cluster uses. Only one subnet per failure domain (Prism Element) in an OpenShift Container
Platform cluster is supported.
2. As required, configure additional failure domains.
3. To distribute control plane and compute machines across the failure domains, do one of the
following:
If compute and control plane machines can share the same set of failure domains, add the
failure domain names under the cluster’s default machine configuration.
Example of control plane and compute machines sharing a set of failure domains
apiVersion: v1
compute:
# ...
platform:
nutanix:
failureDomains:
- failure-domain-1
- failure-domain-2
- failure-domain-3
# ...
If compute and control plane machines must use different failure domains, add the failure
domain names under the respective machine pools.
Example of control plane and compute machines using different failure domains
apiVersion: v1
compute:
# ...
controlPlane:
platform:
nutanix:
failureDomains:
- failure-domain-1
- failure-domain-2
- failure-domain-3
# ...
compute:
platform:
nutanix:
failureDomains:
- failure-domain-1
- failure-domain-2
# ...
4. Save the file.
1971
Prerequisites
NOTE
(169.254.169.254).
Procedure
apiVersion: v1
proxy:
must be http.

destinations.
1972
NOTE
NOTE
NOTE
created.

IMPORTANT

Procedure
Portal.
1973
$ tar xvf <file>

$ echo $PATH
Verification
$ oc <command>

Procedure
Portal.

C:\> path
Verification
C:\> oc <command>

Procedure
Portal.
1974
NOTE

$ echo $PATH
Verification
$ oc <command>
11.3.9. Configuring IAM for Nutanix

management secrets.
Prerequisites
You have an install-config.yaml file.
Procedure
1. Create a YAML file that contains the credentials data in the following format:
Credentials data format
credentials:
- type: basic_auth 1
data:
prismCentral: 2
username: <username_for_prism_central>
password: <password_for_prism_central>
prismElements: 3
- name: <name_of_prism_element>
username: <username_for_prism_element>
password: <password_for_prism_element>
1 Specify the authentication type. Only basic authentication is supported.
2 Specify the Prism Central credentials.
1975
3 Optional: Specify the Prism Element credentials.

--included \ 1
metadata:
annotations:
include.release.openshift.io/self-managed-high-availability: "true"
labels:
name: openshift-machine-api-nutanix
spec:
providerSpec:
kind: NutanixProviderSpec
secretRef:
name: nutanix-credentials
command:
$ ccoctl nutanix create-shared-secrets \

--credentials-source-filepath=<path_to_credentials_file> 3
1976
1 Specify the path to the directory that contains the files for the component
CredentialsRequests objects.
3 Optional: Specify the directory that contains the credentials data YAML file. By default,
ccoctl expects this file to be in <home_directory>/.nutanix/credentials.
5. Edit the install-config.yaml configuration file so that the credentialsMode parameter is set to
Manual.
apiVersion: v1
...
1 Add this line to set the credentialsMode parameter to Manual.
6. Create the installation manifests by running the following command:
$ openshift-install create manifests --dir <installation_directory> 1
1 Specify the path to the directory that contains the install-config.yaml file for your cluster.
7. Copy the generated credential files to the target manifests directory by running the following
command:
$ cp <ccoctl_output_dir>/manifests/*credentials.yaml ./<installation_directory>/manifests
Verification
Ensure that the appropriate secrets exist in the manifests directory.
$ ls ./<installation_directory>/manifests
Example output
cluster-config.yaml
cluster-dns-02-config.yml
cluster-infrastructure-02-config.yml
cluster-ingress-02-config.yml
cluster-network-01-crd.yml
cluster-network-02-config.yml
cluster-proxy-01-config.yaml
cluster-scheduler-02-config.yml
cvo-overrides.yaml
kube-cloud-config.yaml
kube-system-configmap-root-ca.yaml
1977
machine-config-server-tls-secret.yaml
openshift-config-secret-pull-secret.yaml
openshift-cloud-controller-manager-nutanix-credentials-credentials.yaml
openshift-machine-api-nutanix-credentials-credentials.yaml
11.3.10. Adding config map and secret resources required for Nutanix CCM
Installations on Nutanix require additional ConfigMap and Secret resources to integrate with the
Nutanix Cloud Controller Manager (CCM).
Prerequisites
You have created a manifests directory within your installation directory.
Procedure
1. Navigate to the manifests directory:
$ cd <path_to_installation_directory>/manifests
2. Create the cloud-conf ConfigMap file with the name openshift-cloud-controller-manager-

cloud-config.yaml and add the following information:
apiVersion: v1
kind: ConfigMap
metadata:
name: cloud-conf
namespace: openshift-cloud-controller-manager
data:
cloud.conf: "{
\"prismCentral\": {
\"address\": \"<prism_central_FQDN/IP>\", 1
\"port\": 9440,
\"credentialRef\": {
\"kind\": \"Secret\",
\"name\": \"nutanix-credentials\",
\"namespace\": \"openshift-cloud-controller-manager\"
}
},
\"topologyDiscovery\": {
\"type\": \"Prism\",
\"topologyCategories\": null
},
\"enableCustomLabeling\": true
}"
1 Specify the Prism Central FQDN/IP.
3. Verify that the file cluster-infrastructure-02-config.yml exists and has the following
information:
spec:
cloudConfig:
1978
key: config
name: cloud-provider-config

IMPORTANT
Prerequisites
cluster.
Procedure
deployment:

--log-level=info 2

config.yaml file.
Verification
IMPORTANT
Example output
...
1979
IMPORTANT
11.3.12. Configuring the default storage container

After you install the cluster, you must install the Nutanix CSI Operator and configure the default storage
container for the cluster.
For more information, see the Nutanix documentation for installing the CSI Operator and configuring
registry storage.


11.3.15. Next steps

Opt out of remote health reporting
Customize your cluster
11.4. INSTALLING A CLUSTER ON NUTANIX IN A RESTRICTED

NETWORK
In OpenShift Container Platform 4.15, you can install a cluster on Nutanix infrastructure in a restricted
1980
In OpenShift Container Platform 4.15, you can install a cluster on Nutanix infrastructure in a restricted
network by creating an internal mirror of the installation release content.
You have reviewed details about the OpenShift Container Platform installation and update
processes.
The installation program requires access to port 9440 on Prism Central and Prism Element. You
verified that port 9440 is accessible.
If you use a firewall, you have met these prerequisites:
You confirmed that port 9440 is accessible. Control plane nodes must be able to reach
Prism Central and Prism Element on port 9440 for the installation to succeed.
You configured the firewall to grant access to the sites that OpenShift Container Platform
requires. This includes the use of Telemetry.
If your Nutanix environment is using the default self-signed SSL/TLS certificate, replace it with
a certificate that is signed by a CA. The installation program requires a valid CA-signed
certificate to access to the Prism Central API. For more information about replacing the self-
signed certificate, see the Nutanix AOS Security Guide .
If your Nutanix environment uses an internal CA to issue certificates, you must configure a
cluster-wide proxy as part of the installation process. For more information, see Configuring a
custom PKI.
IMPORTANT
Use 2048-bit certificates. The installation fails if you use 4096-bit certificates
with Prism Central 2022.x.
You have a container image registry, such as Red Hat Quay. If you do not already have a registry,
you can create a mirror registry using mirror registry for Red Hat OpenShift .
You have used the oc-mirror OpenShift CLI (oc) plugin to mirror all of the required OpenShift
Container Platform content and other images, including the Nutanix CSI Operator, to your
mirror registry.
IMPORTANT

1981
your restrictions.

authentication.
user.
IMPORTANT
NOTE
Procedure
NOTE
1982
NOTE
ecdsa algorithm.
NOTE

task:
Example output
Agent pid 31874
NOTE

Example output
Next steps
1983
program.
11.4.4. Adding Nutanix root CA certificates to your system trust

Because the installation program requires access to the Prism Central API, you must add your Nutanix
trusted root CA certificates to your system trust before you install an OpenShift Container Platform
cluster.
Procedure
1. From the Prism Central web console, download the Nutanix root CA certificates.
2. Extract the compressed file that contains the Nutanix root CA certificates.
3. Add the files for your operating system to the system trust. For example, on a Fedora operating
# cp certs/lin/* /etc/pki/ca-trust/source/anchors
4. Update your system trust. For example, on a Fedora operating system, run the following
command:
# update-ca-trust extract
11.4.5. Downloading the RHCOS cluster image

Prism Central requires access to the Red Hat Enterprise Linux CoreOS (RHCOS) image to install the
cluster. You can use the installation program to locate and download the RHCOS image and make it
available through an internal HTTP server or Nutanix Objects.
Prerequisites
Procedure
$ ./openshift-install coreos print-stream-json
2. Use the output of the command to find the location of the Nutanix image, and click the link to
download it.
Example output
"nutanix": {
"release": "411.86.202210041459-0",
"formats": {
"qcow2": {
"disk": {
"location": "https://rhcos.mirror.openshift.com/art/storage/releases/rhcos-
1984
4.11/411.86.202210041459-0/x86_64/rhcos-411.86.202210041459-0-
nutanix.x86_64.qcow2",
"sha256":
"42e227cac6f11ac37ee8a2f9528bb3665146566890577fd55f9b950949e5a54b"
3. Make the image available through an internal HTTP server or Nutanix Objects.
4. Note the location of the downloaded image. You update the platform section in the installation
configuration file (install-config.yaml) with the image’s location before deploying the cluster.
Snippet of an install-config.yaml file that specifies the RHCOS image
platform:
nutanix:
clusterOSImage: http://example.com/images/rhcos-411.86.202210041459-0-
nutanix.x86_64.qcow2

You can customize the OpenShift Container Platform cluster you install on Nutanix.
Prerequisites
You have the imageContentSourcePolicy.yaml file that was created when you mirrored your
registry.
You have the location of the Red Hat Enterprise Linux CoreOS (RHCOS) image you download.
You have verified that you have met the Nutanix networking requirements. For more
information, see "Preparing to install on Nutanix".
Procedure
command:
1985
NOTE

ii. Select nutanix as the platform to target.
iii. Enter the Prism Central domain name or IP address.
iv. Enter the port that is used to log into Prism Central.
v. Enter the credentials that are used to log into Prism Central.
The installation program connects to Prism Central.
vi. Select the Prism Element that will manage the OpenShift Container Platform cluster.
vii. Select the network subnet to use.
viii. Enter the virtual IP address that you configured for control plane API access.
ix. Enter the virtual IP address that you configured for cluster ingress.
x. Enter the base domain. This base domain must be the same one that you configured in
the DNS records.
xi. Enter a descriptive name for your cluster.

The cluster name you enter must match the cluster name you specified when
configuring the DNS records.
2. In the install-config.yaml file, set the value of platform.nutanix.clusterOSImage to the image

location or name. For example:
platform:
nutanix:
clusterOSImage: http://mirror.example.com/images/rhcos-47.83.202103221318-0-
nutanix.x86_64.qcow2
1986

c. Add the image content resources, which resemble the following YAML excerpt:
- mirrors:
- mirrors:
For these values, use the imageContentSourcePolicy.yaml file that was created when you
mirrored the registry.
d. Optional: Set the publishing strategy to Internal:
publish: Internal

install.config.yaml file to customize the installation.
NOTE

parameter to 0. This ensures that cluster’s control planes are schedulable. For
more information, see "Installing a three-node cluster on {platform}".
IMPORTANT

1987
Installation configuration parameters for Nutanix
11.4.6.1. Sample customized install-config.yaml file for Nutanix
IMPORTANT
apiVersion: v1
compute: 2
name: worker
replicas: 3
platform:
nutanix: 4
cpus: 2
coresPerSocket: 2
memoryMiB: 8196
osDisk:
diskSizeGiB: 120
categories: 5
controlPlane: 6
name: master
replicas: 3
platform:
nutanix: 8
cpus: 4
coresPerSocket: 2
memoryMiB: 16384
osDisk:
diskSizeGiB: 120
categories: 9
metadata:
networking:
clusterNetwork:
- cidr: 10.128.0.0/14
hostPrefix: 23
machineNetwork:
- cidr: 10.0.0.0/16
1988
serviceNetwork:
- 172.30.0.0/16
platform:
nutanix:
apiVIP: 10.40.142.7 12
ingressVIP: 10.40.142.8 13
bootType: Legacy
categories: 14
project: 15
type: name
name: <project_name>
prismCentral:
endpoint:
address: your.prismcentral.domainname 16
port: 9440 17
password: <password> 18
username: <username> 19
prismElements:
- endpoint:
address: your.prismelement.domainname
port: 9440
uuid: 0005b0f1-8f43-a0f2-02b7-3cecef193712
subnetUUIDs:
- c7938dc6-7659-453e-a688-e26020c68e43
clusterOSImage: http://example.com/images/rhcos-47.83.202103221318-0-nutanix.x86_64.qcow2
20
publish: External
fips: false 22
- mirrors:
- mirrors:
1 10 12 13 16 17 18 19 Required. The installation program prompts you for this value.
1989

IMPORTANT

accounts for the dramatically decreased machine performance.
4 8 Optional: Provide additional configuration for the machine pool parameters for the compute and
5 9 14 Optional: Provide one or more pairs of a prism category key and a prism category value. These
category key-value pairs must exist in Prism Central. You can provide separate categories to
compute machines, control plane machines, or all machines.
11 TThe cluster network plugin to install. The default value OVNKubernetes is the only supported
value.
15 Optional: Specify a project with which VMs are associated. Specify either name or uuid for the
project type, and then provide the corresponding UUID or project name. You can associate
projects to compute machines, control plane machines, or all machines.
20 Optional: By default, the installation program downloads and installs the Red Hat Enterprise Linux
CoreOS (RHCOS) image. If Prism Central does not have internet access, you can override the
default behavior by hosting the RHCOS image on any HTTP server or Nutanix Objects and pointing
the installation program to the image.
registry uses to serve content. For example registry.example.com or
IMPORTANT
architectures.
23 Optional: You can provide the sshKey value that you use to access the machines in your cluster.
NOTE
1990
NOTE
process uses.
25 Provide these values from the metadata.name: release-0 section of the

imageContentSourcePolicy.yaml file that was created when you mirrored the registry.
11.4.6.2. Configuring failure domains
Failure domains improve the fault tolerance of an OpenShift Container Platform cluster by distributing
control plane and compute machines across multiple Nutanix Prism Elements (clusters).
TIP
It is recommended that you configure three failure domains to ensure high-availability.
Prerequisites
You have an installation configuration file (install-config.yaml).
Procedure
1. Edit the install-config.yaml file and add the following stanza to configure the first failure
domain:
apiVersion: v1
compute:
# ...
platform:
nutanix:
failureDomains:
- name: <failure_domain_name>
prismElement:
name: <prism_element_name>
uuid: <prism_element_uuid>
subnetUUIDs:
- <network_uuid>
# ...
where:
<failure_domain_name>
Specifies a unique name for the failure domain. The name is limited to 64 or fewer
characters, which can include lower-case letters, digits, and a dash (-). The dash cannot be in
the leading or ending position of the name.
<prism_element_name>
Optional. Specifies the name of the Prism Element.
1991
<prism_element_uuid>
Specifies the UUID of the Prism Element.
<network_uuid>
Specifies the UUID of the Prism Element subnet object. The subnet’s IP address prefix
(CIDR) should contain the virtual IP addresses that the OpenShift Container Platform
cluster uses. Only one subnet per failure domain (Prism Element) in an OpenShift Container
Platform cluster is supported.
2. As required, configure additional failure domains.
3. To distribute control plane and compute machines across the failure domains, do one of the
following:
If compute and control plane machines can share the same set of failure domains, add the
failure domain names under the cluster’s default machine configuration.
Example of control plane and compute machines sharing a set of failure domains
apiVersion: v1
compute:
# ...
platform:
nutanix:
failureDomains:
- failure-domain-1
- failure-domain-2
- failure-domain-3
# ...
If compute and control plane machines must use different failure domains, add the failure
domain names under the respective machine pools.
Example of control plane and compute machines using different failure domains
apiVersion: v1
compute:
# ...
controlPlane:
platform:
nutanix:
failureDomains:
- failure-domain-1
- failure-domain-2
- failure-domain-3
# ...
compute:
platform:
nutanix:
failureDomains:
- failure-domain-1
- failure-domain-2
# ...
1992
4. Save the file.
Prerequisites
NOTE
(169.254.169.254).
Procedure
apiVersion: v1
proxy:
must be http.

destinations.
1993
NOTE
NOTE
NOTE
created.

IMPORTANT

Procedure
Portal.
1994
$ tar xvf <file>

$ echo $PATH
Verification
$ oc <command>

Procedure
Portal.

C:\> path
Verification
C:\> oc <command>

Procedure
Portal.
1995
NOTE

$ echo $PATH
Verification
$ oc <command>
11.4.8. Configuring IAM for Nutanix

management secrets.
Prerequisites
You have an install-config.yaml file.
Procedure
1. Create a YAML file that contains the credentials data in the following format:
Credentials data format
credentials:
- type: basic_auth 1
data:
prismCentral: 2
username: <username_for_prism_central>
password: <password_for_prism_central>
prismElements: 3
- name: <name_of_prism_element>
username: <username_for_prism_element>
password: <password_for_prism_element>
1 Specify the authentication type. Only basic authentication is supported.
2 Specify the Prism Central credentials.
3 Optional: Specify the Prism Element credentials.
1996

--included \ 1
metadata:
annotations:
include.release.openshift.io/self-managed-high-availability: "true"
labels:
name: openshift-machine-api-nutanix
spec:
providerSpec:
kind: NutanixProviderSpec
secretRef:
name: nutanix-credentials
command:
$ ccoctl nutanix create-shared-secrets \

--credentials-source-filepath=<path_to_credentials_file> 3
1 Specify the path to the directory that contains the files for the component
CredentialsRequests objects.
1997
3 Optional: Specify the directory that contains the credentials data YAML file. By default,
ccoctl expects this file to be in <home_directory>/.nutanix/credentials.
5. Edit the install-config.yaml configuration file so that the credentialsMode parameter is set to
Manual.
apiVersion: v1
...
1 Add this line to set the credentialsMode parameter to Manual.
6. Create the installation manifests by running the following command:
$ openshift-install create manifests --dir <installation_directory> 1
1 Specify the path to the directory that contains the install-config.yaml file for your cluster.
7. Copy the generated credential files to the target manifests directory by running the following
command:
$ cp <ccoctl_output_dir>/manifests/*credentials.yaml ./<installation_directory>/manifests
Verification
Ensure that the appropriate secrets exist in the manifests directory.
$ ls ./<installation_directory>/manifests
Example output
cluster-config.yaml
cluster-dns-02-config.yml
cluster-infrastructure-02-config.yml
cluster-ingress-02-config.yml
cluster-network-01-crd.yml
cluster-network-02-config.yml
cluster-proxy-01-config.yaml
cluster-scheduler-02-config.yml
cvo-overrides.yaml
kube-cloud-config.yaml
kube-system-configmap-root-ca.yaml
machine-config-server-tls-secret.yaml
1998
openshift-config-secret-pull-secret.yaml
openshift-cloud-controller-manager-nutanix-credentials-credentials.yaml
openshift-machine-api-nutanix-credentials-credentials.yaml

IMPORTANT
Prerequisites
cluster.
Procedure
deployment:

--log-level=info 2

config.yaml file.
Verification
IMPORTANT
Example output
...
1999
IMPORTANT
11.4.10. Post installation

Complete the following steps to complete the configuration of your cluster.
Procedure
OperatorHub object:

TIP
11.4.10.2. Installing the policy resources into the cluster
Mirroring the OpenShift Container Platform content using the oc-mirror OpenShift CLI (oc) plugin
creates resources, which include catalogSource-certified-operator-index.yaml and
imageContentSourcePolicy.yaml.
2000
registry and redirects image pull requests from the online registries to the mirror registry.
The CatalogSource resource is used by Operator Lifecycle Manager (OLM) to retrieve

information about the available Operators in the mirror registry, which lets users discover and
install Operators.
After you install the cluster, you must install these resources into the cluster.
Prerequisites
Procedure
2. Apply the YAML files from the results directory to the cluster:
$ oc apply -f ./oc-mirror-workspace/results-<id>/
Verification
1. Verify that the ImageContentSourcePolicy resources were successfully installed:
2. Verify that the CatalogSource resources were successfully installed:
$ oc get catalogsource --all-namespaces
11.4.10.3. Configuring the default storage container
After you install the cluster, you must install the Nutanix CSI Operator and configure the default storage
container for the cluster.
For more information, see the Nutanix documentation for installing the CSI Operator and configuring
registry storage.


2001
11.4.13. Next steps

If necessary, see Opt out of remote health reporting
Customize your cluster
11.5. INSTALLING A THREE-NODE CLUSTER ON NUTANIX

In OpenShift Container Platform version 4.15, you can install a three-node cluster on Nutanix. A three-
node cluster consists of three control plane machines, which also act as compute machines. This type of
cluster provides a smaller, more resource efficient cluster, for cluster administrators and developers to
use for testing, development, and production.

NOTE
Prerequisites
Procedure
Set the number of compute replicas to 0 in your install-config.yaml file, as shown in the
apiVersion: v1
compute:
- name: worker
platform: {}
replicas: 0
# ...
11.5.2. Next steps

Installing a cluster on Nutanix
11.6. UNINSTALLING A CLUSTER ON NUTANIX

You can remove a cluster that you deployed to Nutanix.
2002

NOTE
Prerequisites
Procedure

NOTE
11.7. INSTALLATION CONFIGURATION PARAMETERS FOR NUTANIX

Before you deploy an OpenShift Container Platform cluster on Nutanix, you provide parameters to
11.7.1. Available installation configuration parameters for Nutanix

The following tables specify the required, optional, and Nutanix-specific installation configuration
NOTE
2003
NOTE

versions.

combination of the
baseDomain and
<metadata.name>.

consumed.
The name of the cluster. DNS String of lowercase letters and hyphens (- ), such as
metadata: records for the cluster are all dev.
name: subdomains of
{{.metadata.name}}.
{{.baseDomain}}.
2004

alibabacloud, aws,
ibmcloud, nutanix,

"auth":"b3Blb=",
}
}
}
NOTE
2005

NOTE
You cannot modify

by the networking
object after
installation.


networking:
networking:
clusterNetwork:
- cidr: 10.128.0.0/14
hostPrefix: 23

An IPv4 network.

23 provides 510 (2^(32 - 23) - 2) pod
IP addresses.
serviceNetwork:
networking:
- 172.30.0.0/16

networking:
networking:
machineNetwork:
- cidr: 10.0.0.0/16
2006

NOTE
the CIDR that the
in.



et:

2007

section.


(the default).

cores.
IMPORTANT
If you disable
simultaneous
that your capacity
the dramatically
decreased machine
performance.

name:
2008
value.



(the default).
2009

cores.
IMPORTANT
If you disable
simultaneous
that your capacity
the dramatically
decreased machine
performance.

name:

parameter value.

replicas:

2010

IMPORTANT
To enable FIPS mode

must run the
from a Red Hat
Enterprise Linux
(RHEL) computer
in FIPS mode. For
more information
about configuring
FIPS mode on RHEL,
see Installing the
Enterprise Linux
(RHEL) or Red Hat
Enterprise Linux
CoreOS (RHCOS)
OpenShift Container
Platform core
components use the
RHEL cryptographic
libraries that have
been submitted to
NIST for FIPS 140-
only the x86_64,
ppc64le, and s390x
architectures.
NOTE

File storage, you
cannot enable FIPS
mode.

2011


ces:
mirrors:

IMPORTANT

cluster will become
non-functional. For
more information,

NOTE
For production
OpenShift Container
which you want to
SSH key that your
ssh-agent process
uses.
11.7.1.4. Additional Nutanix configuration parameters
2012
Additional Nutanix configuration parameters are described in the following table:
Table 11.6. Additional Nutanix cluster parameters
The name of a prism category key to String

compute: apply to compute VMs. This parameter
platform: must be accompanied by the value
nutanix: parameter, and both key and value
categories: parameters must exist in Prism
key: Central. For more information on
categories, see Category management.
The value of a prism category key- String

compute: value pair to apply to compute VMs.
platform: This parameter must be accompanied
nutanix: by the key parameter, and both key
categories: and value parameters must exist in
value: Prism Central.
The failure domains that apply to only List.

platform: The name of one or more failures
nutanix: Failure domains are specified in domains.
failureDomains: platform.nutanix.failureDomains.
The type of identifier you use to select name or uuid

compute: a project for compute VMs. Projects
platform: define logical groups of user roles for
nutanix: managing permissions, networks, and
project: other parameters. For more
type: information on projects, see Projects
Overview.
The name or UUID of a project with String

compute: which compute VMs are associated.
platform: This parameter must be accompanied
nutanix: by the type parameter.
project:
name: or
uuid:
The boot type that the compute Legacy, SecureBoot or UEFI. The
compute: machines use. You must use the default is Legacy.
platform: Legacy boot type in OpenShift
nutanix: Container Platform 4.15. For more
bootType: information on boot types, see
Understanding UEFI, Secure Boot, and
TPM in the Virtualized Environment.
2013

controlPlane: apply to control plane VMs. This
platform: parameter must be accompanied by
nutanix: the value parameter, and both key
categories: and value parameters must exist in
key: Prism Central. For more information
on categories, see Category
management.

controlPlane: value pair to apply to control plane
platform: VMs. This parameter must be
nutanix: accompanied by the key parameter,
categories: and both key and value parameters
value: must exist in Prism Central.
The failure domains that apply to only List.

platform: The name of one or more failures
nutanix: Failure domains are specified in domains.
failureDomains: platform.nutanix.failureDomains.
The type of identifier you use to select name or uuid

controlPlane: a project for control plane VMs.
platform: Projects define logical groups of user
nutanix: roles for managing permissions,
project: networks, and other parameters. For
type: more information on projects, see
Projects Overview.

controlPlane: which control plane VMs are
platform: associated. This parameter must be
nutanix: accompanied by the type parameter.
project:
name: or
uuid:

platform: apply to all VMs. This parameter must
nutanix: be accompanied by the value
parameter, and both key and value
defaultMachinePlatf parameters must exist in Prism
orm: Central. For more information on
categories: categories, see Category management.
key:
2014

platform: value pair to apply to all VMs. This
nutanix: parameter must be accompanied by
the key parameter, and both key and
defaultMachinePlatf value parameters must exist in Prism
orm: Central.
categories:
value:
The failure domains that apply to both List.

platform: control plane and compute machines.
nutanix: The name of one or more failures
Failure domains are specified in domains.
defaulatMachinePla platform.nutanix.failureDomains.
tform:
failureDomains:
The type of identifier you use to select name or uuid.

platform: a project for all VMs. Projects define
nutanix: logical groups of user roles for
managing permissions, networks, and
defaultMachinePlatf other parameters. For more
orm: information on projects, see Projects
project: Overview.
type:

platform: which all VMs are associated. This
nutanix: parameter must be accompanied by
the type parameter.
defaultMachinePlatf
orm:
project:
name: or
uuid:
The boot type for all machines. You Legacy, SecureBoot or UEFI. The
platform: must use the Legacy boot type in default is Legacy.
nutanix: OpenShift Container Platform 4.15.
For more information on boot types,
defaultMachinePlatf see Understanding UEFI, Secure Boot,
orm: and TPM in the Virtualized
bootType: Environment.
2015
The virtual IP (VIP) address that you IP address

platform: configured for control plane API
nutanix: access.
apiVIP:
By default, the installation program A list of configured failure domains.

platform: installs cluster machines to a single
nutanix: Prism Element instance. You can For more information on usage, see
failureDomains: specify additional Prism Element "Configuring a failure domain" in
- name: instances for fault tolerance, and then "Installing a cluster on Nutanix".
prismElement: apply them to:
name:
The cluster’s default machine
uuid:
configuration
subnetUUIDs:
- Only control plane or
compute machine pools
The virtual IP (VIP) address that you IP address

platform: configured for cluster ingress.
nutanix:
ingressVIP:
The Prism Central domain name or IP String

platform: address.
nutanix:
prismCentral:
endpoint:
address:
The port that is used to log into Prism String

platform: Central.
nutanix:
prismCentral:
endpoint:
port:
The password for the Prism Central String

platform: user name.
nutanix:
prismCentral:
password:
2016
The user name that is used to log into String

platform: Prism Central.
nutanix:
prismCentral:
username:
The Prism Element domain name or IP String

platform: address. [1]
nutanix:
prismElements:
endpoint:
address:
The port that is used to log into Prism String

platform: Element.
nutanix:
prismElements:
endpoint:
port:
The universally unique identifier (UUID) String

platform: for Prism Element.
nutanix:
prismElements:
uuid:
The UUID of the Prism Element String

platform: network that contains the virtual IP
nutanix: addresses and DNS records that you
subnetUUIDs: configured. [ 2]
Optional: By default, the installation An HTTP or HTTPS URL, optionally

platform: program downloads and installs the with a SHA-256 checksum. For
nutanix: Red Hat Enterprise Linux CoreOS example,
(RHCOS) image. If Prism Central does http://example.com/images/rhcos-
clusterOSImage: not have internet access, you can 47.83.202103221318-0-
override the default behavior by nutanix.x86_64.qcow2
hosting the RHCOS image on any
HTTP server and pointing the
installation program to the image.
1. The prismElements section holds a list of Prism Elements (clusters). A Prism Element
encompasses all of the Nutanix resources, for example virtual machines and subnets, that are
used to host the OpenShift Container Platform cluster.
2. Only one subnet per Prism Element in an OpenShift Container Platform cluster is supported.
2017
CHAPTER 12. INSTALLING ON BARE METAL
12.1. PREPARING FOR BARE METAL CLUSTER INSTALLATION
processes.
You have read the documentation on selecting a cluster installation method and preparing it for
users.
12.1.2. Planning a bare metal cluster for OpenShift Virtualization

If you will use OpenShift Virtualization, it is important to be aware of several requirements before you
install your bare metal cluster.
If you want to use live migration features, you must have multiple worker nodes at the time of
cluster installation. This is because live migration requires the cluster-level high availability (HA)
flag to be set to true. The HA flag is set when a cluster is installed and cannot be changed
afterwards. If there are fewer than two worker nodes defined when you install your cluster, the
HA flag is set to false for the life of the cluster.
NOTE
You can install OpenShift Virtualization on a single-node cluster, but single-node

OpenShift does not support high availability.
Live migration requires shared storage. Storage for OpenShift Virtualization must support and
use the ReadWriteMany (RWX) access mode.
If you plan to use Single Root I/O Virtualization (SR-IOV), ensure that your network interface
controllers (NICs) are supported by OpenShift Container Platform.
Getting started with OpenShift Virtualization
Preparing your cluster for OpenShift Virtualization
About Single Root I/O Virtualization (SR-IOV) hardware networks
Connecting a virtual machine to an SR-IOV network
12.1.3. NIC partitioning for SR-IOV devices (Technology Preview)

OpenShift Container Platform can be deployed on a server with a dual port network interface card
(NIC). You can partition a single, high-speed dual port NIC into multiple virtual functions (VFs) and
enable SR-IOV.
This feature supports the use of bonds for high availability with the Link Aggregation Control Protocol
(LACP).
NOTE
2018
NOTE
Only one LACP can be declared by physical NIC.
An OpenShift Container Platform cluster can be deployed on a bond interface with 2 VFs on 2 physical
functions (PFs) using the following methods:
Agent-based installer
NOTE
The minimum required version of nmstate is:
1.4.2-4 for RHEL 8 versions
2.2.7 for RHEL 9 versions
Installer-provisioned infrastructure installation
User-provisioned infrastructure installation
IMPORTANT
Support for Day 1 operations associated with enabling NIC partitioning for SR-IOV
devices is a Technology Preview feature only. Technology Preview features are not
supported with Red Hat production service level agreements (SLAs) and might not be
functionally complete. Red Hat does not recommend using them in production. These
Example: Bonds and SR-IOV dual-nic node network configuration
Optional: Configuring host network interfaces for dual port NIC
Bonding multiple SR-IOV network interfaces to a dual port NIC interface
12.1.4. Choosing a method to install OpenShift Container Platform on bare metal

The OpenShift Container Platform installation program offers four methods for deploying a cluster:
Interactive: You can deploy a cluster with the web-based Assisted Installer. This is the
recommended approach for clusters with networks connected to the internet. The Assisted
Installer is the easiest way to install OpenShift Container Platform, it provides smart defaults,
and it performs pre-flight validations before installing the cluster. It also provides a RESTful API
for automation and advanced configuration scenarios.
Local Agent-based: You can deploy a cluster locally with the agent-based installer for air-
gapped or restricted networks. It provides many of the benefits of the Assisted Installer, but you
must download and configure the agent-based installer first. Configuration is done with a
commandline interface. This approach is ideal for air-gapped or restricted networks.
2019
Automated: You can deploy a cluster on installer-provisioned infrastructure and the cluster it
maintains. The installer uses each cluster host’s baseboard management controller (BMC) for
provisioning. You can deploy clusters with both connected or air-gapped or restricted networks.
Full control: You can deploy a cluster on infrastructure that you prepare and maintain , which
provides maximum customizability. You can deploy clusters with both connected or air-gapped
or restricted networks.
The clusters have the following characteristics:
Highly available infrastructure with no single points of failure is available by default.
Administrators maintain control over what updates are applied and when.
You can install a cluster on bare metal infrastructure that is provisioned by the OpenShift Container
Platform installation program, by using the following method:
Installing an installer-provisioned cluster on bare metal: You can install OpenShift Container
Platform on bare metal by using installer provisioning.
You can install a cluster on bare metal infrastructure that you provision, by using one of the following
methods:
Installing a user-provisioned cluster on bare metal: You can install OpenShift Container
Platform on bare metal infrastructure that you provision. For a cluster that contains user-
provisioned infrastructure, you must deploy all of the required machines.
Installing a user-provisioned bare metal cluster with network customizations: You can install
a bare metal cluster on user-provisioned infrastructure with network-customizations. By
customizing your network configuration, your cluster can coexist with existing IP address
allocations in your environment and integrate with existing MTU and VXLAN configurations.
Most of the network customizations must be applied at the installation stage.
Installing a user-provisioned bare metal cluster on a restricted network: You can install a
user-provisioned bare metal cluster on a restricted or disconnected network by using a mirror
registry. You can also use this installation method to ensure that your clusters only use
container images that satisfy your organizational controls on external content.
12.2. INSTALLING A USER-PROVISIONED CLUSTER ON BARE METAL

In OpenShift Container Platform 4.15, you can install a cluster on bare metal infrastructure that you
provision.
IMPORTANT
2020
IMPORTANT
While you might be able to follow this procedure to deploy a cluster on virtualized or cloud
environments, you must be aware of additional considerations for non-bare metal
platforms. Review the information in the guidelines for deploying OpenShift Container
Platform on non-tested platforms before you attempt to install an OpenShift Container
Platform cluster in such an environment.
processes.
users.
NOTE

IMPORTANT
See Installing a user-provisioned bare metal cluster on a restricted network for more
information about performing a restricted network installation on bare metal infrastructure that
you provision.

2021
Hosts Description
control plane.
NOTE
As an exception, you can run zero compute machines in a bare metal cluster that consists
of three control plane machines only. This provides smaller, more resource efficient
clusters for cluster administrators and developers to use for testing, development, and
production. Running one compute machine is not supported.
IMPORTANT
machines.
2022
Machine Operating CPU [1] RAM Storage Input/Output

System Per Second
(IOPS)[2]

8.6 and later
[3]
1. One CPU is equivalent to one physical core when simultaneous multithreading (SMT), or
corresponding ratio: (threads per core × cores) × sockets = CPUs.
Optimizing storage
and approving them.
See Configuring a three-node cluster for details about deploying three-node clusters in bare
metal environments.
See Approving the certificate signing requests for your machines for more information about
approving cluster certificate signing requests after installation.
2023
12.2.3.4. Requirements for baremetal clusters on vSphere
Ensure you enable the disk.EnableUUID parameter on all virtual machines in your cluster.
See Installing RHCOS and starting the OpenShift Container Platform bootstrap process for
details on setting the disk.EnableUUID parameter’s value to TRUE on VMware vSphere for
user-provisioned infrastructure.
12.2.3.5. Networking requirements for user-provisioned infrastructure
During the initial boot, the machines require an IP address configuration that is set either through a
DHCP server or statically by providing the required boot options. After a network connection is
established, the machines download their Ignition config files from an HTTP or HTTPS server. The
Ignition config files are then used to set the exact state of each machine. The Machine Config Operator
completes more changes to the machines, such as the application of new certificates or keys, after
installation.
It is recommended to use a DHCP server for long-term management of the cluster machines. Ensure
that the DHCP server is configured to provide persistent IP addresses, DNS server information, and
hostnames to the cluster machines.
NOTE
If a DHCP service is not available for your user-provisioned infrastructure, you can instead
provide the IP networking configuration and the address of the DNS server to the nodes
at RHCOS install time. These can be passed as boot arguments if you are installing from
an ISO image. See the Installing RHCOS and starting the OpenShift Container Platform
bootstrap process section for more information about static IP provisioning and advanced
networking options.
The Kubernetes API server must be able to resolve the node names of the cluster machines. If the API
servers and worker nodes are in different zones, you can configure a default DNS search zone to allow
the API server to resolve the node names. Another supported approach is to always refer to hosts by
their fully-qualified domain names in both the node objects and all DNS requests.
12.2.3.5.1. Setting the cluster node hostnames through DHCP
12.2.3.5.2. Network connectivity requirements
2024
IMPORTANT
Hat.
TCP 1936 Metrics
UDP 4789 VXLAN
6081 Geneve
9100- 9101.
2025
NTP configuration for user-provisioned infrastructure

OpenShift Container Platform clusters are configured to use a public Network Time Protocol (NTP)
server by default. If you want to use a local enterprise NTP server, or if your cluster is being deployed in a
disconnected network, you can configure the cluster to use a specific time server. For more information,
see the documentation for Configuring chrony time service .
If a DHCP server provides NTP server information, the chrony time service on the Red Hat Enterprise
Linux CoreOS (RHCOS) machines read the information and can sync the clock with the NTP servers.
Configuring chrony time service
12.2.3.6. User-provisioned DNS requirements
In OpenShift Container Platform deployments, DNS name resolution is required for the following
components:
The Kubernetes API
The OpenShift Container Platform application wildcard
The bootstrap, control plane, and compute machines
Reverse DNS resolution is also required for the Kubernetes API, the bootstrap machine, the control
plane machines, and the compute machines.
DNS A/AAAA or CNAME records are used for name resolution and PTR records are used for reverse
name resolution. The reverse records are important because Red Hat Enterprise Linux CoreOS
(RHCOS) uses the reverse records to set the hostnames for all the nodes, unless the hostnames are
provided by DHCP. Additionally, the reverse records are used to generate the certificate signing
requests (CSR) that OpenShift Container Platform needs to operate.
NOTE
It is recommended to use a DHCP server to provide the hostnames to each cluster node.
See the DHCP recommendations for user-provisioned infrastructure section for more
information.
The following DNS records are required for a user-provisioned OpenShift Container Platform cluster
and they must be in place before installation. In each record, <cluster_name> is the cluster name and
<base_domain> is the base domain that you specify in the install-config.yaml file. A complete DNS
record takes the form: <component>.<cluster_name>.<base_domain>..
Table 12.6. Required DNS records
2026
Compo Record Description

nent
Kuberne api.<cluster_name>. A DNS A/AAAA or CNAME record, and a DNS PTR record,
tes API <base_domain>. to identify the API load balancer. These records must be
resolvable by both clients external to the cluster and from
api-int.<cluster_name>. A DNS A/AAAA or CNAME record, and a DNS PTR record,

<base_domain>. to internally identify the API load balancer. These records
must be resolvable from all the nodes within the cluster.
IMPORTANT
The API server must be able to resolve the

worker nodes by the hostnames that are
recorded in Kubernetes. If the API server
cannot resolve the node names, then
proxied API calls can fail, and you cannot
retrieve logs from pods.
Routes *.apps.<cluster_name>. A wildcard DNS A/AAAA or CNAME record that refers to

<base_domain>. the application ingress load balancer. The application
ingress load balancer targets the machines that run the
Ingress Controller pods. The Ingress Controller pods run on
the compute machines by default. These records must be
resolvable by both clients external to the cluster and from
For example, console-openshift-console.apps.

<cluster_name>.<base_domain> is used as a wildcard
route to the OpenShift Container Platform console.
Bootstra bootstrap.<cluster_name>. A DNS A/AAAA or CNAME record, and a DNS PTR record,
p <base_domain>. to identify the bootstrap machine. These records must be
machine resolvable by the nodes within the cluster.
Control <control_plane><n>. DNS A/AAAA or CNAME records and DNS PTR records to
plane <cluster_name>. identify each machine for the control plane nodes. These
machine <base_domain>. records must be resolvable by the nodes within the cluster.
s
Comput <compute><n>. DNS A/AAAA or CNAME records and DNS PTR records to
e <cluster_name>. identify each machine for the worker nodes. These records
machine <base_domain>. must be resolvable by the nodes within the cluster.
s
NOTE
In OpenShift Container Platform 4.4 and later, you do not need to specify etcd host and
SRV records in your DNS configuration.
2027
TIP
You can use the dig command to verify name and reverse name resolution. See the section on
Validating DNS resolution for user-provisioned infrastructure for detailed validation steps.
12.2.3.6.1. Example DNS configuration for user-provisioned clusters
This section provides A and PTR record configuration samples that meet the DNS requirements for
deploying OpenShift Container Platform on user-provisioned infrastructure. The samples are not meant
to provide advice for choosing one DNS solution over another.
In the examples, the cluster name is ocp4 and the base domain is example.com.
Example DNS A record configuration for a user-provisioned cluster

The following example is a BIND zone file that shows sample A records for name resolution in a user-
provisioned cluster.
Example 12.1. Sample DNS zone database
$TTL 1W
@ IN SOA ns1.example.com. root (
2019070700 ; serial
3H ; refresh (3 hours)
30M ; retry (30 minutes)
2W ; expiry (2 weeks)
1W ) ; minimum (1 week)
IN NS ns1.example.com.
IN MX 10 smtp.example.com.
;
;
ns1.example.com. IN A 192.168.1.5
smtp.example.com. IN A 192.168.1.5
;
helper.example.com. IN A 192.168.1.5
helper.ocp4.example.com. IN A 192.168.1.5
;
api.ocp4.example.com. IN A 192.168.1.5 1
api-int.ocp4.example.com. IN A 192.168.1.5 2
;
*.apps.ocp4.example.com. IN A 192.168.1.5 3
;
bootstrap.ocp4.example.com. IN A 192.168.1.96 4
;
control-plane0.ocp4.example.com. IN A 192.168.1.97 5
;
compute0.ocp4.example.com. IN A 192.168.1.11 8
compute1.ocp4.example.com. IN A 192.168.1.7 9
;
;EOF
1 Provides name resolution for the Kubernetes API. The record refers to the IP address of the API
load balancer.
2028
2 Provides name resolution for the Kubernetes API. The record refers to the IP address of the API
load balancer and is used for internal cluster communications.
3 Provides name resolution for the wildcard routes. The record refers to the IP address of the
application ingress load balancer. The application ingress load balancer targets the machines
that run the Ingress Controller pods. The Ingress Controller pods run on the compute machines
by default.
NOTE
In the example, the same load balancer is used for the Kubernetes API and
application ingress traffic. In production scenarios, you can deploy the API and
application ingress load balancers separately so that you can scale the load
balancer infrastructure for each in isolation.
4 Provides name resolution for the bootstrap machine.
5 6 7 Provides name resolution for the control plane machines.
8 9 Provides name resolution for the compute machines.
Example DNS PTR record configuration for a user-provisioned cluster

The following example BIND zone file shows sample PTR records for reverse name resolution in a user-
provisioned cluster.
Example 12.2. Sample DNS zone database for reverse records
$TTL 1W
@ IN SOA ns1.example.com. root (
2019070700 ; serial
3H ; refresh (3 hours)
30M ; retry (30 minutes)
2W ; expiry (2 weeks)
1W ) ; minimum (1 week)
IN NS ns1.example.com.
;
5.1.168.192.in-addr.arpa. IN PTR api.ocp4.example.com. 1
5.1.168.192.in-addr.arpa. IN PTR api-int.ocp4.example.com. 2
;
96.1.168.192.in-addr.arpa. IN PTR bootstrap.ocp4.example.com. 3
;
97.1.168.192.in-addr.arpa. IN PTR control-plane0.ocp4.example.com. 4
;
11.1.168.192.in-addr.arpa. IN PTR compute0.ocp4.example.com. 7
7.1.168.192.in-addr.arpa. IN PTR compute1.ocp4.example.com. 8
;
;EOF
Provides reverse DNS resolution for the Kubernetes API. The PTR record refers to the record
2029
1 Provides reverse DNS resolution for the Kubernetes API. The PTR record refers to the record
name of the API load balancer.
2 Provides reverse DNS resolution for the Kubernetes API. The PTR record refers to the record
name of the API load balancer and is used for internal cluster communications.
3 Provides reverse DNS resolution for the bootstrap machine.
4 5 6 Provides reverse DNS resolution for the control plane machines.
7 8 Provides reverse DNS resolution for the compute machines.
NOTE
A PTR record is not required for the OpenShift Container Platform application wildcard.
Validating DNS resolution for user-provisioned infrastructure
12.2.3.7. Load balancing requirements for user-provisioned infrastructure
Before you install OpenShift Container Platform, you must provision the API and application Ingress
load balancing infrastructure. In production scenarios, you can deploy the API and application Ingress
load balancers separately so that you can scale the load balancer infrastructure for each in isolation.
NOTE
If you want to deploy the API and application Ingress load balancers with a Red Hat
Enterprise Linux (RHEL) instance, you must purchase the RHEL subscription separately.
The load balancing infrastructure must meet the following requirements:
1. API load balancer: Provides a common endpoint for users, both human and machine, to interact
with and configure the platform. Configure the following conditions:
Layer 4 load balancing only. This can be referred to as Raw TCP or SSL Passthrough mode.
A stateless load balancing algorithm. The options vary based on the load balancer
implementation.
IMPORTANT
Do not configure session persistence for an API load balancer. Configuring

session persistence for a Kubernetes API server might cause performance issues
from excess application traffic for your OpenShift Container Platform cluster
and the Kubernetes API that runs inside the cluster.
Configure the following ports on both the front and back of the load balancers:
Table 12.7. API load balancer
2030
Port Back-end machines (pool members) Internal External Description
6443 Bootstrap and control plane. You X X Kubernetes

remove the bootstrap machine from API server
the load balancer after the bootstrap
machine initializes the cluster control
plane. You must configure the
/readyz endpoint for the API server
health check probe.
22623 Bootstrap and control plane. You X Machine

remove the bootstrap machine from config
the load balancer after the bootstrap server
machine initializes the cluster control
plane.
NOTE
The load balancer must be configured to take a maximum of 30 seconds from

the time the API server turns off the /readyz endpoint to the removal of the API
server instance from the pool. Within the time frame after /readyz returns an
error or becomes healthy, the endpoint must have been removed or added.
Probing every 5 or 10 seconds, with two successful requests to become healthy
and three to become unhealthy, are well-tested values.
2. Application Ingress load balancer: Provides an ingress point for application traffic flowing in
from outside the cluster. A working configuration for the Ingress router is required for an
Configure the following conditions:
Layer 4 load balancing only. This can be referred to as Raw TCP or SSL Passthrough mode.
A connection-based or session-based persistence is recommended, based on the options

available and types of applications that will be hosted on the platform.
TIP
If the true IP address of the client can be seen by the application Ingress load balancer, enabling
source IP-based session persistence can improve performance for applications that use end-
to-end TLS encryption.
Configure the following ports on both the front and back of the load balancers:
Table 12.8. Application Ingress load balancer
443 The machines that run the Ingress X X HTTPS

Controller pods, compute, or worker, traffic
by default.
2031
80 The machines that run the Ingress X X HTTP

Controller pods, compute, or worker, traffic
by default.
NOTE
If you are deploying a three-node cluster with zero compute nodes, the Ingress
Controller pods run on the control plane nodes. In three-node cluster
deployments, you must configure your application Ingress load balancer to route
HTTP and HTTPS traffic to the control plane nodes.
12.2.3.7.1. Example load balancer configuration for user-provisioned clusters
This section provides an example API and application Ingress load balancer configuration that meets the
load balancing requirements for user-provisioned clusters. The sample is an /etc/haproxy/haproxy.cfg
configuration for an HAProxy load balancer. The example is not meant to provide advice for choosing
one load balancing solution over another.
In the example, the same load balancer is used for the Kubernetes API and application ingress traffic. In
production scenarios, you can deploy the API and application ingress load balancers separately so that
you can scale the load balancer infrastructure for each in isolation.
NOTE
If you are using HAProxy as a load balancer and SELinux is set to enforcing, you must
ensure that the HAProxy service can bind to the configured TCP port by running
setsebool -P haproxy_connect_any=1.
Example 12.3. Sample API and application Ingress load balancer configuration
global
log 127.0.0.1 local2
pidfile /var/run/haproxy.pid
maxconn 4000
daemon
defaults
mode http
log global
option dontlognull
option http-server-close
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
2032
maxconn 3000
listen api-server-6443 1
bind *:6443
mode tcp
option httpchk GET /readyz HTTP/1.0
option log-health-checks
balance roundrobin
server bootstrap bootstrap.ocp4.example.com:6443 verify none check check-ssl inter 10s fall 2
rise 3 backup 2
server master0 master0.ocp4.example.com:6443 weight 1 verify none check check-ssl inter 10s
fall 2 rise 3
fall 2 rise 3
fall 2 rise 3
listen machine-config-server-22623 3
bind *:22623
mode tcp
server bootstrap bootstrap.ocp4.example.com:22623 check inter 1s backup 4
server master0 master0.ocp4.example.com:22623 check inter 1s
listen ingress-router-443 5
bind *:443
mode tcp
balance source
server compute0 compute0.ocp4.example.com:443 check inter 1s
listen ingress-router-80 6
bind *:80
mode tcp
balance source
1 Port 6443 handles the Kubernetes API traffic and points to the control plane machines.
2 4 The bootstrap entries must be in place before the OpenShift Container Platform cluster
installation and they must be removed after the bootstrap process is complete.
3 Port 22623 handles the machine config server traffic and points to the control plane machines.
5 Port 443 handles the HTTPS traffic and points to the machines that run the Ingress Controller
pods. The Ingress Controller pods run on the compute machines by default.
6 Port 80 handles the HTTP traffic and points to the machines that run the Ingress Controller
pods. The Ingress Controller pods run on the compute machines by default.
NOTE
If you are deploying a three-node cluster with zero compute nodes, the Ingress
Controller pods run on the control plane nodes. In three-node cluster
deployments, you must configure your application Ingress load balancer to route
HTTP and HTTPS traffic to the control plane nodes.
2033
TIP
If you are using HAProxy as a load balancer, you can check that the haproxy process is listening on ports
6443, 22623, 443, and 80 by running netstat -nltupe on the HAProxy node.
12.2.4. Preparing the user-provisioned infrastructure

Before you install OpenShift Container Platform on user-provisioned infrastructure, you must prepare
the underlying infrastructure.
This section provides details about the high-level steps required to set up your cluster infrastructure in
preparation for an OpenShift Container Platform installation. This includes configuring IP networking
and network connectivity for your cluster nodes, enabling the required ports through your firewall, and
setting up the required DNS and load balancing infrastructure.
After preparation, your cluster infrastructure must meet the requirements outlined in the Requirements
for a cluster with user-provisioned infrastructure section.
Prerequisites
You have reviewed the OpenShift Container Platform 4.x Tested Integrations page.
You have reviewed the infrastructure requirements detailed in the Requirements for a cluster
with user-provisioned infrastructure section.
Procedure
1. If you are using DHCP to provide the IP networking configuration to your cluster nodes,
configure your DHCP service.
a. Add persistent IP addresses for the nodes to your DHCP server configuration. In your
configuration, match the MAC address of the relevant network interface to the intended IP
address for each node.
b. When you use DHCP to configure IP addressing for the cluster machines, the machines also
obtain the DNS server information through DHCP. Define the persistent DNS server
address that is used by the cluster nodes through your DHCP server configuration.
NOTE
If you are not using a DHCP service, you must provide the IP networking
configuration and the address of the DNS server to the nodes at RHCOS
install time. These can be passed as boot arguments if you are installing from
an ISO image. See the Installing RHCOS and starting the OpenShift
Container Platform bootstrap process section for more information about
static IP provisioning and advanced networking options.
c. Define the hostnames of your cluster nodes in your DHCP server configuration. See the
Setting the cluster node hostnames through DHCP section for details about hostname
considerations.
NOTE
2034
NOTE
If you are not using a DHCP service, the cluster nodes obtain their hostname
through a reverse DNS lookup.
2. Ensure that your network infrastructure provides the required network connectivity between
the cluster components. See the Networking requirements for user-provisioned infrastructure
section for details about the requirements.
3. Configure your firewall to enable the ports required for the OpenShift Container Platform
cluster components to communicate. See Networking requirements for user-provisioned
infrastructure section for details about the ports that are required.
IMPORTANT
By default, port 1936 is accessible for an OpenShift Container Platform cluster,

because each control plane node needs access to this port.
Avoid using the Ingress load balancer to expose this port, because doing so
might result in the exposure of sensitive information, such as statistics and
metrics, related to Ingress Controllers.
4. Setup the required DNS infrastructure for your cluster.
a. Configure DNS name resolution for the Kubernetes API, the application wildcard, the
bootstrap machine, the control plane machines, and the compute machines.
b. Configure reverse DNS resolution for the Kubernetes API, the bootstrap machine, the
control plane machines, and the compute machines.
See the User-provisioned DNS requirements section for more information about the
OpenShift Container Platform DNS requirements.
5. Validate your DNS configuration.
a. From your installation node, run DNS lookups against the record names of the Kubernetes
API, the wildcard routes, and the cluster nodes. Validate that the IP addresses in the
responses correspond to the correct components.
b. From your installation node, run reverse DNS lookups against the IP addresses of the load
balancer and the cluster nodes. Validate that the record names in the responses correspond
to the correct components.
See the Validating DNS resolution for user-provisioned infrastructure section for detailed
DNS validation steps.
6. Provision the required API and application ingress load balancing infrastructure. See the Load
balancing requirements for user-provisioned infrastructure section for more information about
the requirements.
NOTE
Some load balancing solutions require the DNS name resolution for the cluster nodes to
be in place before the load balancing is initialized.
2035
Requirements for a cluster with user-provisioned infrastructure
Installing RHCOS and starting the OpenShift Container Platform bootstrap process
Setting the cluster node hostnames through DHCP
Advanced RHCOS installation configuration
Networking requirements for user-provisioned infrastructure
User-provisioned DNS requirements
Validating DNS resolution for user-provisioned infrastructure
Load balancing requirements for user-provisioned infrastructure
12.2.5. Validating DNS resolution for user-provisioned infrastructure

You can validate your DNS configuration before installing OpenShift Container Platform on user-
IMPORTANT
The validation steps detailed in this section must succeed before you install your cluster.
Prerequisites
You have configured the required DNS records for your user-provisioned infrastructure.
Procedure
1. From your installation node, run DNS lookups against the record names of the Kubernetes API,
the wildcard routes, and the cluster nodes. Validate that the IP addresses contained in the
responses correspond to the correct components.
a. Perform a lookup against the Kubernetes API record name. Check that the result points to
the IP address of the API load balancer:
$ dig +noall +answer @<nameserver_ip> api.<cluster_name>.<base_domain> 1
1 Replace <nameserver_ip> with the IP address of the nameserver, <cluster_name>

with your cluster name, and <base_domain> with your base domain name.
Example output
api.ocp4.example.com. 604800 IN A 192.168.1.5
b. Perform a lookup against the Kubernetes internal API record name. Check that the result
points to the IP address of the API load balancer:
$ dig +noall +answer @<nameserver_ip> api-int.<cluster_name>.<base_domain>
Example output
2036
api-int.ocp4.example.com. 604800 IN A 192.168.1.5
c. Test an example *.apps.<cluster_name>.<base_domain> DNS wildcard lookup. All of the

application wildcard lookups must resolve to the IP address of the application ingress load
balancer:
$ dig +noall +answer @<nameserver_ip> random.apps.<cluster_name>.<base_domain>
Example output
random.apps.ocp4.example.com. 604800 IN A 192.168.1.5
NOTE
In the example outputs, the same load balancer is used for the Kubernetes
API and application ingress traffic. In production scenarios, you can deploy
the API and application ingress load balancers separately so that you can
scale the load balancer infrastructure for each in isolation.
You can replace random with another wildcard value. For example, you can query the route
to the OpenShift Container Platform console:
$ dig +noall +answer @<nameserver_ip> console-openshift-console.apps.

<cluster_name>.<base_domain>
Example output
console-openshift-console.apps.ocp4.example.com. 604800 IN A 192.168.1.5
d. Run a lookup against the bootstrap DNS record name. Check that the result points to the IP
address of the bootstrap node:
$ dig +noall +answer @<nameserver_ip> bootstrap.<cluster_name>.<base_domain>
Example output
bootstrap.ocp4.example.com. 604800 IN A 192.168.1.96
e. Use this method to perform lookups against the DNS record names for the control plane
and compute nodes. Check that the results correspond to the IP addresses of each node.
2. From your installation node, run reverse DNS lookups against the IP addresses of the load
balancer and the cluster nodes. Validate that the record names contained in the responses
correspond to the correct components.
a. Perform a reverse lookup against the IP address of the API load balancer. Check that the
response includes the record names for the Kubernetes API and the Kubernetes internal
API:
$ dig +noall +answer @<nameserver_ip> -x 192.168.1.5
2037
Example output
5.1.168.192.in-addr.arpa. 604800 IN PTR api-int.ocp4.example.com. 1

5.1.168.192.in-addr.arpa. 604800 IN PTR api.ocp4.example.com. 2
1 Provides the record name for the Kubernetes internal API.
2 Provides the record name for the Kubernetes API.
NOTE
A PTR record is not required for the OpenShift Container Platform

application wildcard. No validation step is needed for reverse DNS resolution
against the IP address of the application ingress load balancer.
b. Perform a reverse lookup against the IP address of the bootstrap node. Check that the
result points to the DNS record name of the bootstrap node:
$ dig +noall +answer @<nameserver_ip> -x 192.168.1.96
Example output
96.1.168.192.in-addr.arpa. 604800 IN PTR bootstrap.ocp4.example.com.
c. Use this method to perform reverse lookups against the IP addresses for the control plane
and compute nodes. Check that the results correspond to the DNS record names of each
node.
User-provisioned DNS requirements
Load balancing requirements for user-provisioned infrastructure

authentication.
user.
IMPORTANT
2038
IMPORTANT
NOTE
Procedure
NOTE
ecdsa algorithm.
NOTE

task:
Example output
2039
Example output
Agent pid 31874
NOTE

Example output
Next steps
Verifying node health

for installation.
Prerequisites
Procedure
IMPORTANT
2040
IMPORTANT
IMPORTANT
components.

IMPORTANT

Procedure
Portal.
$ tar xvf <file>
2041
$ echo $PATH
Verification
$ oc <command>

Procedure
Portal.

C:\> path
Verification
C:\> oc <command>

Procedure
Portal.
NOTE
2042

$ echo $PATH
Verification
$ oc <command>

Prerequisites
recovery.
for your cluster.
Procedure
IMPORTANT
NOTE
IMPORTANT
2043
Installation configuration parameters for bare metal
12.2.9.1. Sample install-config.yaml file for bare metal
apiVersion: v1
compute: 2
name: worker
replicas: 0 4
controlPlane: 5
name: master
replicas: 3 7
metadata:
name: test 8
networking:
clusterNetwork:
- cidr: 10.128.0.0/14 9
hostPrefix: 23 10
serviceNetwork: 12
- 172.30.0.0/16
platform:
none: {} 13
fips: false 14
sshKey: 'ssh-ed25519 AAAA...' 16
1 The base domain of the cluster. All DNS records must be sub-domains of this base and include the
cluster name.
3 6 Specifies whether to enable or disable simultaneous multithreading (SMT), or hyperthreading. By

default, SMT is enabled to increase the performance of the cores in your machines. You can
disable it by setting the parameter value to Disabled. If you disable SMT, you must disable it in all
cluster machines; this includes both control plane and compute machines.
NOTE
Simultaneous multithreading (SMT) is enabled by default. If SMT is not enabled in

your BIOS settings, the hyperthreading parameter has no effect.
IMPORTANT
2044
IMPORTANT
If you disable hyperthreading, whether in the BIOS or in the install-config.yaml file,

ensure that your capacity planning accounts for the dramatically decreased machine
performance.
4 You must set this value to 0 when you install OpenShift Container Platform on user-provisioned
infrastructure. In installer-provisioned installations, the parameter controls the number of compute
machines that the cluster creates and manages for you. In user-provisioned installations, you must
manually deploy the compute machines before you finish installing the cluster.
NOTE
If you are installing a three-node cluster, do not deploy any compute machines when
you install the Red Hat Enterprise Linux CoreOS (RHCOS) machines.
7 The number of control plane machines that you add to the cluster. Because the cluster uses these
values as the number of etcd endpoints in the cluster, the value must match the number of control
plane machines that you deploy.
8 The cluster name that you specified in your DNS records.
9 A block of IP addresses from which pod IP addresses are allocated. This block must not overlap
with existing physical networks. These IP addresses are used for the pod network. If you need to
access the pods from an external network, you must configure load balancers and routers to
manage the traffic.
NOTE
Class E CIDR range is reserved for a future use. To use the Class E CIDR range, you
must ensure your networking environment accepts the IP addresses within the Class
E CIDR range.
10 The subnet prefix length to assign to each individual node. For example, if hostPrefix is set to 23,
then each node is assigned a /23 subnet out of the given cidr, which allows for 510 (2^(32 - 23) - 2)
pod IP addresses. If you are required to provide access to nodes from an external network,
configure load balancers and routers to manage the traffic.
value.
12 The IP address pool to use for service IP addresses. You can enter only one IP address pool. This
block must not overlap with existing physical networks. If you need to access the services from an
external network, configure load balancers and routers to manage the traffic.
13 You must set the platform to none. You cannot provide additional platform configuration variables
for your platform.
IMPORTANT
2045
IMPORTANT
Clusters that are installed with the platform type none are unable to use some
features, such as managing compute machines with the Machine API. This limitation
applies even if the compute machines that are attached to the cluster are installed
on a platform that would normally support the feature. This parameter cannot be
changed after installation.
IMPORTANT
15 The pull secret from Red Hat OpenShift Cluster Manager . This pull secret allows you to
authenticate with the services that are provided by the included authorities, including Quay.io,
which serves the container images for OpenShift Container Platform components.
16 The SSH public key for the core user in Red Hat Enterprise Linux CoreOS (RHCOS).
NOTE
process uses.
See Load balancing requirements for user-provisioned infrastructure for more information on
the API and application ingress load balancing requirements.
See Enabling cluster capabilities for more information on enabling cluster capabilities that were
disabled prior to installation.
See Optional cluster capabilities in OpenShift Container Platform 4.15 for more information
about the features provided by each capability.
NOTE
2046
NOTE
For bare metal installations, if you do not assign node IP addresses from the range that is
specified in the networking.machineNetwork[].cidr field in the install-config.yaml file,
you must include them in the proxy.noProxy field.
Prerequisites
NOTE
(169.254.169.254).
Procedure
apiVersion: v1
proxy:
must be http.

destinations.
2047
NOTE
NOTE
NOTE
created.
12.2.9.3. Configuring a three-node cluster
Optionally, you can deploy zero compute machines in a bare metal cluster that consists of three control
plane machines only. This provides smaller, more resource efficient clusters for cluster administrators
and developers to use for testing, development, and production.
In three-node OpenShift Container Platform environments, the three control plane machines are
schedulable, which means that your application workloads are scheduled to run on them.
Prerequisites
Procedure
Ensure that the number of compute replicas is set to 0 in your install-config.yaml file, as shown
in the following compute stanza:
compute:
2048
- name: worker
platform: {}
replicas: 0
NOTE
You must set the value of the replicas parameter for the compute machines to 0
when you install OpenShift Container Platform on user-provisioned
infrastructure, regardless of the number of compute machines you are deploying.
In installer-provisioned installations, the parameter controls the number of
compute machines that the cluster creates and manages for you. This does not
apply to user-provisioned installations, where the compute machines are
deployed manually.
For three-node cluster installations, follow these next steps:
If you are deploying a three-node cluster with zero compute nodes, the Ingress Controller pods
run on the control plane nodes. In three-node cluster deployments, you must configure your
application ingress load balancer to route HTTP and HTTPS traffic to the control plane nodes.
See the Load balancing requirements for user-provisioned infrastructure section for more
information.
When you create the Kubernetes manifest files in the following procedure, ensure that the
mastersSchedulable parameter in the <installation_directory>/manifests/cluster-
scheduler-02-config.yml file is set to true. This enables your application workloads to run on
Do not deploy any compute nodes when you create the Red Hat Enterprise Linux CoreOS
(RHCOS) machines.
12.2.10. Creating the Kubernetes manifest and Ignition config files

machines.
IMPORTANT
2049
Prerequisites
Procedure

WARNING
IMPORTANT

machines:
2050
.
├── auth
See Recovering from expired control plane certificates for more information about recovering
kubelet certificates.
12.2.11. Installing RHCOS and starting the OpenShift Container Platform bootstrap
process
To install OpenShift Container Platform on bare metal infrastructure that you provision, you must install
Red Hat Enterprise Linux CoreOS (RHCOS) on the machines. When you install RHCOS, you must
provide the Ignition config file that was generated by the OpenShift Container Platform installation
program for the type of machine you are installing. If you have configured suitable networking, DNS, and
load balancing infrastructure, the OpenShift Container Platform bootstrap process begins automatically
after the RHCOS machines have rebooted.
To install RHCOS on the machines, follow either the steps to use an ISO image or network PXE booting.
NOTE
The compute node deployment steps included in this installation document are RHCOS-
specific. If you choose instead to deploy RHEL-based compute nodes, you take
responsibility for all operating system life cycle management and maintenance, including
performing system updates, applying patches, and completing all other required tasks.
Only RHEL 8 compute machines are supported.
You can configure RHCOS during ISO and PXE installations by using the following methods:
Kernel arguments: You can use kernel arguments to provide installation-specific information.
For example, you can specify the locations of the RHCOS installation files that you uploaded to
your HTTP server and the location of the Ignition config file for the type of node you are
installing. For a PXE installation, you can use the APPEND parameter to pass the arguments to
the kernel of the live installer. For an ISO installation, you can interrupt the live installation boot
process to add the kernel arguments. In both installation cases, you can use special
coreos.inst.* arguments to direct the live installer, as well as standard installation boot
arguments for turning standard kernel services on or off.
Ignition configs: OpenShift Container Platform Ignition config files (*.ign) are specific to the
type of node you are installing. You pass the location of a bootstrap, control plane, or compute
node Ignition config file during the RHCOS installation so that it takes effect on first boot. In
special cases, you can create a separate, limited Ignition config to pass to the live system. That
Ignition config could do a certain set of tasks, such as reporting success to a provisioning system
after completing installation. This special Ignition config is consumed by the coreos-installer to
be applied on first boot of the installed system. Do not provide the standard control plane and
compute node Ignition configs to the live ISO directly.
coreos-installer: You can boot the live ISO installer to a shell prompt, which allows you to
2051
coreos-installer: You can boot the live ISO installer to a shell prompt, which allows you to
prepare the permanent system in a variety of ways before first boot. In particular, you can run
the coreos-installer command to identify various artifacts to include, work with disk partitions,
and set up networking. In some cases, you can configure features on the live system and copy
them to the installed system.
Whether to use an ISO or PXE install depends on your situation. A PXE install requires an available DHCP
service and more preparation, but can make the installation process more automated. An ISO install is a
more manual process and can be inconvenient if you are setting up more than a few machines.
NOTE
As of OpenShift Container Platform 4.6, the RHCOS ISO and other installation artifacts
provide support for installation on disks with 4K sectors.
12.2.11.1. Installing RHCOS by using an ISO image
You can use an ISO image to install RHCOS on the machines.
Prerequisites
You have created the Ignition config files for your cluster.
You have configured suitable network, DNS and load balancing infrastructure.
You have an HTTP server that can be accessed from your computer, and from the machines
that you create.
You have reviewed the Advanced RHCOS installation configuration section for different ways to
configure features, such as networking and disk partitioning.
Procedure
1. Obtain the SHA512 digest for each of your Ignition config files. For example, you can use the
following on a system running Linux to get the SHA512 digest for your bootstrap.ign Ignition
config file:
$ sha512sum <installation_directory>/bootstrap.ign
The digests are provided to the coreos-installer in a later step to validate the authenticity of
the Ignition config files on the cluster nodes.
2. Upload the bootstrap, control plane, and compute node Ignition config files that the installation
program created to your HTTP server. Note the URLs of these files.
IMPORTANT
You can add or change configuration settings in your Ignition configs before
saving them to your HTTP server. If you plan to add more compute machines to
your cluster after you finish installation, do not delete these files.
3. From the installation host, validate that the Ignition config files are available on the URLs. The
following example gets the Ignition config file for the bootstrap node:
2052
$ curl -k http://<HTTP_server>/bootstrap.ign 1
Example output
% Total % Received % Xferd Average Speed Time Time Time Current

Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0{"ignition":
{"version":"3.2.0"},"passwd":{"users":[{"name":"core","sshAuthorizedKeys":["ssh-rsa...
Replace bootstrap.ign with master.ign or worker.ign in the command to validate that the
Ignition config files for the control plane and compute nodes are also available.
4. Although it is possible to obtain the RHCOS images that are required for your preferred method
of installing operating system instances from the RHCOS image mirror page, the recommended
way to obtain the correct version of your RHCOS images are from the output of openshift-
install command:
$ openshift-install coreos print-stream-json | grep '\.iso[^.]'
Example output
"location": "<url>/art/storage/releases/rhcos-4.15-aarch64/<release>/aarch64/rhcos-
<release>-live.aarch64.iso",
"location": "<url>/art/storage/releases/rhcos-4.15-ppc64le/<release>/ppc64le/rhcos-
<release>-live.ppc64le.iso",
"location": "<url>/art/storage/releases/rhcos-4.15-s390x/<release>/s390x/rhcos-<release>-
live.s390x.iso",
"location": "<url>/art/storage/releases/rhcos-4.15/<release>/x86_64/rhcos-<release>-
live.x86_64.iso",
IMPORTANT
Platform. You must download images with the highest version that is less than or
equal to the OpenShift Container Platform version that you install. Use the image
versions that match your OpenShift Container Platform version if they are
available. Use only ISO images for this procedure. RHCOS qcow2 images are not
supported for this installation type.
ISO file names resemble the following example:
rhcos-<version>-live.<architecture>.iso
5. Use the ISO to start the RHCOS installation. Use one of the following installation options:
Burn the ISO image to a disk and boot it directly.
Use ISO redirection by using a lights-out management (LOM) interface.
6. Boot the RHCOS ISO image without specifying any options or interrupting the live boot
sequence. Wait for the installer to boot into a shell prompt in the RHCOS live environment.
NOTE
2053
NOTE
It is possible to interrupt the RHCOS installation boot process to add kernel

arguments. However, for this ISO procedure you should use the coreos-installer
command as outlined in the following steps, instead of adding kernel arguments.
7. Run the coreos-installer command and specify the options that meet your installation
requirements. At a minimum, you must specify the URL that points to the Ignition config file for
the node type, and the device that you are installing to:
$ sudo coreos-installer install --ignition-url=http://<HTTP_server>/<node_type>.ign <device>

--ignition-hash=sha512-<digest> 1 2
1 1 You must run the coreos-installer command by using sudo, because the core user does
not have the required root privileges to perform the installation.
2 The --ignition-hash option is required when the Ignition config file is obtained through an
HTTP URL to validate the authenticity of the Ignition config file on the cluster node.
<digest> is the Ignition config file SHA512 digest obtained in a preceding step.
NOTE
If you want to provide your Ignition config files through an HTTPS server that
uses TLS, you can add the internal certificate authority (CA) to the system trust
store before running coreos-installer.
The following example initializes a bootstrap node installation to the /dev/sda device. The
Ignition config file for the bootstrap node is obtained from an HTTP web server with the IP
address 192.168.1.2:
$ sudo coreos-installer install --ignition-

url=http://192.168.1.2:80/installation_directory/bootstrap.ign /dev/sda --ignition-hash=sha512-
a5a2d43879223273c9b60af66b44202a1d1248fc01cf156c46d4a79f552b6bad47bc8cc78ddf011
6e80c59d2ea9e32ba53bc807afbca581aa059311def2c3e3b
8. Monitor the progress of the RHCOS installation on the console of the machine.
IMPORTANT
Be sure that the installation is successful on each node before commencing with
the OpenShift Container Platform installation. Observing the installation process
can also help to determine the cause of RHCOS installation issues that might
arise.
9. After RHCOS installs, you must reboot the system. During the system reboot, it applies the
Ignition config file that you specified.
10. Check the console output to verify that Ignition ran.
Example command
2054
Ignition: ran on 2022/03/14 14:48:33 UTC (this boot)

Ignition: user-provided config was applied
11. Continue to create the other machines for your cluster.
IMPORTANT
You must create the bootstrap and control plane machines at this time. If the
control plane machines are not made schedulable, also create at least two
compute machines before you install OpenShift Container Platform.
If the required network, DNS, and load balancer infrastructure are in place, the OpenShift
Container Platform bootstrap process begins automatically after the RHCOS nodes have
rebooted.
NOTE
RHCOS nodes do not include a default password for the core user. You can
access the nodes by running ssh core@<node>.<cluster_name>.
<base_domain> as a user with access to the SSH private key that is paired to
the public key that you specified in your install_config.yaml file. OpenShift
Container Platform 4 cluster nodes running RHCOS are immutable and rely on
Operators to apply cluster changes. Accessing cluster nodes by using SSH is not
recommended. However, when investigating installation issues, if the OpenShift
Container Platform API is not available, or the kubelet is not properly functioning
on a target node, SSH access might be required for debugging or disaster
recovery.
12.2.11.2. Installing RHCOS by using PXE or iPXE booting
You can use PXE or iPXE booting to install RHCOS on the machines.
Prerequisites
You have created the Ignition config files for your cluster.
You have configured suitable network, DNS and load balancing infrastructure.
You have configured suitable PXE or iPXE infrastructure.
You have an HTTP server that can be accessed from your computer, and from the machines
that you create.
You have reviewed the Advanced RHCOS installation configuration section for different ways to
configure features, such as networking and disk partitioning.
Procedure
1. Upload the bootstrap, control plane, and compute node Ignition config files that the installation
program created to your HTTP server. Note the URLs of these files.
IMPORTANT
2055
IMPORTANT
You can add or change configuration settings in your Ignition configs before
saving them to your HTTP server. If you plan to add more compute machines to
your cluster after you finish installation, do not delete these files.
2. From the installation host, validate that the Ignition config files are available on the URLs. The
following example gets the Ignition config file for the bootstrap node:
$ curl -k http://<HTTP_server>/bootstrap.ign 1
Example output
% Total % Received % Xferd Average Speed Time Time Time Current

Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0{"ignition":
{"version":"3.2.0"},"passwd":{"users":[{"name":"core","sshAuthorizedKeys":["ssh-rsa...
Replace bootstrap.ign with master.ign or worker.ign in the command to validate that the
Ignition config files for the control plane and compute nodes are also available.
3. Although it is possible to obtain the RHCOS kernel, initramfs and rootfs files that are required
for your preferred method of installing operating system instances from the RHCOS image
mirror page, the recommended way to obtain the correct version of your RHCOS files are from
the output of openshift-install command:
$ openshift-install coreos print-stream-json | grep -Eo '"https.*(kernel-|initramfs.|rootfs.)\w+

(\.img)?"'
Example output
"<url>/art/storage/releases/rhcos-4.15-aarch64/<release>/aarch64/rhcos-<release>-live-
kernel-aarch64"
initramfs.aarch64.img"
rootfs.aarch64.img"
"<url>/art/storage/releases/rhcos-4.15-ppc64le/49.84.202110081256-0/ppc64le/rhcos-
<release>-live-kernel-ppc64le"
"<url>/art/storage/releases/rhcos-4.15-ppc64le/<release>/ppc64le/rhcos-<release>-live-
initramfs.ppc64le.img"
"<url>/art/storage/releases/rhcos-4.15-ppc64le/<release>/ppc64le/rhcos-<release>-live-
rootfs.ppc64le.img"
"<url>/art/storage/releases/rhcos-4.15-s390x/<release>/s390x/rhcos-<release>-live-kernel-
s390x"
"<url>/art/storage/releases/rhcos-4.15-s390x/<release>/s390x/rhcos-<release>-live-
initramfs.s390x.img"
"<url>/art/storage/releases/rhcos-4.15-s390x/<release>/s390x/rhcos-<release>-live-
rootfs.s390x.img"
"<url>/art/storage/releases/rhcos-4.15/<release>/x86_64/rhcos-<release>-live-kernel-
x86_64"
"<url>/art/storage/releases/rhcos-4.15/<release>/x86_64/rhcos-<release>-live-
2056
initramfs.x86_64.img"
"<url>/art/storage/releases/rhcos-4.15/<release>/x86_64/rhcos-<release>-live-
rootfs.x86_64.img"
IMPORTANT
The RHCOS artifacts might not change with every release of OpenShift
Container Platform. You must download images with the highest version that is
less than or equal to the OpenShift Container Platform version that you install.
Only use the appropriate kernel, initramfs, and rootfs artifacts described below
for this procedure. RHCOS QCOW2 images are not supported for this installation
type.
The file names contain the OpenShift Container Platform version number. They resemble the
following examples:
kernel: rhcos-<version>-live-kernel-<architecture>
initramfs: rhcos-<version>-live-initramfs.<architecture>.img
rootfs: rhcos-<version>-live-rootfs.<architecture>.img
4. Upload the rootfs, kernel, and initramfs files to your HTTP server.
IMPORTANT
If you plan to add more compute machines to your cluster after you finish
installation, do not delete these files.
5. Configure the network boot infrastructure so that the machines boot from their local disks after
RHCOS is installed on them.
6. Configure PXE or iPXE installation for the RHCOS images and begin the installation.
Modify one of the following example menu entries for your environment and verify that the
image and Ignition files are properly accessible:
For PXE (x86_64):
DEFAULT pxeboot
TIMEOUT 20
PROMPT 0
LABEL pxeboot
KERNEL http://<HTTP_server>/rhcos-<version>-live-kernel-<architecture> 1
APPEND initrd=http://<HTTP_server>/rhcos-<version>-live-initramfs.
<architecture>.img coreos.live.rootfs_url=http://<HTTP_server>/rhcos-<version>-live-
rootfs.<architecture>.img coreos.inst.install_dev=/dev/sda
coreos.inst.ignition_url=http://<HTTP_server>/bootstrap.ign 2 3
1 1 Specify the location of the live kernel file that you uploaded to your HTTP server. The
URL must be HTTP, TFTP, or FTP; HTTPS and NFS are not supported.
2 If you use multiple NICs, specify a single interface in the ip option. For example, to use
DHCP on a NIC that is named eno1, set ip=eno1:dhcp.
2057
3 Specify the locations of the RHCOS files that you uploaded to your HTTP server. The
initrd parameter value is the location of the initramfs file, the coreos.live.rootfs_url
NOTE
This configuration does not enable serial console access on machines with a
graphical console. To configure a different console, add one or more
console= arguments to the APPEND line. For example, add console=tty0
console=ttyS0 to set the first PC serial port as the primary console and the
graphical console as a secondary console. For more information, see How
does one set up a serial terminal and/or console in Red Hat Enterprise Linux?
and "Enabling the serial console for PXE and ISO installation" in the
"Advanced RHCOS installation configuration" section.
For iPXE (x86_64 + aarch64 ):
kernel http://<HTTP_server>/rhcos-<version>-live-kernel-<architecture> initrd=main

coreos.live.rootfs_url=http://<HTTP_server>/rhcos-<version>-live-rootfs.
<architecture>.img coreos.inst.install_dev=/dev/sda
initrd --name main http://<HTTP_server>/rhcos-<version>-live-initramfs.
<architecture>.img 3
boot
1 Specify the locations of the RHCOS files that you uploaded to your HTTP server. The
kernel parameter value is the location of the kernel file, the initrd=main argument is
needed for booting on UEFI systems, the coreos.live.rootfs_url parameter value is
the location of the rootfs file, and the coreos.inst.ignition_url parameter value is the
location of the bootstrap Ignition config file.
3 Specify the location of the initramfs file that you uploaded to your HTTP server.
NOTE
This configuration does not enable serial console access on machines with a
graphical console. To configure a different console, add one or more
console= arguments to the kernel line. For example, add console=tty0
console=ttyS0 to set the first PC serial port as the primary console and the
graphical console as a secondary console. For more information, see How
does one set up a serial terminal and/or console in Red Hat Enterprise Linux?
and "Enabling the serial console for PXE and ISO installation" in the
"Advanced RHCOS installation configuration" section.
NOTE
To network boot the CoreOS kernel on aarch64 architecture, you need to

use a version of iPXE build with the IMAGE_GZIP option enabled. See
IMAGE_GZIP option in iPXE .
2058
For PXE (with UEFI and Grub as second stage) on aarch64:
menuentry 'Install CoreOS' {

linux rhcos-<version>-live-kernel-<architecture>
coreos.live.rootfs_url=http://<HTTP_server>/rhcos-<version>-live-rootfs.
<architecture>.img coreos.inst.install_dev=/dev/sda
initrd rhcos-<version>-live-initramfs.<architecture>.img 3
}
1 Specify the locations of the RHCOS files that you uploaded to your HTTP/TFTP
server. The kernel parameter value is the location of the kernel file on your TFTP
server. The coreos.live.rootfs_url parameter value is the location of the rootfs file,
and the coreos.inst.ignition_url parameter value is the location of the bootstrap
Ignition config file on your HTTP Server.
3 Specify the location of the initramfs file that you uploaded to your TFTP server.
7. Monitor the progress of the RHCOS installation on the console of the machine.
IMPORTANT
Be sure that the installation is successful on each node before commencing with
the OpenShift Container Platform installation. Observing the installation process
can also help to determine the cause of RHCOS installation issues that might
arise.
8. After RHCOS installs, the system reboots. During reboot, the system applies the Ignition config
file that you specified.
9. Check the console output to verify that Ignition ran.
Example command
Ignition: ran on 2022/03/14 14:48:33 UTC (this boot)

Ignition: user-provided config was applied
10. Continue to create the machines for your cluster.
IMPORTANT
You must create the bootstrap and control plane machines at this time. If the
control plane machines are not made schedulable, also create at least two
compute machines before you install the cluster.
If the required network, DNS, and load balancer infrastructure are in place, the OpenShift
Container Platform bootstrap process begins automatically after the RHCOS nodes have
rebooted.
NOTE
2059
NOTE
RHCOS nodes do not include a default password for the core user. You can
access the nodes by running ssh core@<node>.<cluster_name>.
<base_domain> as a user with access to the SSH private key that is paired to
the public key that you specified in your install_config.yaml file. OpenShift
Container Platform 4 cluster nodes running RHCOS are immutable and rely on
Operators to apply cluster changes. Accessing cluster nodes by using SSH is not
recommended. However, when investigating installation issues, if the OpenShift
Container Platform API is not available, or the kubelet is not properly functioning
on a target node, SSH access might be required for debugging or disaster
recovery.
12.2.11.3. Advanced RHCOS installation configuration
A key benefit for manually provisioning the Red Hat Enterprise Linux CoreOS (RHCOS) nodes for
OpenShift Container Platform is to be able to do configuration that is not available through default
OpenShift Container Platform installation methods. This section describes some of the configurations
that you can do using techniques that include:
Passing kernel arguments to the live installer
Running coreos-installer manually from the live system
Customizing a live ISO or PXE boot image
The advanced configuration topics for manual Red Hat Enterprise Linux CoreOS (RHCOS) installations
detailed in this section relate to disk partitioning, networking, and using Ignition configs in different ways.
12.2.11.3.1. Using advanced networking options for PXE and ISO installations
Networking for OpenShift Container Platform nodes uses DHCP by default to gather all necessary
configuration settings. To set up static IP addresses or configure special settings, such as bonding, you
can do one of the following:
Pass special kernel parameters when you boot the live installer.
Use a machine config to copy networking files to the installed system.
Configure networking from a live installer shell prompt, then copy those settings to the installed
system so that they take effect when the installed system first boots.
To configure a PXE or iPXE installation, use one of the following options:
See the "Advanced RHCOS installation reference" tables.
Use a machine config to copy networking files to the installed system.
To configure an ISO installation, use the following procedure.
Procedure
1. Boot the ISO installer.
2. From the live system shell prompt, configure networking for the live system using available
RHEL tools, such as nmcli or nmtui.
3. Run the coreos-installer command to install the system, adding the --copy-network option to
2060
3. Run the coreos-installer command to install the system, adding the --copy-network option to
copy networking configuration. For example:
$ sudo coreos-installer install --copy-network \

--ignition-url=http://host/worker.ign /dev/disk/by-id/scsi-<serial_number>
IMPORTANT
The --copy-network option only copies networking configuration found under

/etc/NetworkManager/system-connections. In particular, it does not copy the
system hostname.
4. Reboot into the installed system.
See Getting started with nmcli and Getting started with nmtui in the RHEL 8 documentation for
more information about the nmcli and nmtui tools.
12.2.11.3.2. Disk partitioning
Disk partitions are created on OpenShift Container Platform cluster nodes during the Red Hat
Enterprise Linux CoreOS (RHCOS) installation. Each RHCOS node of a particular architecture uses the
same partition layout, unless you override the default partitioning configuration. During the RHCOS
installation, the size of the root file system is increased to use any remaining available space on the
target device.
IMPORTANT
The use of a custom partition scheme on your node might result in OpenShift Container
Platform not monitoring or alerting on some node partitions. If you override the default
partitioning, see Understanding OpenShift File System Monitoring (eviction conditions)
for more information about how OpenShift Container Platform monitors your host file
systems.
OpenShift Container Platform monitors the following two filesystem identifiers:
nodefs, which is the filesystem that contains /var/lib/kubelet
imagefs, which is the filesystem that contains /var/lib/containers
For the default partition scheme, nodefs and imagefs monitor the same root filesystem, /.
To override the default partitioning when installing RHCOS on an OpenShift Container Platform cluster
node, you must create separate partitions. Consider a situation where you want to add a separate
storage partition for your containers and container images. For example, by mounting
/var/lib/containers in a separate partition, the kubelet separately monitors /var/lib/containers as the
imagefs directory and the root file system as the nodefs directory.
IMPORTANT
If you have resized your disk size to host a larger file system, consider creating a separate
/var/lib/containers partition. Consider resizing a disk that has an xfs format to reduce
CPU time issues caused by a high number of allocation groups.
2061
12.2.11.3.2.1. Creating a separate /var partition
In general, you should use the default disk partitioning that is created during the RHCOS installation.
However, there are cases where you might want to create a separate partition for a directory that you
expect to grow.
/var directory or a subdirectory of /var. For example:
IMPORTANT
For disk sizes larger than 100GB, and especially larger than 1TB, create a separate
/var partition.
The use of a separate partition for the /var directory or a subdirectory of /var also prevents data growth
in the partitioned directory from filling up the root file system.
The following procedure sets up a separate /var partition by adding a machine config manifest that is
wrapped into the Ignition config file for a node type during the preparation phase of an installation.
Procedure
1. On your installation host, change to the directory that contains the OpenShift Container
Platform installation program and generate the Kubernetes manifests for the cluster:
variant: openshift
version: 4.15.0
metadata:
labels:
storage:
disks:
partitions:
2062
- label: var
number: 5
filesystems:
path: /var
format: xfs
2 When adding a data partition to the boot disk, a minimum offset value of 25000 mebibytes
to the specified offset. If no offset value is specified, or if the specified value is smaller than
the recommended minimum, the resulting root file system will be too small, and future
NOTE
for compute nodes, if the different instance types do not have the same device
name.

partition.yaml
4. Create the Ignition config files:
$ openshift-install create ignition-configs --dir <installation_directory> 1
installation directory:
.
├── auth
2063
The files in the <installation_directory>/manifest and <installation_directory>/openshift

directories are wrapped into the Ignition config files, including the file that contains the 98-var-
partition custom MachineConfig object.
Next steps
You can apply the custom disk partitioning by referencing the Ignition config files during the
RHCOS installations.
12.2.11.3.2.2. Retaining existing partitions
For an ISO installation, you can add options to the coreos-installer command that cause the installer to
maintain one or more existing partitions. For a PXE installation, you can add coreos.inst.* options to the
APPEND parameter to preserve partitions.
Saved partitions might be data partitions from an existing OpenShift Container Platform system. You
can identify the disk partitions you want to keep either by partition label or by number.
NOTE
If you save existing partitions, and those partitions do not leave enough space for
RHCOS, the installation will fail without damaging the saved partitions.
Retaining existing partitions during an ISO installation

This example preserves any partition in which the partition label begins with data (data*):
# coreos-installer install --ignition-url http://10.0.2.2:8080/user.ign \

--save-partlabel 'data*' /dev/disk/by-id/scsi-<serial_number>
The following example illustrates running the coreos-installer in a way that preserves the sixth (6)
partition on the disk:
# coreos-installer install --ignition-url http://10.0.2.2:8080/user.ign \

--save-partindex 6 /dev/disk/by-id/scsi-<serial_number>
This example preserves partitions 5 and higher:
# coreos-installer install --ignition-url http://10.0.2.2:8080/user.ign

--save-partindex 5- /dev/disk/by-id/scsi-<serial_number>
In the previous examples where partition saving is used, coreos-installer recreates the partition
immediately.
Retaining existing partitions during a PXE installation

This APPEND option preserves any partition in which the partition label begins with 'data' ('data*'):
coreos.inst.save_partlabel=data*
This APPEND option preserves partitions 5 and higher:
coreos.inst.save_partindex=5-
2064
This APPEND option preserves partition 6:
coreos.inst.save_partindex=6
12.2.11.3.3. Identifying Ignition configs
When doing an RHCOS manual installation, there are two types of Ignition configs that you can provide,
with different reasons for providing each one:
Permanent install Ignition config: Every manual RHCOS installation needs to pass one of the
Ignition config files generated by openshift-installer, such as bootstrap.ign, master.ign and
worker.ign, to carry out the installation.
IMPORTANT
It is not recommended to modify these Ignition config files directly. You can
update the manifest files that are wrapped into the Ignition config files, as
outlined in examples in the preceding sections.
For PXE installations, you pass the Ignition configs on the APPEND line using the
coreos.inst.ignition_url= option. For ISO installations, after the ISO boots to the shell prompt,
you identify the Ignition config on the coreos-installer command line with the --ignition-url=
option. In both cases, only HTTP and HTTPS protocols are supported.
Live install Ignition config: This type can be created by using the coreos-installer customize
subcommand and its various options. With this method, the Ignition config passes to the live
install medium, runs immediately upon booting, and performs setup tasks before or after the
RHCOS system installs to disk. This method should only be used for performing tasks that must
be done once and not applied again later, such as with advanced partitioning that cannot be
done using a machine config.
For PXE or ISO boots, you can create the Ignition config and APPEND the ignition.config.url=
option to identify the location of the Ignition config. You also need to append ignition.firstboot
ignition.platform.id=metal or the ignition.config.url option will be ignored.
12.2.11.3.4. Default console configuration
Red Hat Enterprise Linux CoreOS (RHCOS) nodes installed from an OpenShift Container Platform 4.15
boot image use a default console that is meant to accomodate most virtualized and bare metal setups.
Different cloud and virtualization platforms may use different default settings depending on the chosen
architecture. Bare metal installations use the kernel default settings which typically means the graphical
console is the primary console and the serial console is disabled.
The default consoles may not match your specific hardware configuration or you might have specific
needs that require you to adjust the default console. For example:
You want to access the emergency shell on the console for debugging purposes.
Your cloud platform does not provide interactive access to the graphical console, but provides a
serial console.
You want to enable multiple consoles.
Console configuration is inherited from the boot image. This means that new nodes in existing clusters
are unaffected by changes to the default console.
2065
You can configure the console for bare metal installations in the following ways:
Using coreos-installer manually on the command line.
Using the coreos-installer iso customize or coreos-installer pxe customize subcommands

with the --dest-console option to create a custom image that automates the process.
NOTE
For advanced customization, perform console configuration using the coreos-installer

iso or coreos-installer pxe subcommands, and not kernel arguments.
12.2.11.3.5. Enabling the serial console for PXE and ISO installations
By default, the Red Hat Enterprise Linux CoreOS (RHCOS) serial console is disabled and all output is
written to the graphical console. You can enable the serial console for an ISO installation and
reconfigure the bootloader so that output is sent to both the serial console and the graphical console.
Procedure
1. Boot the ISO installer.
2. Run the coreos-installer command to install the system, adding the --console option once to
specify the graphical console, and a second time to specify the serial console:
$ coreos-installer install \
--console=tty0 \ 1
--console=ttyS0,<options> \ 2
--ignition-url=http://host/worker.ign /dev/disk/by-id/scsi-<serial_number>
1 The desired secondary console. In this case, the graphical console. Omitting this option will
disable the graphical console.
2 The desired primary console. In this case the serial console. The options field defines the
baud rate and other settings. A common value for this field is 11520n8. If no options are
provided, the default kernel value of 9600n8 is used. For more information on the format
of this option, see Linux kernel serial console documentation.
3. Reboot into the installed system.
NOTE
A similar outcome can be obtained by using the coreos-installer install --

append-karg option, and specifying the console with console=. However, this
will only set the console for the kernel and not the bootloader.
To configure a PXE installation, make sure the coreos.inst.install_dev kernel command line option is
omitted, and use the shell prompt to run coreos-installer manually using the above ISO installation
procedure.
12.2.11.3.6. Customizing a live RHCOS ISO or PXE install
You can use the live ISO image or PXE environment to install RHCOS by injecting an Ignition config file
2066
You can use the live ISO image or PXE environment to install RHCOS by injecting an Ignition config file
directly into the image. This creates a customized image that you can use to provision your system.
For an ISO image, the mechanism to do this is the coreos-installer iso customize subcommand, which
modifies the .iso file with your configuration. Similarly, the mechanism for a PXE environment is the
coreos-installer pxe customize subcommand, which creates a new initramfs file that includes your
customizations.
The customize subcommand is a general purpose tool that can embed other types of customizations as
well. The following tasks are examples of some of the more common customizations:
Inject custom CA certificates for when corporate security policy requires their use.
Configure network settings without the need for kernel arguments.
Embed arbitrary preinstall and post-install scripts or binaries.
12.2.11.3.7. Customizing a live RHCOS ISO image
You can customize a live RHCOS ISO image directly with the coreos-installer iso customize
subcommand. When you boot the ISO image, the customizations are applied automatically.
You can use this feature to configure the ISO image to automatically install RHCOS.
Procedure
1. Download the coreos-installer binary from the coreos-installer image mirror page.
2. Retrieve the RHCOS ISO image from the RHCOS image mirror page and the Ignition config file,
and then run the following command to inject the Ignition config directly into the ISO image:
$ coreos-installer iso customize rhcos-<version>-live.x86_64.iso \

--dest-ignition bootstrap.ign \ 1
--dest-device /dev/disk/by-id/scsi-<serial_number> 2
1 The Ignition config file that is generated from the openshift-installer installation program.
2 When you specify this option, the ISO image automatically runs an installation. Otherwise,
the image remains configured for installation, but does not install automatically unless you
specify the coreos.inst.install_dev kernel argument.
3. Optional: To remove the ISO image customizations and return the image to its pristine state,
run:
$ coreos-installer iso reset rhcos-<version>-live.x86_64.iso
You can now re-customize the live ISO image or use it in its pristine state.
Applying your customizations affects every subsequent boot of RHCOS.
12.2.11.3.7.1. Modifying a live install ISO image to enable the serial console
On clusters installed with OpenShift Container Platform 4.12 and above, the serial console is disabled by
2067
default and all output is written to the graphical console. You can enable the serial console with the
following procedure.
Procedure
2. Retrieve the RHCOS ISO image from the RHCOS image mirror page and run the following
command to customize the ISO image to enable the serial console to receive output:

--dest-ignition <path> \ 1
--dest-console tty0 \ 2
--dest-console ttyS0,<options> \ 3
--dest-device /dev/disk/by-id/scsi-<serial_number> 4
1 The location of the Ignition config to install.
3 The desired primary console. In this case, the serial console. The options field defines the
of this option, see the Linux kernel serial console documentation.
4 The specified disk to install to. If you omit this option, the ISO image automatically runs the
installation program which will fail unless you also specify the coreos.inst.install_dev
kernel argument.
NOTE
The --dest-console option affects the installed system and not the live ISO
system. To modify the console for a live ISO system, use the --live-karg-append
option and specify the console with console=.
Your customizations are applied and affect every subsequent boot of the ISO image.
3. Optional: To remove the ISO image customizations and return the image to its original state,
$ coreos-installer iso reset rhcos-<version>-live.x86_64.iso
You can now recustomize the live ISO image or use it in its original state.
12.2.11.3.7.2. Modifying a live install ISO image to use a custom certificate authority
You can provide certificate authority (CA) certificates to Ignition with the --ignition-ca flag of the
customize subcommand. You can use the CA certificates during both the installation boot and when
provisioning the installed system.
NOTE
2068
NOTE
Custom CA certificates affect how Ignition fetches remote resources but they do not
affect the certificates installed onto the system.
Procedure
command to customize the ISO image for use with a custom CA:
$ coreos-installer iso customize rhcos-<version>-live.x86_64.iso --ignition-ca cert.pem
IMPORTANT
The coreos.inst.ignition_url kernel parameter does not work with the --ignition-ca flag.
You must use the --dest-ignition flag to create a customized image for each cluster.
Applying your custom CA certificate affects every subsequent boot of RHCOS.
12.2.11.3.7.3. Modifying a live install ISO image with customized network settings
You can embed a NetworkManager keyfile into the live ISO image and pass it through to the installed
system with the --network-keyfile flag of the customize subcommand.

WARNING
When creating a connection profile, you must use a .nmconnection filename

extension in the filename of the connection profile. If you do not use a
.nmconnection filename extension, the cluster will apply the connection profile to
the live environment, but it will not apply the configuration when the cluster first
boots up the nodes, resulting in a setup that does not work.
Procedure
2. Create a connection profile for a bonded interface. For example, create the
bond0.nmconnection file in your local directory with the following content:
[connection]
id=bond0
type=bond
interface-name=bond0
multi-connect=1
permissions=
[ethernet]
2069
mac-address-blacklist=
[bond]
miimon=100
mode=active-backup
[ipv4]
method=auto
[ipv6]
method=auto
[proxy]
3. Create a connection profile for a secondary interface to add to the bond. For example, create
the bond0-proxy-em1.nmconnection file in your local directory with the following content:
[connection]
id=em1
type=ethernet
interface-name=em1
master=bond0
multi-connect=1
permissions=
slave-type=bond
[ethernet]
[connection]
id=em2
type=ethernet
interface-name=em2
master=bond0
multi-connect=1
permissions=
slave-type=bond
[ethernet]
command to customize the ISO image with your configured networking:

--network-keyfile bond0.nmconnection \
--network-keyfile bond0-proxy-em1.nmconnection \
--network-keyfile bond0-proxy-em2.nmconnection
Network settings are applied to the live system and are carried over to the destination system.
2070
12.2.11.3.8. Customizing a live RHCOS PXE environment
You can customize a live RHCOS PXE environment directly with the coreos-installer pxe customize
subcommand. When you boot the PXE environment, the customizations are applied automatically.
You can use this feature to configure the PXE environment to automatically install RHCOS.
Procedure
2. Retrieve the RHCOS kernel, initramfs and rootfs files from the RHCOS image mirror page and
the Ignition config file, and then run the following command to create a new initramfs file that
contains the customizations from your Ignition config:
$ coreos-installer pxe customize rhcos-<version>-live-initramfs.x86_64.img \

--dest-ignition bootstrap.ign \ 1
--dest-device /dev/disk/by-id/scsi-<serial_number> \ 2
-o rhcos-<version>-custom-initramfs.x86_64.img 3
1 The Ignition config file that is generated from openshift-installer.
2 When you specify this option, the PXE environment automatically runs an install.
Otherwise, the image remains configured for installing, but does not do so automatically
unless you specify the coreos.inst.install_dev kernel argument.
3 Use the customized initramfs file in your PXE configuration. Add the ignition.firstboot
and ignition.platform.id=metal kernel arguments if they are not already present.
Applying your customizations affects every subsequent boot of RHCOS.
12.2.11.3.8.1. Modifying a live install PXE environment to enable the serial console
default and all output is written to the graphical console. You can enable the serial console with the
following procedure.
Procedure
the Ignition config file, and then run the following command to create a new customized
initramfs file that enables the serial console to receive output:

--dest-ignition <path> \ 1
--dest-console tty0 \ 2
--dest-console ttyS0,<options> \ 3
--dest-device /dev/disk/by-id/scsi-<serial_number> \ 4
-o rhcos-<version>-custom-initramfs.x86_64.img 5
1 The location of the Ignition config to install.
The desired secondary console. In this case, the graphical console. Omitting this option will
2071
3 The desired primary console. In this case, the serial console. The options field defines the
of this option, see the Linux kernel serial console documentation.
4 The specified disk to install to. If you omit this option, the PXE environment automatically
runs the installer which will fail unless you also specify the coreos.inst.install_dev kernel
argument.
5 Use the customized initramfs file in your PXE configuration. Add the ignition.firstboot
and ignition.platform.id=metal kernel arguments if they are not already present.
Your customizations are applied and affect every subsequent boot of the PXE environment.
12.2.11.3.8.2. Modifying a live install PXE environment to use a custom certificate authority
You can provide certificate authority (CA) certificates to Ignition with the --ignition-ca flag of the
customize subcommand. You can use the CA certificates during both the installation boot and when
provisioning the installed system.
NOTE
Custom CA certificates affect how Ignition fetches remote resources but they do not
affect the certificates installed onto the system.
Procedure
run the following command to create a new customized initramfs file for use with a custom CA:

--ignition-ca cert.pem \
-o rhcos-<version>-custom-initramfs.x86_64.img
3. Use the customized initramfs file in your PXE configuration. Add the ignition.firstboot and
ignition.platform.id=metal kernel arguments if they are not already present.
IMPORTANT
The coreos.inst.ignition_url kernel parameter does not work with the --ignition-ca flag.
You must use the --dest-ignition flag to create a customized image for each cluster.
Applying your custom CA certificate affects every subsequent boot of RHCOS.
12.2.11.3.8.3. Modifying a live install PXE environment with customized network settings
You can embed a NetworkManager keyfile into the live PXE environment and pass it through to the
installed system with the --network-keyfile flag of the customize subcommand.
2072

WARNING
When creating a connection profile, you must use a .nmconnection filename

extension in the filename of the connection profile. If you do not use a
.nmconnection filename extension, the cluster will apply the connection profile to
the live environment, but it will not apply the configuration when the cluster first
boots up the nodes, resulting in a setup that does not work.
Procedure
2. Create a connection profile for a bonded interface. For example, create the
bond0.nmconnection file in your local directory with the following content:
[connection]
id=bond0
type=bond
interface-name=bond0
multi-connect=1
permissions=
[ethernet]
[bond]
miimon=100
mode=active-backup
[ipv4]
method=auto
[ipv6]
method=auto
[proxy]
[connection]
id=em1
type=ethernet
interface-name=em1
master=bond0
multi-connect=1
permissions=
slave-type=bond
[ethernet]
2073
[connection]
id=em2
type=ethernet
interface-name=em2
master=bond0
multi-connect=1
permissions=
slave-type=bond
[ethernet]
run the following command to create a new customized initramfs file that contains your
configured networking:

--network-keyfile bond0.nmconnection \
-o rhcos-<version>-custom-initramfs.x86_64.img
6. Use the customized initramfs file in your PXE configuration. Add the ignition.firstboot and
ignition.platform.id=metal kernel arguments if they are not already present.
Network settings are applied to the live system and are carried over to the destination system.
12.2.11.3.9. Advanced RHCOS installation reference
This section illustrates the networking configuration and other advanced options that allow you to
modify the Red Hat Enterprise Linux CoreOS (RHCOS) manual installation process. The following tables
describe the kernel arguments and command-line options you can use with the RHCOS live installer and
the coreos-installer command.
12.2.11.3.9.1. Networking and bonding options for ISO installations
If you install RHCOS from an ISO image, you can add kernel arguments manually when you boot the
image to configure networking for a node. If no networking arguments are specified, DHCP is activated
in the initramfs when RHCOS detects that networking is required to fetch the Ignition config file.
IMPORTANT
When adding networking arguments manually, you must also add the rd.neednet=1
kernel argument to bring the network up in the initramfs.
The following information provides examples for configuring networking and bonding on your RHCOS
nodes for ISO installations. The examples describe how to use the ip=, nameserver=, and bond= kernel
arguments.
NOTE
2074
NOTE
Ordering is important when adding the kernel arguments: ip=, nameserver=, and then
bond=.
The networking options are passed to the dracut tool during system boot. For more information about
the networking options supported by dracut, see the dracut.cmdline manual page.
The following examples are the networking options for ISO installation.
Configuring DHCP or static IP addresses

To configure an IP address, either use DHCP (ip=dhcp) or set an individual static IP address ( ip=
<host_ip>). If setting a static IP, you must then identify the DNS server IP address ( nameserver=
<dns_ip>) on each node. The following example sets:
The node’s IP address to 10.10.10.2
The gateway address to 10.10.10.254
The netmask to 255.255.255.0
The hostname to core0.example.com
The DNS server address to 4.4.4.41
The auto-configuration value to none. No auto-configuration is required when IP networking is

configured statically.
ip=10.10.10.2::10.10.10.254:255.255.255.0:core0.example.com:enp1s0:none
nameserver=4.4.4.41
NOTE
When you use DHCP to configure IP addressing for the RHCOS machines, the machines
also obtain the DNS server information through DHCP. For DHCP-based deployments,
you can define the DNS server address that is used by the RHCOS nodes through your
DHCP server configuration.
Configuring an IP address without a static hostname

You can configure an IP address without assigning a static hostname. If a static hostname is not set by
the user, it will be picked up and automatically set by a reverse DNS lookup. To configure an IP address
without a static hostname refer to the following example:
The node’s IP address to 10.10.10.2
The gateway address to 10.10.10.254
The netmask to 255.255.255.0
The DNS server address to 4.4.4.41
The auto-configuration value to none. No auto-configuration is required when IP networking is

configured statically.
2075
ip=10.10.10.2::10.10.10.254:255.255.255.0::enp1s0:none
nameserver=4.4.4.41
Specifying multiple network interfaces

You can specify multiple network interfaces by setting multiple ip= entries.
Configuring default gateway and route

Optional: You can configure routes to additional networks by setting an rd.route= value.
NOTE
When you configure one or multiple networks, one default gateway is required. If the
additional network gateway is different from the primary network gateway, the default
gateway must be the primary network gateway.
Run the following command to configure the default gateway:
ip=::10.10.10.254::::
Enter the following command to configure the route for the additional network:
rd.route=20.20.20.0/24:20.20.20.254:enp2s0
Disabling DHCP on a single interface

You can disable DHCP on a single interface, such as when there are two or more network interfaces and
only one interface is being used. In the example, the enp1s0 interface has a static networking
configuration and DHCP is disabled for enp2s0, which is not used:
ip=::::core0.example.com:enp2s0:none
Combining DHCP and static IP configurations

You can combine DHCP and static IP configurations on systems with multiple network interfaces, for
example:
ip=enp1s0:dhcp
Configuring VLANs on individual interfaces

Optional: You can configure VLANs on individual interfaces by using the vlan= parameter.
To configure a VLAN on a network interface and use a static IP address, run the following
command:
ip=10.10.10.2::10.10.10.254:255.255.255.0:core0.example.com:enp2s0.100:none
vlan=enp2s0.100:enp2s0
To configure a VLAN on a network interface and to use DHCP, run the following command:
2076
ip=enp2s0.100:dhcp
vlan=enp2s0.100:enp2s0
Providing multiple DNS servers

You can provide multiple DNS servers by adding a nameserver= entry for each server, for example:
nameserver=1.1.1.1
nameserver=8.8.8.8
Bonding multiple network interfaces to a single interface

Optional: You can bond multiple network interfaces to a single interface by using the bond= option.
Refer to the following examples:
The syntax for configuring a bonded interface is: bond=<name>[:<network_interfaces>]

[:options]
<name> is the bonding device name (bond0), <network_interfaces> represents a comma-
separated list of physical (ethernet) interfaces (em1,em2), and options is a comma-separated
list of bonding options. Enter modinfo bonding to see available options.
When you create a bonded interface using bond=, you must specify how the IP address is
assigned and other information for the bonded interface.
To configure the bonded interface to use DHCP, set the bond’s IP address to dhcp. For
example:
bond=bond0:em1,em2:mode=active-backup
ip=bond0:dhcp
To configure the bonded interface to use a static IP address, enter the specific IP address
you want and related information. For example:
bond=bond0:em1,em2:mode=active-backup
ip=10.10.10.2::10.10.10.254:255.255.255.0:core0.example.com:bond0:none
Bonding multiple SR-IOV network interfaces to a dual port NIC interface
IMPORTANT
Support for Day 1 operations associated with enabling NIC partitioning for SR-IOV
devices is a Technology Preview feature only. Technology Preview features are not
supported with Red Hat production service level agreements (SLAs) and might not be
functionally complete. Red Hat does not recommend using them in production. These
Optional: You can bond multiple SR-IOV network interfaces to a dual port NIC interface by using the
bond= option.
On each node, you must perform the following tasks:
1. Create the SR-IOV virtual functions (VFs) following the guidance in Managing SR-IOV devices.
2077
1. Create the SR-IOV virtual functions (VFs) following the guidance in Managing SR-IOV devices.
Follow the procedure in the "Attaching SR-IOV networking devices to virtual machines" section.
2. Create the bond, attach the desired VFs to the bond and set the bond link state up following
the guidance in Configuring network bonding. Follow any of the described procedures to create
the bond.
The following examples illustrate the syntax you must use:
The syntax for configuring a bonded interface is bond=<name>[:<network_interfaces>]

[:options].
<name> is the bonding device name (bond0), <network_interfaces> represents the virtual
functions (VFs) by their known name in the kernel and shown in the output of the ip link
command(eno1f0, eno2f0), and options is a comma-separated list of bonding options. Enter
modinfo bonding to see available options.
When you create a bonded interface using bond=, you must specify how the IP address is
assigned and other information for the bonded interface.
To configure the bonded interface to use DHCP, set the bond’s IP address to dhcp. For
example:
bond=bond0:eno1f0,eno2f0:mode=active-backup
ip=bond0:dhcp
To configure the bonded interface to use a static IP address, enter the specific IP address
you want and related information. For example:
bond=bond0:eno1f0,eno2f0:mode=active-backup
ip=10.10.10.2::10.10.10.254:255.255.255.0:core0.example.com:bond0:none
Using network teaming

Optional: You can use a network teaming as an alternative to bonding by using the team= parameter:
The syntax for configuring a team interface is: team=name[:network_interfaces]

name is the team device name (team0) and network_interfaces represents a comma-separated
list of physical (ethernet) interfaces (em1, em2).
NOTE
Teaming is planned to be deprecated when RHCOS switches to an upcoming version of

RHEL. For more information, see this Red Hat Knowledgebase Article

Openshift Container Platform 4.15 Installing en Us

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Openshift Container Platform 4.15 Installing en Us

Uploaded by

Copyright:

Available Formats

OpenShift Container Platform 4.

Installing and configuring OpenShift Container Platform clusters

Last Updated: 2024-04-25

Java ® is a registered trademark of Oracle and/or its affiliates.

All other trademarks are the property of their respective owners.

3.2.11. Operator Lifecycle Manager capability 100

4.2.10.22.1. Bug fixes 116

Installing the OpenShift CLI on macOS 200

5.6.10. Telemetry access for OpenShift Container Platform 244

6.4.2. Internet access for OpenShift Container Platform 291

defaultNetwork object configuration 346

6.7.2.2. VPC validation 395

6.8.7.4. Sample customized install-config.yaml file for AWS 438

6.9.12.2. Configuring an AWS cluster to use short-term credentials 484

6.11. INSTALLING A CLUSTER ON AWS CHINA 533

6.12.4. Required AWS infrastructure components 575

6.13.2. About installations in restricted networks 684

6.14.1. Infrastructure prerequisites 790

6.15.4.1. Minimum resource requirements for cluster installation 836

7.2.5.3. Using Azure managed identities 899

7.6. INSTALLING A CLUSTER ON AZURE WITH NETWORK CUSTOMIZATIONS 945

7.7.7. Installing the OpenShift CLI by downloading the binary 999

7.8.14. Next steps 1043

7.10.4.1. Obtaining the installation program 1089

7.11.4.2. Minimum resource requirements for cluster installation 1160

7.12.6.1. Minimum resource requirements for cluster installation 1228

8.3.4. Uploading the RHCOS cluster image 1288

9.2.3. Configuring DNS for GCP 1385

9.5. INSTALLING A CLUSTER ON GCP WITH NETWORK CUSTOMIZATIONS 1438

Installing the OpenShift CLI on Linux 1489

9.8.4. Obtaining the installation program 1534

9.9.10.2.3. Incorporating the Cloud Credential Operator utility manifests 1579

9.10.16. Creating the control plane machines in GCP 1633

9.11.11. Creating a private DNS zone in GCP 1682

10.2. CONFIGURING AN IBM CLOUD ACCOUNT 1829

10.6.7. Manually creating IAM 1863

10.8.9.1. Minimum resource requirements for cluster installation 1903

11.1. PREPARING TO INSTALL ON NUTANIX 1955

11.4.10.1. Disabling the default OperatorHub catalog sources 2000

12.2.9.3. Configuring a three-node cluster 2048

12.3.2. Internet access for OpenShift Container Platform 2098

Configuring DHCP or static IP addresses 2158

12.4.11.3. Advanced RHCOS installation configuration 2218

14.3.4. Gathering log data from a failed Agent-based installation 2316

Redfish network boot for HPE iLO 2418

16.6.9. Cluster nodes will not PXE boot 2486

18.2.3.2. Minimum resource requirements for cluster installation 2523

Using network teaming 2567

Hardware requirements 2696

18.5.17. Next steps 2750

Disabling DHCP on a single interface 2794

18.7.8.3. Configuring a three-node cluster 2832

Operating system requirements 2883

19.2.13. Waiting for the bootstrap process to complete 2930

19.3.11.1. Installing RHCOS by using an ISO image 2977

20.6.1. Prerequisites 3060

20.9.1.1. Required configuration parameters 3098

21.3.12.2. Completing installation without floating IP addresses 3156

21.5.3.3. Bootstrap machine 3200

The Ingress Load Balancer in OCI is not at a healthy status 3265

23.2.4. Installing a cluster on vSphere with customizations 3319

23.2.6.5. VMware vSphere region and zone enablement 3385

23.3.3.2. Internet access for OpenShift Container Platform 3450

23.3.4.16. Completing installation on user-provisioned infrastructure 3520