You are on page 1of 58

EIS Installation Checklist for the

ORACLE® Database Appliance X8-2 High Availability (X8-2 HA) -


Bare Metal Platform

Customer:

TASK Number:

Technician:

Date:

This checklist is for the “Limited Software Install” service for the Oracle Database Appliance X8-2
HA – Bare Metal Platform.
• Be sure to check the EIS web page for the latest versions prior to commencing the
installation.
• The purpose of this checklist is to help the installer achieve a consistent, "good" installation.
• Installers must have attended the appropriate training classes.
◦ EIS checklists are NOT a replacement for proper training.
• Use of a laptop is required during the installation.
◦ Linux would make life easier.
• Do NOT hand this checklist over to the customer.
• Feedback on issues with EIS content or product quality is welcome.
◦ Oracle staff should send email to EIS-SUPPORT_WW_GRP@oracle.com
◦ Partners should contact the PartnerHelp Portal for assistance and feedback.

System Type Top Level Identification (TLI)


Oracle Database Appliance X8-2 HA
In this document, we refer to the servers as Node 0 and Node 1. Node 0 is the bottom server (oak0
prompt) and Node 1 is the top server (oak1 prompt).
The hardware essentially comprises of two Oracle X8-2 server nodes with an interconnect and one or
two DE3-24C storage shelves. The Top Level Identification (TLI) sticker which contains the serial
number is on the top of a server node or a storage shelf, on the left-hand side towards the front.
NOTE: When multiple ODA systems are being installed, make sure that for each ODA system that
the TLI sticker shows the same serial number on both nodes (do not mix up the nodes!).
The original storage shelf and expansion shelf (if they have one) will also come with a TLI sticker to
match the nodes. Although it is preferred, it is not mandatory to match the storage TLI stickers with
the nodes.

Oracle Internal and Approved Partners Only Page 1 of 58 Version 2.1 – Created: 9 Jun 2022
ENGINEERED SYSTEMS ENTERPRISE SUPPORT TEAM EEST
If a Field Support Engineer (FSE) requires assistance while installing an Engineered System, a
streamlined method of engaging an Engineered Systems Enterprise Support Team (EEST) engineer
when the SR owner is not available is described below. The SR referred to is the installation SR.
An FSE would use this option if they are onsite and require immediate assistance from EEST. This
process is specifically for FSE Callbacks for hardware, installation and Field Change Order (FCOs)
support and should NOT be used by ACS or partners.
The complete process is described in MOS Document ID 1803744.1 EEST: Silent Menu Option :
GCSEXA (internal-only). Ensure that you understand the contents before going onsite.
If you encounter an issue during the installation, this may be documented in the ODA Current
Product Issues page: MOS Document ID 1538187.1 (internal-only). Refer to this document for a
possible workaround to any bug you may encounter during the installation.
For additional help with ODA deployment, you can send an email to the following mailing list:
global-field-oda-sme-assist_grp@oracle.com
This mailing list is intended for use by Installers/FEs to reach out to other Field resources, who have
been identified as ODA Subject Matter Experts, for help with ODA installs.

INSTALLATION OVERVIEW– EIS DELIVERABLES


For an overview of the activities carried out during the installation of this product via the EIS
Methodology, refer to the EIS-Deliverables page within the EIS website for the Oracle Database
Appliance ODA.

This EIS does not cover KVM configuration. After completing the EIS, if KVM is desired, then the
administrator of the ODA system can refer to Oracle Database Appliance Documentation in Deploy
KVM Database System.

Oracle Internal and Approved Partners Only Page 2 of 58 Version 2.1 – Created: 9 Jun 2022
MEMORY DIMM FAULT BEST PRACTICES
TSC are seeing many DIMMs being replaced in the field due to initial boot up and Memory DIMM
fault either due to Memory DIMM movement during shipping or improper seating during memory
upgrades.
If Memory DIMM faults are encountered during the installation or upgrade of any engineered
systems, please follow the "BEST PRACTICE" as supplied by the hardware team.
Best Practice for resolving DIMM Faults during Installation or upgrade:
1. We (the TSC) are recommending the re-seating of Memory DIMMs when seeing faults during
installation or memory upgrades, as a first step.
2. If we see these failures outside of upgrades or rack installs, we don't want to take the chance
and risk of a second visit if they're truly failed Memory DIMMs. So we recommend, in these
instances, to simply replace the Memory DIMM. If the replaced Memory DIMM experiences
Training errors, then it might be prudent to re-seat the Memory DIMM.

INFORMATION: EIS Checklist Steps & Estimated Timing


Item First Page Est. Duration
Preparation 6 4 hours
Unpacking 8 30 mins
Assembly / Cabling 9 45 mins
ILOM Service Processor Configuration 13 10 mins
Server Power On & POST 15 20 mins
Bare Metal (Non-Virtualized) ODA 105 (+60) mins
Verify Pre-Installed Version 19 10 mins
Re-imaging for the Bare Metal configuration
20 (60 mins)
(if necessary)
Bare Metal ODA Configuration 30 mins
Changing the ODA CPU Core Count 15 mins
Deploying the ODA Using the Oracle Appliance Manager
43 40 mins
Configuration
Post Validation 55 10 mins
Common Actions
Securing the ILOM Service Processors (SP) 57 10 mins
Handover 57 15 mins
Appendix A: Using the odacli Command to Monitor
10 mins
Deployment

Oracle Internal and Approved Partners Only Page 3 of 58 Version 2.1 – Created: 9 Jun 2022
AUTOMATIC SERVICE REQUEST (ASR) ACTIVITIES
If the ODA will not be configured to use ASR go to the section entitled ODA
PREPARATION below.
While it is highly recommended to use an external ASR Manager with ODA it is supported to
use an internal ASR manager.
If the ASR Manager has not been previously installed AND the Customer requests that Oracle
install it, go to the documentation page for Oracle ASR and select the URL for the ASR
Manager installation and operations guide, quick install guide, security white paper. There
you will find the current Installation and Operations Guide. Follow the instructions to install
the ASRM based on the system you have been given to install it in.
Register the ASR manager with Oracle – follow the installation process through the point
where the ASR is registered to the backend.
This activity must be completed prior to installation of the ODA(s).
CHECKING A CUSTOMER'S EXISTING ASR MANAGER SERVER
If the Customer has a configured and operational ASR Manager system available, the
components within the ODA will be configured to use ASR. Ensure that the IP address &
root password of the ASR Manager host are available.

ODA PREPARATION
EIS Site-Audit Report complete? EISdoc V4:
The server nodes require 1RU each & each storage
shelf requires 4 RU hence an ODA with one storage
shelf requires a total of 6 RUs
EIS Installation Configuration Plan & Test EISdoc V4:
Procedures Plan complete? Use appropriate BUILD & TPP templates, inserted
into “Master” files (Chapter Servers):
EIS-BUILD-Server-ODA-X8HA.odt
EIS-TPP-Server-ODA-X8HA.odt
IMPORTANT: Review the network IP addresses are required for both the hosts & the
configuration information supplied by the Service Processors (two of each).
customer to ensure the correct data is
recorded. Do this BEFORE going onsite.

The checklist information for the deployment using the configurator can be found in the
Oracle Database Appliance Documentation – Deployment Checklists
Refer to the Oracle Database Appliance documentation for more details, including a setup
poster that is included with the server delivery.
An Oracle Internal-only source of information is the FAQ for the Oracle Database Appliance
- Externally available documentation and information on all ODA products is available on the
Oracle Technical Network (OTN). Any additional questions should be referred to their sales
representative.

Oracle Internal and Approved Partners Only Page 4 of 58 Version 2.1 – Created: 9 Jun 2022
Terminology
Oracle Database Appliance patches and software consists the following components:
• ODA Bundle Patch
Contains patches to the following components: OS, Component Firmware and Oracle
Appliance Manager Modules, Grid Infrastructure and RDBMS, dcs components
• GI/RDBMS Clone files
Contains the latest Grid Infrastructure and RDBMS components, for deployment on an
Oracle Database Appliance in the "shipped from factory" state, or an Oracle Database
Appliance that has been re-imaged using the OS ISO Image.
• OS ISO Image
Used to restore the OS to the "shipped from factory" state. Use only when necessary.
• Bare Metal
Refers to the non-virtualized ODA configuration.

PREPARING THE DOWNLOADS

Oracle Internal and Approved Partners Only Page 5 of 58 Version 2.1 – Created: 9 Jun 2022
It is highly recommended that you always obtain the latest ODA software releases directly
from MOS. The Bare Metal software image is installed by the factory and is most likely not
the latest version available on MOS. At the time of this publication, the current software
version is 19.15.0.0.0. We recommend always installing the latest version.

The latest patches and ISO images can be found in MOS in the Patches and Updates tab.
In the Patch Search section, click on the Product or Family (Advanced) side tab, and next to
the Product field enter “Oracle Database Appliance Software”. Next to the Release field, in
the drop-down box, select “Oracle Appliance Kit 19.15.0.0.0”.

Click on Search button.

Oracle Internal and Approved Partners Only Page 6 of 58 Version 2.1 – Created: 9 Jun 2022
PATCHES NEEDED TO DEPLOY BARE METAL
Required Patches:
• 30403643 – ODA ISO Bare Metal Restore file
p30403643_1915000_Linux-x86-64.zip    5.0 GB

• 30403673 – GI Clone files. This file is required for deployment.


p30403673_1915000_Linux-x86-64.zip    6.4 GB

• 33702951 – Server patch bundle


p34069644_1915000_Linux-x86-64.zip    3.1 GB

Database Clone patches. It is only required to install the RDBMS patch that is desired
by the customer. Software 19.15 will only allow RDBMS 19.15 or RDBMS 12.1.0.2 for
initial deployment.

• 30403662 – Oracle 19.15.0.0.220419 RDBMS Software Clone file for creating any
new 19.15 database homes.

p30403662_1915000_Linux-x86-64.zip    4.9 GB

• 23494992 – Oracle 12.1.0.2.220419 RDBMS Software Clone file for creating any
new 12.1.0.2 database homes.

p23494992_1915000_Linux-x86-64.zip    8.9 GB

NOTE: We recommend transferring the files directly to the server over a network
connection.
Alternatively, if you prefer, you can also use a USB drive to transfer the files. The
larger files (over 4GB) will not transfer to a FAT32 formatted USB drive due to
filesystem limitations. The drive must be formatted with “EXT4” (Linux) or Windows
NTFS filesystem.

Oracle Internal and Approved Partners Only Page 7 of 58 Version 2.1 – Created: 9 Jun 2022
UNPACKING Check

Delivery complete?
Collect the white Customer Information File away carefully!
Sheets (CIS).
Allow the system to acclimatise (power off) at Refer to EIS standard “Acclimatisation of Oracle
the customer site. Hardware products”.

Unpack outside data center to ensure no Collect packing material together for disposal.
contamination/dust is released inside
customer's controlled environment.

Verify all packing material has been removed, Fans & air vents must be free to operate.
i.e. nothing is blocked.
Gather excess parts, cables etc. in a box. • (2) Tool-less rack mount kits (Server nodes)
• (2) Cable Management Arms (Server nodes)
• (1) Fixed rack mount kit (Storage shelf)
• (2) 3 meter SFP28 passive copper cables (for
Interconnect)
• (4) 1 meter SAS cables (Storage server
Connectivity to Server nodes)
• Setup poster
Collect documentation together. Give to customer during handover.

Refer to the Oracle Database Appliance Owner's Guide for details on inserting the system into a rack.
A 8RU space is required for a one storage shelf configuration and a 12RU space is required for the two
storage shelf configuration.
Supported in any racks meeting the below criteria:
• Four-post rack (mounting at both front and rear), not supported in two-post racks.
• Rack should have a horizontal opening and unit vertical pitch conforming to ANSI/EIA 310-
D-1992 or IEC 60927 standards.
• Distance between front and rear mounting planes between approximately 24 and 36 inches
(610 mm and 915 mm).
• Minimum clearance depth (to front cabinet door) in front of front rack mounting plane: 1 inch
(25.4 mm).
• Minimum clearance depth (to rear cabinet door) behind front rack mounting plane: 35.43
inches (900 mm) with the cable management arm, or 800 mm (31.5 in) without the cable
management arm.
• Minimum clearance width (between structural supports and cable troughs) between front and
rear mounting planes: 18 inches (456 mm).
• Minimum clearance for front service access is 48.5 inches (123.2 cm).
• Minimum clearance for rear service access is 36 inches (91 cm).
• Always load equipment into a rack from the bottom upwards so that it will not become top-
heavy and tip over. Deploy the rack's anti-tilt bar to prevent the rack from tipping during
equipment installation.

Oracle Internal and Approved Partners Only Page 8 of 58 Version 2.1 – Created: 9 Jun 2022
ASSEMBLY/CABLING Check

Note the serial number. The TLI sticker which contains the serial number is on
the top of a server node or a storage shelf, on the left-
hand side towards the front.
Distribute systems across power-sequencers considering failure-safety.
Connect Ethernet cables for both SPs to the To the NET MGT port on each node.
administration network. Node 1 is the top server.
Node 0 is the bottom server.
Connect serial cables between the SPs and a terminal device or PC – used for the initial
configuration to the SER MGT ports.

PRIVATE INTERCONNECT
The ODA ships with two 1-meter SFP28 passive copper cables. Use these for the
interconnects.
Connect green cable Connect cable from PCIe slot 1, PORT2 on node0 to
PCIe slot 1, PORT2 on node1.
Connect yellow cable Connect cable from PCIe slot 1, PORT1 on node0 to
PCIe slot 1, PORT1 on node1.

These two private interfaces are named p1p1/p1p2, which in turn is icbond0.

Oracle Internal and Approved Partners Only Page 9 of 58 Version 2.1 – Created: 9 Jun 2022
CLIENT ACCESS CABLING
On each node, there is a network card installed in slot 7. Below is a picture of the cards if the
customer has chosen the dual 10/25gb Whitney+ card (btbond1). If the customer has chosen
the 10gb Ethernet Fortpond card, the cards in slot 7 have four RJ45 ethernet ports (btbond1
and btbond2).

The client access ports above are p7p1/p7p2. For a quad port card, you will also have
p7p3/p7p4.

Oracle Internal and Approved Partners Only Page 10 of 58 Version 2.1 – Created: 9 Jun 2022
CABLING THE STORAGE SHELF Check

Connect Node 0 to Storage shelf with dark Connect into dark blue port (SAS0) in PCIe slot 3 Node
blue cable. 0 to dark blue port in top IO Module (port 0) on the
Storage shelf.
Connect Node 0 to Storage shelf with light Connect into light blue port (SAS1) in PCIe slot 8 Node
blue cable. 0 to light blue port in bottom IO Module (port 0) on the
Storage shelf.
Connect Node 1 to Storage shelf with dark red Connect into dark red port (SAS1) in PCIe slot 3 Node
cable. 1 to dark red port in top IO Module (port 1) on the
Storage shelf.
Connect Node 1 to Storage shelf with light red Connect into light red port (SAS0) in PCIe slot 8 Node
cable. 1 to light red port in bottom IO Module (port 1) on the
Storage shelf.

Oracle Internal and Approved Partners Only Page 11 of 58 Version 2.1 – Created: 9 Jun 2022
CABLING THE EXPANSION STORAGE SHELF (OPTIONAL) Check

By default, the ODA X8-2 HA will have one storage shelf. An additional shelf is optional.
Connect Node 1 to Storage Expansion shelf Connect into dark blue port (SAS0) in PCIe slot 3 in
with dark blue cable. node 1 to dark blue port in top IO Module (port 0) on
Exp. Storage Shelf.
Connect Node 1 to Storage Expansion shelf Connect into light blue port (SAS1) in PCIe slot 8 in
with light blue cable. Node 1 to light blue port in bottom IO Module (port 0)
on Exp. Storage Shelf.
Connect Node 0 to Storage Expansion shelf Connect into dark red port (SAS1) in PCIe slot 3 in
with dark red cable. Node 0 to dark red port in top IO Module (port 1) on
Exp. Storage Shelf.
Connect Node 0 to Storage Expansion shelf Connect into light red port (SAS0) in PCIe slot 8 in
with light red cable. Node 0 to light red port in bottom IO Module (port 1)
on Exp. Storage Shelf.

COMPLETING THE CABLING


Label the cables clearly & use cable binders.
Perform initial power on by attaching the The SPs will boot and then will reduce the speed of the
power cables. Both AC power connectors on cooling fans. The Power/OK LEDs on the front panel
flash (standby mode).
the back panel must be attached via grounded
AC power cords.

Oracle Internal and Approved Partners Only Page 12 of 58 Version 2.1 – Created: 9 Jun 2022
ILOM SERVICE PROCESSOR CONFIGURATION Node
0 1
The hardware essentially comprises of two nodes with an interconnect, with one or two storage
shelves. Hence each node has a Service Processor that needs to be configured.
This checklist assumes IPv4 networking.
Alternatively, if you are using a KVM or directly attached monitor and keyboard, the ILOM may be
configured using the configurator if you choose the Custom install option. This will save the task of
connecting to the serial port to configure the ILOM net management port manually. In this case skip
this entire section and go to page 15 (Server Power on & Post).
When the SP initially boots it will attempt to If DHCP server is available you can connect to the SP
obtain an IP address from a DHCP server. via network (ssh or https), otherwise you have to
connect to the SP via a serial line (see below) and
You must have a cable connected to the SP specify a static Ethernet configuration.
Network Management port (NET MGT).
Obtain the Service Processor IP address and record the IP address for future reference. This IP
address is usually assigned by the DHCP server.
To get the SP address, you can use one of the following methods:
• The system BIOS setup screen.
◦ Advanced → BMC Network Configuration → Current Ipv4 address in BMC
• Command-line interface – connect to serial port via terminal device (such as a laptop).
Log into the SP via CLI and display the IP address.
User = root
Password = changeme
-> show /SP/network

CONFIGURING ILOM VIA CLI AND A SERIAL CONNECTION


To set a static IP address for the SP using the CLI and a serial line connection perform the
steps below.
Establish a serial connection to the SP Serial When using a laptop, start a terminal interface to the
Management Port (SER MGT). serial port that is connected to the ILOM SP.
Note that the default serial port speed is now 115200
baud (previously 9600) with no flow control.
Log in to the ILOM SP. User = root
Password = changeme
Set the working directory. -> cd /SP/network

Enter the following commands to specify a The following are example values:
static Ethernet configuration. Note that there -> set pendingipaddress=129.144.82.26
-> set pendingipnetmask=255.255.255.0
are no spaces in command line structure. -> set pendingipgateway=129.144.82.254
It is also possible to configure ILOM network -> set pendingipdiscovery=static
settings via the system BIOS in the Advanced -> set commitpending=true
-> set state=enabled
Menu tab under BMC Network. Note that the commitpending command requires
several seconds to complete.
Verify the changes. -> show /SP/network

Oracle Internal and Approved Partners Only Page 13 of 58 Version 2.1 – Created: 9 Jun 2022
ILOM SERVICE PROCESSOR CONFIGURATION Node
0 1
Verify that the gateway can be reached:
-> set /SP/network/test ping=<IP of gateway>
ping of <IP of gateway> succeeded

After setting up the SP, exit. -> exit

CONFIGURING THE SP'S CLOCK PROPERTIES VIA CLI


Refer to the Oracle ILOM Administrator's Guide for Configuration and Maintenance Firmware
Release 5.0.x
To examine the current settings:
-> show /SP/clock
/SP/clock
Targets:
Properties:
datetime = Fri Aug 23 10:10:09 2013
timezone = GMT (GMT)
uptime = 1 day, 19:17:16
usentpserver = disabled

If required, the timezone can be changed from GMT (default) to EST or PST8PDT:
-> set /SP/clock timezone=EST5EDT

To manually set the ILOM clock property values:


-> set /SP/clock datetime=MMDDhhmmYYYY

Example:
-> set /SP/clock datetime=082310132013
Set 'datetime' to '082310132013'
-> show /SP/clock
/SP/clock
Targets:

Properties:
datetime = Fri Aug 23 10:13:04 2013
timezone = GMT (GMT)
uptime = 1 day, 19:17:16
usentpserver = disabled

To synchronize the SP's clock property values with an NTP server:


-> set /SP/clients/ntp/server/1 address=ip_address
-> set /SP/clock usentpserver=enabled

Oracle Internal and Approved Partners Only Page 14 of 58 Version 2.1 – Created: 9 Jun 2022
SERVER POWER ON & POST Node
0 1
Power on each Node. There are several ways to do this:
• Press the power-on button on the front panel of each server.
• Using the GUI on each expand the Host Management menu on the left, select Power
Control. You will see Host is currently off – select Power On. Then press the save
button.
The fans will speed up and after a few seconds The console will show the results of POST. Expect to
back down. Observe the POST results. see “BMC Responding”.

Using GUI: gain console access to Server Start console redirection.


Node SP.
At the OS login prompt: User root with welcome1 as password.

VALIDATE STORAGE CABLING Node

0 1

Run the validate scripts to confirm that the cabling is correct:

[root@oak0 ~]# odacli validate-storagetopology


INFO : ODA Topology Verification
INFO : Running on Node0
INFO : Check hardware type
SUCCESS : Type of hardware found : X8-2
INFO : Check for Environment(Bare Metal or Virtual Machine)
SUCCESS : Type of environment found : Bare Metal
INFO : Check number of Controllers
SUCCESS : Number of ahci controller found : 1
SUCCESS : Number of External SCSI controllers found : 2
INFO : Check for Controllers correct PCIe slot address
SUCCESS : Internal LSI SAS controller :
SUCCESS : External LSI SAS controller 0 : 3b:00.0
SUCCESS : External LSI SAS controller 1 : e3:00.0
INFO : Check if JBOD powered on
SUCCESS : 1JBOD : Powered-on
INFO : Check for correct number of EBODS(2 or 4)
SUCCESS : EBOD found : 2
INFO : Check for External Controller 0
SUCCESS : Controller connected to correct EBOD number
SUCCESS : Controller port connected to correct EBOD port
SUCCESS : Overall Cable check for controller 0
INFO : Check for External Controller 1
SUCCESS : Controller connected to correct EBOD number
SUCCESS : Controller port connected to correct EBOD port
SUCCESS : Overall Cable check for Controller 1
INFO : Check for overall status of cable validation on Node0
SUCCESS : Overall Cable Validation on Node0
INFO : Check Node Identification status
SUCCESS : Node Identification
SUCCESS : Node name based on cable configuration found : NODE0

Oracle Internal and Approved Partners Only Page 15 of 58 Version 2.1 – Created: 9 Jun 2022
If there is a storage expansion unit attached, now is the time to power it on. Re-run the
storagetopology command on both nodes to make sure the cabling is correct:

[root@oak0 ~]# odacli validate-storagetopology


INFO : ODA Topology Verification
INFO : Running on Node0
INFO : Check hardware type
SUCCESS : Type of hardware found : X8-2
INFO : Check for Environment(Bare Metal or Virtual Machine)
SUCCESS : Type of environment found : Bare Metal
INFO : Check number of Controllers
SUCCESS : Number of ahci controller found : 1
SUCCESS : Number of External SCSI controllers found : 2
INFO : Check for Controllers correct PCIe slot address
SUCCESS : Internal RAID controller :
SUCCESS : External LSI SAS controller 0 : 3b:00.0
SUCCESS : External LSI SAS controller 1 : 5e:00.0
INFO : Check if JBOD powered on
SUCCESS : 2JBOD : Powered-on
INFO : Check for correct number of EBODS(2 or 4)
SUCCESS : EBOD found : 4
INFO : Check for External Controller 0
SUCCESS : Cable check for port 0 on controller 0
SUCCESS : Cable check for port 1 on controller 0
SUCCESS : Overall Cable check for controller 0
INFO : Check for External Controller 1
SUCCESS : Cable check for port 0 on controller 1
SUCCESS : Cable check for port 1 on controller 1
SUCCESS : Overall Cable check for controller 1
INFO : Check for overall status of cable validation on Node0
SUCCESS : Overall Cable Validation on Node0
SUCCESS : JBOD0 Nickname set correctly
SUCCESS : JBOD1 Nickname set correctly

Oracle Internal and Approved Partners Only Page 16 of 58 Version 2.1 – Created: 9 Jun 2022
CHECK system_identifier FIELD Node

0 1

On both nodes display the system identifier:


# ipmitool sunoem getval /X/system_identifier
Target Value: Oracle Database Appliance X8-2 High Availability <serial #>

The output from both nodes should have the same system_identifier denoted with the same
serial number.
-> show /System
/System
Targets:
Open_Problems (0)
Processors
Memory
Power
Cooling
Storage
Networking
PCI_Devices
Firmware
BIOS
Log
Properties:
health = OK
health_details = -
open_problems_count = 0
type = Rack Mount
model = ORACLE SERVER X8-2L
qpart_id = Q13629
part_number = 7600551-8533
serial_number = 1936XL0003 <<< this is the serial # you want to use for system_identifier
rfid_serial_number = 341A583DE580000000045159
component_model = ORACLE SERVER X8-2L
component_part_number = 7600551-8533
component_serial_number = 1936XLA07C
chassis_model = ORACLE SERVER X8-2L
chassis_part_number = 7600551-8533-1
chassis_serial_number = 1936XLA07C
system_identifier = Oracle Database Appliance X8-2 High Availability 1936XL0003
system_fw_version = 4.0.4.51.a
primary_operating_system = Not Available

Oracle Internal and Approved Partners Only Page 17 of 58 Version 2.1 – Created: 9 Jun 2022
VALIDATE INTERCONNECT CABLING Node

0 1

# ethtool p1p1
Settings for p1p1:
Supported ports: [ FIBRE ]
Supported link modes: 1000baseT/Full
10000baseT/Full
25000baseCR/Full
Supported pause frame use: Symmetric Receive-only
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 1000baseT/Full
10000baseT/Full
25000baseCR/Full
Advertised pause frame use: No
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: 25000Mb/s <<<< Check the speed
Duplex: Full
Port: Direct Attach Copper
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000000 (0)

Link detected: yes <<<< This should be "yes"

# ethtool p1p2
Settings for p1p2:
Supported ports: [ FIBRE ]
Supported link modes: 1000baseT/Full
10000baseT/Full
25000baseCR/Full
Supported pause frame use: Symmetric Receive-only
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 1000baseT/Full
10000baseT/Full
25000baseCR/Full
Advertised pause frame use: No
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: 25000Mb/s <<<< Check the speed
Duplex: Full
Port: Direct Attach Copper
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000000 (0)

Link detected: yes <<<< This should be "yes"

Oracle Internal and Approved Partners Only Page 18 of 58 Version 2.1 – Created: 9 Jun 2022
VERIFY PRE-INSTALLED VERSION Node
0
To verify the current ODA system version, run the following command on both nodes:
# odacli describe-component -v
System Version
---------------
19.15.0.0.0

System node Name


---------------
oak0

Local System Version


---------------
19.15.0.0.0

System node Name


---------------
oak1

Local System Version


---------------
19.15.0.0.0

If the version is the latest available on MOS, then you can proceed with the Bare Metal
deployment.

If the version is older than what is available on MOS, you can re-image using the ISO file
contained in patch 30403643 (which was downloaded during the Preparation activities on page
5 of this checklist) and continue to follow the instructions starting on the next page of this
checklist.

Oracle Internal and Approved Partners Only Page 19 of 58 Version 2.1 – Created: 9 Jun 2022
RE-IMAGING FOR THE BARE METAL CONFIGURATION Node
0 1
To re-image with a newer version, you will require the Bare Metal ISO image.
The file can be unzipped on a Linux or Windows environment using appropriate commands to unzip
the file. For example on Linux use:
# unzip p30403643_1915000_Linux-x86-64.zip
Archive: p30403643_1915000_Linux-x86-64.zip
inflating: oda_bm_19.15.0.0.0_220512.iso <<<<< here is the iso image

Most actions here need to be carried out on both nodes.


The re-imaging can be performed in parallel on both nodes. There is no need to wait until one node is
complete before starting the other node.
Connect to ILOM from the Browser and login as root:

On successful login, the Browser will redirect to the Oracle Integrated Lights out Manager
home page. Click on the Launch button to continue:

Oracle Internal and Approved Partners Only Page 20 of 58 Version 2.1 – Created: 9 Jun 2022
Depending on the state of the system, there may be some message on the console. Click on
KVMS and then Storage as highlighted below:

Click on Add, then browser to choose the iso image as shown below:

Oracle Internal and Approved Partners Only Page 21 of 58 Version 2.1 – Created: 9 Jun 2022
Click on the file you've just selected and press Connect at the bottom of the screen.

After you have pressed Connect, if the Java window suddenly crashes or you are having issues
booting from the ISO image, perform the following steps:
• On the next reset, repeat the above steps.
• But this time, uncheck the SSL Enabled box on this screen and mount the ISO image
again.
• Proceed only after the ISO image successfully mounts.

Oracle Internal and Approved Partners Only Page 22 of 58 Version 2.1 – Created: 9 Jun 2022
Now that the ISO image has successfully been mounted, return to the ILOM web GUI to set
the boot device to CDROM and press Save.

Now click on the Power Control tab as shown below and choose Power Cycle from the drop-
down list and press Save.

Oracle Internal and Approved Partners Only Page 23 of 58 Version 2.1 – Created: 9 Jun 2022
The Database Appliance Bare Metal install will start after reboot. This process will continue
for about 30 minutes and then the following screen will be displayed:

NOTE: After the boot, the localhost login prompt will appear. Press enter after that for the
oak0/oak1 login, before proceeding to login using user name root and password welcome1.
However, if this is ODA has two storage trays attached, the localhost prompt will remain, this
is normal, it will change after configure-firstnet.

Only proceed to the next section when the re-imaging on both nodes is complete.

Oracle Internal and Approved Partners Only Page 24 of 58 Version 2.1 – Created: 9 Jun 2022
CONFIGURING THE NETWORK INTERFACE Node
0 1
Run the command configure-firstnet on both nodes.
NOTE: If the command is only run on one node, the deployment will give an error.
Respond to the prompts to configure the public network interface along with the netmask and
gateway IP address.
Example output below includes sample IP addresses for network, netmask and gateway. You
may get different options, other than btbond1/btbond3, depending on what type of network
cards and how many network cards are installed. For the X8-2 systems, breaking the bond is
not an option.

[root@oak0 ~]# odacli configure-firstnet


Select the Interface to configure the network on (btbond1 btbond3) [btbond1]:
Configure DHCP on btbond1 (yes/no) [no]:
INFO: You have chosen Static configuration
Use VLAN on btbond1 (yes/no) [no]:
Enter the IP address to configure : 10.145.230.224
Enter the Netmask address to configure : 255.255.252.0
Enter the Gateway address to configure [10.145.228.1] :
INFO: Restarting the network
[root@oak0 ~]#

Below is a table that shows the correlation between network cards and Linux
interface names.

For a system with two storage trays, you will see this:
Oracle Internal and Approved Partners Only Page 25 of 58 Version 2.1 – Created: 9 Jun 2022
[root@localhost ~]# odacli configure-firstnet
Enter node number: 0 OR 1
0
Do you want to continue yes | no :
yes

User has entered node number=0


Initialize MySQL - Done
Created symlink from /etc/systemd/system/multi-user.target.wants/oda-mysql.service
to /etc/systemd/system/oda-mysql.service.
Starting MySQL for ODA - Done
mysql: [Warning] Using a password on the command line interface can be insecure.
Enable server-side auth_socket authentication plugin for MySQL root user - Done
Generating RSA private key, 2048 bit long modulus
....................................................................................
+++
........................................................+++
e is 65537 (0x10001)
Generating a 2048 bit RSA private key
........+++
...........................+++
writing new private key to 'server-key.pem'
-----
writing RSA key
Signature ok
subject=/CN=oak0
Getting CA Private Key
Generating a 2048 bit RSA private key
........+++
.......................+++
writing new private key to 'dcsagent-client-key.pem'
-----
writing RSA key
Signature ok
subject=/CN=dcsagent
Getting CA Private Key
Generating a 2048 bit RSA private key
.............+++
...................................................+++
writing new private key to 'rhp-client-key.pem'
-----
writing RSA key
Signature ok
subject=/CN=rhp user
Getting CA Private Key
All certificates : OK
Generate certificates for CA, MySQL Server and Clients (DCS Agent & RHP) - Done
Enable SSL and Restart MySQL - Done
Create database dcsagentdb - Done
Create user dcsagent - Done
Grant privileges to user dcsagent - Done
Create database GHSUSER21 - Done
Create user GHSUSER21 - Done
Grant privileges to user GHSUSER21 - Done
Create schemas, users for CS Agent & RHP - Done
Warning: Unable to load '/usr/share/zoneinfo/iso3166.tab' as time zone. Skipping it.
Warning: Unable to load '/usr/share/zoneinfo/leapseconds' as time zone. Skipping it.
Warning: Unable to load '/usr/share/zoneinfo/tzdata.zi' as time zone. Skipping it.
Warning: Unable to load '/usr/share/zoneinfo/zone.tab' as time zone. Skipping it.
Warning: Unable to load '/usr/share/zoneinfo/zone1970.tab' as time zone. Skipping
it.
Load the timezone tables in mysql database - Done
Oracle Internal and Approved Partners Only Page 26 of 58 Version 2.1 – Created: 9 Jun 2022
MySQL installed version details :
/opt/oracle/dcs/mysql/bin/mysql Ver 8.0.22-commercial for Linux on x86_64 (MySQL
Enterprise Server - Commercial)
MySQL installation and configuration.....Completed
Resetting dcs-agent config file...
generating credentials for dcs-cli and dcs-agent
done generating credentials for dcs-cli and dcs-agent
Starting oda-zookeeper.service, oda-mysql.service and initdcs*.service services,
which can take 30 seconds
Created symlink from /etc/systemd/system/multi-user.target.wants/oda-
zookeeper.service to /etc/systemd/system/oda-zookeeper.service.
Created symlink from
/etc/systemd/system/multi-user.target.wants/initdcsadmin.service to
/etc/systemd/system/initdcsadmin.service.
Created symlink from
/etc/systemd/system/multi-user.target.wants/initdcsagent.service to
/etc/systemd/system/initdcsagent.service.
Created symlink from
/etc/systemd/system/multi-user.target.wants/initdcscontroller.service to
/etc/systemd/system/initdcscontroller.service.
Using bonding public interface (yes/no) [yes]:
Select the Interface to configure the network on (btbond1) [btbond1]:

Configure DHCP on btbond1 (yes/no) [no]:


INFO: You have chosen Static configuration
Use VLAN on btbond1 (yes/no) [no]:
Enter the IP address to configure : 10.145.228.237
Enter the Netmask address to configure : 255.255.252.0
Enter the Gateway address to configure[10.145.228.1] :
INFO: Restarting the network
INFO: Restarting TFA to configure the host name
Shutting down TFA
. . . . .
. . .
Successfully shutdown TFA..
Starting TFA..
Waiting up to 100 seconds for TFA to be started..
. . . . .
. . . . .
Successfully started TFA Process..
. . . . .
TFA Started and listening for commands

TFA has not synchronised across all nodes yet.


If an install or upgrade is in progress and the operation has not completed on all
nodes then please wait for completion and allow TFA 10 minutes to synchronize.
If no install or upgrade is in progress or you need TFA to synchronize now
Please run 'tfactl syncnodes' to generate and copy TFA Certificates.

Oracle Internal and Approved Partners Only Page 27 of 58 Version 2.1 – Created: 9 Jun 2022
UPLOADING PATCHES TO THE SERVER NODE Node

You need only upload patches to node0, not both nodes. 0

Now that you have an active network interface, you can use scp/ftp/filezilla to transfer the
files to the server node, for example:

# scp p30403662_1915000_Linux-x86-64.zip root@<ip of oda>:/u01


# scp p30403673_1915000_Linux-x86-64.zip root@<ip of oda>:/u01
# scp p23494992_1915000_Linux-x86-64.zip root@<ip of oda>:/u01
# scp p34069644_1915000_Linux-x86-64.zip root@<ip of oda>:/u01

**Important
Starting with 19.12, the file system layout has been changed, /opt has been
significantly reduced. Deploy with 1 RDBMS patch, then after deployment,
you can install the rest if needed.

If you're not able to have access to the client's network, then a good workaround is to configure em1 to
attach your laptop directly to the NET0 port.
To configure em1:
# ifconfig em1 10.10.10.1 netmask 255.255.255.0

# ifconfig em1
em1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.10.10.1 netmask 255.255.255.0 broadcast 10.10.10.255
ether 00:10:e0:eb:ec:4c txqueuelen 1000 (Ethernet)
RX packets 17352 bytes 1537513 (1.4 MiB)
RX errors 0 dropped 229 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device memory 0x9ca00000-9cafffff

On your laptop, you will also need to configure the network port to 10.10.10.2
NOTE: em1 gets unconfigured during deployment, so your direct network connection will get
disconnected shortly after you hit submit.
Now, you can transfer the patches over using winscp, filezilla, etc..

Oracle Internal and Approved Partners Only Page 28 of 58 Version 2.1 – Created: 9 Jun 2022
UNPACK AND INSTALL THE GI PATCH
Unzip the GI patch.
[root@oak0 u01]# unzip p30403673_1915000_Linux-x86-64.zip
Archive: p30403673_1915000_Linux-x86-64.zip
extracting: odacli-dcs-19.15.0.0.0-220425-GI-19.15.0.0.zip
inflating: README.txt

Update the repository.


[root@oak0 u01]# odacli update-repository -f /u01/odacli-dcs-19.15.0.0.0-
220425-GI-19.15.0.0.zip
{
"jobId" : "dc7db71e-e65e-48d5-b672-c317a12d7fe2",
"status" : "Created",
"message" : "/u01/odacli-dcs-19.15.0.0.0-220425-GI-19.15.0.0.zip",
"reports" : [ ],
"createTimestamp" : "May 20, 2022 00:05:03 AM UTC",
"resourceList" : [ ],
"description" : "Repository Update",
"updatedTime" : "May 20, 2022 00:05:03 AM UTC"
}

Confirm that the repository update has been successful.

[root@oak0 u01]# odacli describe-job -i "dc7db71e-e65e-48d5-b672-


c317a12d7fe2"

Job details
----------------------------------------------------------------
ID: dc7db71e-e65e-48d5-b672-c317a12d7fe2
Description: Repository Update
Status: Success
Created: May 20, 2022 12:05:03 AM UTC
Message: /u01/odacli-dcs-19.15.0.0.0-220425-GI-19.15.0.0.zip

Task Name Start Time End Time


Status
---------------------------------------- ----------------------------------- --------
Check AvailableSpace May 20, 2022 12:05:17 AM UTC May 20,
2022 12:05:17 AM UTC Success
Setting up ssh equivalance May 20, 2022 12:05:18 AM UTC May 20,
2022 12:05:18 AM UTC Success
Copy BundleFile May 20, 2022 12:05:19 AM UTC May 20,
2022 12:05:37 AM UTC Success
Validating CopiedFile May 20, 2022 12:05:37 AM UTC May 20,
2022 12:05:53 AM UTC Success
Unzip bundle May 20, 2022 12:05:59 AM UTC May 20,
2022 12:06:54 AM UTC Success
Unzip bundle May 20, 2022 12:06:55 AM UTC May 20,
2022 12:07:46 AM UTC Success
Removing ssh keys May 20, 2022 12:07:46 AM UTC May 20,
2022 12:07:47 AM UTC Success

Oracle Internal and Approved Partners Only Page 29 of 58 Version 2.1 – Created: 9 Jun 2022
Delete unneeded files to free up space:
[root@oak0 u01]# rm p30403673_1915000_Linux-x86-64.zip
rm: remove regular file ‘p30403673_1915000_Linux-x86-64.zip’? Yes

[root@oak0 u01]# rm README.txt


rm: remove regular file ‘README.txt’? Yes

UNPACK AND INSTALL THE SERVER/STORAGE PATCH ON NODE 0 ONLY


Patches only need to be installed on Node0 only. The patches and files will propagate to the
other node automatically.
Unzip the patch files:
[root@oak0 u01]# unzip p34069644_1915000_Linux-x86-64.zip
Archive: p34069644_1915000_Linux-x86-64.zip
extracting: oda-sm-19.15.0.0.0-220512-server.zip
inflating: README.txt

Update the repository.


[root@oak0 u01]# odacli update-repository -f /u01/oda-sm-19.15.0.0.0-220512-
server.zip
{
"jobId" : "5c9d9baa-0913-40ff-81ad-b3643ab2453f",
"status" : "Created",
"message" : "/u01/oda-sm-19.15.0.0.0-220512-server.zip",
"reports" : [ ],
"createTimestamp" : "May 20, 2022 00:11:30 AM UTC",
"resourceList" : [ ],
"description" : "Repository Update",
"updatedTime" : "May 20, 2022 00:11:30 AM UTC"
}

Oracle Internal and Approved Partners Only Page 30 of 58 Version 2.1 – Created: 9 Jun 2022
Confirm the update was successful:
[root@oak0 u01]# odacli describe-job -i "5c9d9baa-0913-40ff-81ad-b3643ab2453f"

Job details
----------------------------------------------------------------
ID: 5c9d9baa-0913-40ff-81ad-b3643ab2453f
Description: Repository Update
Status: Success
Created: May 20, 2022 12:11:30 AM UTC
Message: /u01/oda-sm-19.15.0.0.0-220512-server.zip

Task Name Start Time End


Time Status
---------------------------------------- ----------------------------------- -------
Check AvailableSpace May 20, 2022 12:11:37 AM UTC May 20,
2022 12:11:37 AM UTC Success
Setting up ssh equivalance May 20, 2022 12:11:38 AM UTC May 20,
2022 12:11:38 AM UTC Success
Copy BundleFile May 20, 2022 12:11:39 AM UTC May 20,
2022 12:11:47 AM UTC Success
Validating CopiedFile May 20, 2022 12:11:51 AM UTC May 20,
2022 12:11:58 AM UTC Success
Unzip bundle May 20, 2022 12:11:58 AM UTC May 20,
2022 12:12:24 AM UTC Success
Unzip bundle May 20, 2022 12:12:24 AM UTC May 20,
2022 12:12:50 AM UTC Success
Removing ssh keys May 20, 2022 12:12:50 AM UTC May 20,
2022 12:12:50 AM UTC Success

Delete unneeded files to free up space for deployment.


[root@oak0 u01]# rm p34069644_1915000_Linux-x86-64.zip
rm: remove regular file ‘p34069644_1915000_Linux-x86-64.zip’? Yes

[root@oak0 u01]# rm README.txt


rm: remove regular file ‘README.txt’? yes

Oracle Internal and Approved Partners Only Page 31 of 58 Version 2.1 – Created: 9 Jun 2022
APPLYING THE SERVER UPDATE
It is a requirement to first run prepatchreport, before applying the server update.

[root@oak0 u01]# odacli create-prepatchreport -s -v 19.15.0.0.0

Job details
----------------------------------------------------------------
ID: ab1928cd-ff1c-4e6d-b433-b5c3cd633afa
Description: Patch pre-checks for [OS, ILOM, SERVER]
Status: Created
Created: May 20, 2022 12:15:47 AM UTC
Message: Use 'odacli describe-prepatchreport -i ab1928cd-ff1c-4e6d-
b433-b5c3cd633afa' to check details of results

Task Name Start Time End Time


Status
---------------------------------------- ----------------------------------- -------

Confirm the prepatchreport has completed:


[root@oak0 u01]# odacli describe-job -i ab1928cd-ff1c-4e6d-b433-b5c3cd633afa

Job details
----------------------------------------------------------------
ID: ab1928cd-ff1c-4e6d-b433-b5c3cd633afa
Description: Patch pre-checks for [OS, ILOM, SERVER]
Status: Success
Created: May 20, 2022 12:15:47 AM UTC
Message: Use 'odacli describe-prepatchreport -i <ID>' to check
prepatch results

Task Name Start Time End Time


Status
---------------------------------------- ----------------------------------- -------
Run patching pre-checks May 20, 2022 12:15:49 AM UTC May 20,
2022 12:16:48 AM UTC Success

Alternatively, you can use “odacli describe-prepatchreport -i ab1928cd-ff1c-4e6d-


b433-b5c3cd633afa”

Apply the server patch. The iloms and nodes may reboot, be patient and do not interrupt the
process.
[root@oak0 u01]# odacli update-server -v 19.15.0.0.0
{
"jobId" : "d26c66ce-5a31-4dc8-b6b9-e412619b7446",
"status" : "Created",
"message" : "Success of server update will trigger reboot of the node after 4-5
minutes. Please wait until the node reboots.",
"reports" : [ ],
"createTimestamp" : "May 20, 2022 00:18:14 AM UTC",
"resourceList" : [ ],
"description" : "Server Patching",
"updatedTime" : "May 20, 2022 00:18:14 AM UTC"

Oracle Internal and Approved Partners Only Page 32 of 58 Version 2.1 – Created: 9 Jun 2022
}

Confirm the update was successful.

[root@oak0 ~]# odacli describe-job -i "d26c66ce-5a31-4dc8-b6b9-e412619b7446"

Job details
----------------------------------------------------------------
ID: d26c66ce-5a31-4dc8-b6b9-e412619b7446
Description: Server Patching
Status: Success
Created: May 20, 2022 12:18:14 AM UTC
Message:

Task Name Start Time End Time


Status
---------------------------------------- ----------------------------------- -------
Validate ILOM server reachable May 20, 2022 12:18:16 AM UTC May 20,
2022 12:18:16 AM UTC Success
Validate ILOM server reachable May 20, 2022 12:18:16 AM UTC May 20,
2022 12:18:16 AM UTC Success
SSH equivalance setup May 20, 2022 12:18:16 AM UTC May 20,
2022 12:18:16 AM UTC Success
Stop DCS Admin May 20, 2022 12:18:19 AM UTC May 20,
2022 12:18:19 AM UTC Success
Stop DCS Admin May 20, 2022 12:18:19 AM UTC May 20,
2022 12:18:19 AM UTC Success
Generate mTLS certificates May 20, 2022 12:18:19 AM UTC May 20,
2022 12:18:20 AM UTC Success
Generate mTLS certificates May 20, 2022 12:18:19 AM UTC May 20,
2022 12:18:21 AM UTC Success
Exporting Public Keys May 20, 2022 12:18:21 AM UTC May 20,
2022 12:18:22 AM UTC Success
Exporting Public Keys May 20, 2022 12:18:21 AM UTC May 20,
2022 12:18:22 AM UTC Success
Copy Public Key to Remote node May 20, 2022 12:18:22 AM UTC May 20,
2022 12:18:23 AM UTC Success
Copy Public Key to Remote node May 20, 2022 12:18:23 AM UTC May 20,
2022 12:18:24 AM UTC Success
Creating Trust Store May 20, 2022 12:18:24 AM UTC May 20,
2022 12:18:26 AM UTC Success
Creating Trust Store May 20, 2022 12:18:24 AM UTC May 20,
2022 12:18:26 AM UTC Success
Update config files May 20, 2022 12:18:26 AM UTC May 20,
2022 12:18:26 AM UTC Success
Update config files May 20, 2022 12:18:26 AM UTC May 20,

Oracle Internal and Approved Partners Only Page 33 of 58 Version 2.1 – Created: 9 Jun 2022
2022 12:18:26 AM UTC Success
Restart DCS Admin May 20, 2022 12:18:26 AM UTC May 20,
2022 12:18:46 AM UTC Success
Restart DCS Admin May 20, 2022 12:18:26 AM UTC May 20,
2022 12:18:46 AM UTC Success
Creating repositories using yum May 20, 2022 12:18:50 AM UTC May 20,
2022 12:18:52 AM UTC Success
Updating YumPluginVersionLock rpm May 20, 2022 12:18:52 AM UTC May 20,
2022 12:18:52 AM UTC Success
Applying OS Patches May 20, 2022 12:18:52 AM UTC May 20,
2022 12:18:53 AM UTC Success
Creating repositories using yum May 20, 2022 12:18:54 AM UTC May 20,
2022 12:18:55 AM UTC Success
Applying HMP Patches May 20, 2022 12:18:55 AM UTC May 20,
2022 12:18:56 AM UTC Success
Patch location validation May 20, 2022 12:18:56 AM UTC May 20,
2022 12:18:56 AM UTC Success
Patch location validation May 20, 2022 12:18:56 AM UTC May 20,
2022 12:18:56 AM UTC Success
oda-hw-mgmt upgrade May 20, 2022 12:18:56 AM UTC May 20,
2022 12:18:56 AM UTC Success
oda-hw-mgmt upgrade May 20, 2022 12:18:56 AM UTC May 20,
2022 12:18:56 AM UTC Success
Applying Firmware Disk Patches May 20, 2022 12:19:00 AM UTC May 20,
2022 12:19:04 AM UTC Success
Applying Firmware Controller Patches May 20, 2022 12:20:09 AM UTC May 20,
2022 12:20:16 AM UTC Success
Checking Ilom patch Version May 20, 2022 12:20:17 AM UTC May 20,
2022 12:20:17 AM UTC Success
Checking Ilom patch Version May 20, 2022 12:20:17 AM UTC May 20,
2022 12:20:17 AM UTC Success
Patch location validation May 20, 2022 12:20:17 AM UTC May 20,
2022 12:20:17 AM UTC Success
Patch location validation May 20, 2022 12:20:17 AM UTC May 20,
2022 12:20:17 AM UTC Success
Save password in Wallet May 20, 2022 12:20:18 AM UTC May 20,
2022 12:20:19 AM UTC Success
Disabling IPMI v2 May 20, 2022 12:20:19 AM UTC May 20,
2022 12:20:20 AM UTC Success
Apply Ilom patch May 20, 2022 12:20:20 AM UTC May 20,
2022 12:20:20 AM UTC Success
Apply Ilom patch May 20, 2022 12:20:20 AM UTC May 20,
2022 12:28:26 AM UTC Success
Copying Flash Bios to Temp location May 20, 2022 12:28:26 AM UTC May 20,
2022 12:28:26 AM UTC Success
Copying Flash Bios to Temp location May 20, 2022 12:28:26 AM UTC May 20,
2022 12:28:26 AM UTC Success
Update System version May 20, 2022 12:28:26 AM UTC May 20,
2022 12:28:26 AM UTC Success

Oracle Internal and Approved Partners Only Page 34 of 58 Version 2.1 – Created: 9 Jun 2022
Update System version May 20, 2022 12:28:26 AM UTC May 20,
2022 12:28:26 AM UTC Success
Cleanup JRE Home May 20, 2022 12:28:26 AM UTC May 20,
2022 12:28:26 AM UTC Success
Update lvm.conf file May 20, 2022 12:29:18 AM UTC May 20,
2022 12:29:18 AM UTC Success
Update lvm.conf file May 20, 2022 12:29:18 AM UTC May 20,
2022 12:29:18 AM UTC Success
Update previous workarounds May 20, 2022 12:29:18 AM UTC May 20,
2022 12:29:18 AM UTC Success
Update previous workarounds May 20, 2022 12:29:18 AM UTC May 20,
2022 12:29:18 AM UTC Success
preRebootNode Actions May 20, 2022 12:29:18 AM UTC May 20,
2022 12:29:18 AM UTC Success
preRebootNode Actions May 20, 2022 12:29:18 AM UTC May 20,
2022 12:29:18 AM UTC Success
Reboot Ilom May 20, 2022 12:29:18 AM UTC May 20,
2022 12:29:18 AM UTC Success
Reboot Ilom May 20, 2022 12:29:18 AM UTC May 20,
2022 12:29:18 AM UTC Success

APPLY THE STORAGE PATCH


Update storage components:

[root@oak0 ~]# odacli update-storage -v 19.15.0.0.0


{
"jobId" : "f331acb3-5a95-4238-a059-75a29b0f1160",
"status" : "Created",
"message" : "Success of Storage Update may trigger reboot of node after 4-5
minutes. Please wait till node restart",
"reports" : [ ],
"createTimestamp" : "May 20, 2022 00:51:08 AM UTC",
"resourceList" : [ ],
"description" : "Storage Firmware Patching",
"updatedTime" : "May 20, 2022 00:51:08 AM UTC"
}

Confirm the update was successful. The servers may reboot during this procedure, depending
on which component gets updated.

[root@oak0 ~]# odacli describe-job -i "f331acb3-5a95-4238-a059-75a29b0f1160"

Job details
----------------------------------------------------------------
ID: f331acb3-5a95-4238-a059-75a29b0f1160
Description: Storage Firmware Patching
Status: Success
Oracle Internal and Approved Partners Only Page 35 of 58 Version 2.1 – Created: 9 Jun 2022
Created: May 20, 2022 12:51:08 AM UTC
Message:

Task Name Start Time End Time


Status
---------------------------------------- ----------------------------------- --------
Applying Firmware Disk Patches May 20, 2022 12:51:21 AM UTC May 20,
2022 12:51:24 AM UTC Success
preRebootNode Actions May 20, 2022 12:51:25 AM UTC May 20,
2022 12:51:25 AM UTC Success
preRebootNode Actions May 20, 2022 12:51:25 AM UTC May 20,
2022 12:51:25 AM UTC Success
Reboot Ilom May 20, 2022 12:51:25 AM UTC May 20,
2022 12:51:26 AM UTC Success
Reboot Ilom May 20, 2022 12:51:26 AM UTC May 20,
2022 12:51:26 AM UTC Success

Confirm that all firmware components are up-to-date.

[root@oak0 ~]# odacli describe-component


System Version
---------------
19.15.0.0.0

System node Name


---------------
oak0

Local System Version


---------------
19.15.0.0.0

Component Installed Version Available Version


---------------------------------------- -------------------- --------------------
OAK
19.15.0.0.0 up-to-date
DCSCONTROLLER
19.15.0.0.0 up-to-date
DCSCLI
19.15.0.0.0 up-to-date
DCSAGENT
19.15.0.0.0 up-to-date
DCSADMIN
19.15.0.0.0 up-to-date
OS
7.9 up-to-date
ILOM
5.0.2.24.r141466 up-to-date
BIOS
52050300 up-to-date
LOCAL CONTROLLER FIRMWARE {
[c4]
214.2.271.9 up-to-date
[c6]
80000681 up-to-date
}
SHARED CONTROLLER FIRMWARE
16.00.08.00 up-to-date
FIRMWAREEXPANDER
Oracle Internal and Approved Partners Only Page 36 of 58 Version 2.1 – Created: 9 Jun 2022
0310 up-to-date
LOCAL DISK FIRMWARE
1120 1132
SHARED DISK FIRMWARE {
[c0d0,c0d1,c0d2,c0d3,c0d4,c0d5,c1d0,
c1d1,c1d2,c1d3,c1d4,c1d5] A959 up-to-date
[c0d6,c0d7,c0d8,c0d9,c0d10,c0d11,c0d12,
c0d13,c0d14,c0d15,c0d16,c0d17,c0d18,
c0d19,c0d20,c0d21,c0d22,c0d23,c1d6,c1d7,
c1d8,c1d9,c1d10,c1d11,c1d12,c1d13,c1d14,
c1d15,c1d16,c1d17,c1d18,c1d19,c1d20,
c1d21,c1d22,c1d23] A144 up-to-date
[c2d0,c2d1]
1120 1132
}
HMP
2.4.8.0.601 up-to-date
System node Name
---------------
oak1

Local System Version


---------------
19.15.0.0.0

Component Installed Version Available Version


---------------------------------------- -------------------- --------------------
OAK
19.15.0.0.0 up-to-date
DCSCONTROLLER
19.15.0.0.0 up-to-date
DCSCLI
19.15.0.0.0 up-to-date
DCSAGENT
19.15.0.0.0 up-to-date
DCSADMIN
19.15.0.0.0 up-to-date
OS
7.9 up-to-date
ILOM
5.0.2.24.r141466 up-to-date
BIOS
52050300 up-to-date
LOCAL CONTROLLER FIRMWARE {
[c6]
80000681 up-to-date
}
SHARED CONTROLLER FIRMWARE
16.00.08.00 up-to-date
FIRMWAREEXPANDER
0310 up-to-date
LOCAL DISK FIRMWARE
1102 up-to-date
SHARED DISK FIRMWARE {
[c0d0,c0d1,c0d2,c0d3,c0d4,c0d5,c1d0,
c1d1,c1d2,c1d3,c1d4,c1d5] A959 up-to-date
[c0d6,c0d7,c0d8,c0d9,c0d10,c0d11,c0d12,
c0d13,c0d14,c0d15,c0d16,c0d17,c0d18,
c0d19,c0d20,c0d21,c0d22,c0d23,c1d6,c1d7,
c1d8,c1d9,c1d10,c1d11,c1d12,c1d13,c1d14,
c1d15,c1d16,c1d17,c1d18,c1d19,c1d20,
c1d21,c1d22,c1d23] A144 up-to-date
[c2d0,c2d1]
Oracle Internal and Approved Partners Only Page 37 of 58 Version 2.1 – Created: 9 Jun 2022
1102 up-to-date
}
HMP
2.4.8.0.601 up-to-date

Note: The FIRMWAREDISK entry for c2d0 and c2d1 may report something other than 1120,
this is ok, since we do not update the local disks on this version. You may ignore this line
entry, the local disk does not have to be updated.

Oracle Internal and Approved Partners Only Page 38 of 58 Version 2.1 – Created: 9 Jun 2022
OVERVIEW OF OPTIONAL DATABASE VERSIONS & REQUIRED PATCHES
The web provisioning tool will only allow you to deploy just one database version on initial
deployment. This section will go through installing both the database versions, but not all are
required. Most customers will only use one database version. The database version that the customer
wants to deploy will determine which patch to install:

• p30403662_1915000_Linux-x86-64.zip – Oracle RDBMS Software Clone file for creating any new Oracle
19.15 database homes.

• p23494992_1915000_Linux-x86-64.zip – Oracle RDBMS Software Clone file for creating any new 12.1.0.2
database homes.

Due to the change in file system size, you can only run update-repository on one RDBMS clone
file above. After deployment, then you can install more clone files.

OPTIONAL: DB CLONE FILE TO DEPLOY A 19.15.0.0 DATABASE


Unpack the DB clone file for deploying a 19.15.0.0 database:
[root@oak0 u01]# unzip p30403662_1915000_Linux-x86-64.zip
Archive: p30403662_1915000_DB_Linux-x86-64.zip
extracting: odacli-dcs-19.15.0.0.0-220425-DB-19.15.0.0.zip
inflating: README.txt

Update the repository:


[root@oak0 u01]# odacli update-repository -f /u01/odacli-dcs-19.15.0.0.0-
220425-DB-19.15.0.0.zip
{
"jobId" : "c2b9e934-40fd-4858-9c57-538eaf1826cc",
"status" : "Created",
"message" : "/u01/odacli-dcs-19.15.0.0.0-220425-DB-19.15.0.0.zip",
"reports" : [ ],
"createTimestamp" : "May 20, 2022 13:21:35 PM UTC",
"resourceList" : [ ],
"description" : "Repository Update",
"updatedTime" : "May 20, 2022 13:21:35 PM UTC"
}

Confirm that the repository update has been successful:


[root@oak0 u01]# odacli describe-job -i "c2b9e934-40fd-4858-9c57-
538eaf1826cc"

Job details
----------------------------------------------------------------
ID: c2b9e934-40fd-4858-9c57-538eaf1826cc
Description: Repository Update
Status: Success

Oracle Internal and Approved Partners Only Page 39 of 58 Version 2.1 – Created: 9 Jun 2022
Created: May 20, 2022 1:21:35 PM UTC
Message: /u01/odacli-dcs-19.15.0.0.0-220425-DB-19.15.0.0.zip

Task Name Start Time End Time


Status
---------------------------------------- ----------------------------------- -------
Check AvailableSpace May 20, 2022 1:21:46 PM UTC May 20,
2022 1:21:46 PM UTC Success
Setting up ssh equivalance May 20, 2022 1:21:46 PM UTC May 20,
2022 1:21:46 PM UTC Success
Copy BundleFile May 20, 2022 1:21:47 PM UTC May 20,
2022 1:22:02 PM UTC Success
Validating CopiedFile May 20, 2022 1:22:02 PM UTC May 20,
2022 1:22:12 PM UTC Success
Unzip bundle May 20, 2022 1:22:14 PM UTC May 20,
2022 1:22:51 PM UTC Success
Unzip bundle May 20, 2022 1:22:51 PM UTC May 20,
2022 1:23:28 PM UTC Success
Removing ssh keys May 20, 2022 1:23:28 PM UTC May 20,
2022 1:23:28 PM UTC Success

Delete unneeded files to free up space for deployment:


[root@oak0 u01]# rm p30403662_1915000_Linux-x86-64.zip
rm: remove regular file ‘p30403662_1915000_Linux-x86-64.zip’? Yes

[root@oak0 u01]# rm README.txt


rm: remove regular file ‘README.txt’? yes

Oracle Internal and Approved Partners Only Page 40 of 58 Version 2.1 – Created: 9 Jun 2022
OPTIONAL: DB CLONE FILE TO DEPLOY A 12.1.0.2 DATABASE
Unpack the DB clone file for deploying a 12.1.0.2 database:

# unzip p23494992_1915000_Linux-x86-64.zip
Archive: p23494992_1915000_Linux-x86-64.zip
extracting: odacli-dcs-19.15.0.0.0-220412-DB-12.1.0.2.zip
inflating: README.txt

Apply the update:


# odacli update-repository -f /u01/odacli-dcs-19.15.0.0.0-220412-DB-
12.1.0.2.zip
{
"jobId" : "30f78eb2-8291-4ce0-80ed-208a730dbaad",
"status" : "Created",
"message" : "/u01/odacli-dcs-19.15.0.0.0-220412-DB-12.1.0.2.zip",
"reports" : [ ],
"createTimestamp" : "May 23, 2022 12:18:02 PM MDT",
"resourceList" : [ ],
"description" : "Repository Update",
"updatedTime" : "May 23, 2022 12:18:02 PM MDT"
}

Oracle Internal and Approved Partners Only Page 41 of 58 Version 2.1 – Created: 9 Jun 2022
Confirm the job was successful:

# odacli describe-job -i "30f78eb2-8291-4ce0-80ed-208a730dbaad"

Job details
----------------------------------------------------------------
ID: 30f78eb2-8291-4ce0-80ed-208a730dbaad
Description: Repository Update
Status: Success
Created: May 23, 2022 12:18:02 PM MDT
Message: /u01/odacli-dcs-19.15.0.0.0-220412-DB-12.1.0.2.zip

Task Name Start Time End Time


Status
---------------------------------------- -----------------------------------
Check AvailableSpace May 23, 2022 12:18:23 PM MDT May 23,
2022 12:18:23 PM MDT Success
Setting up ssh equivalance May 23, 2022 12:18:23 PM MDT May 23,
2022 12:18:23 PM MDT Success
Copy BundleFile May 23, 2022 12:18:24 PM MDT May 23,
2022 12:18:52 PM MDT Success
Validating CopiedFile May 23, 2022 12:18:52 PM MDT May 23,
2022 12:19:12 PM MDT Success
Unzip bundle May 23, 2022 12:19:12 PM MDT May 23,
2022 12:20:21 PM MDT Success
Unzip bundle May 23, 2022 12:20:21 PM MDT May 23,
2022 12:21:31 PM MDT Success
Removing ssh keys May 23, 2022 12:21:31 PM MDT May 23,
2022 12:21:32 PM MDT Success

Delete unneeded files to free up space for deployment:


# rm p23494992_1915000_Linux-x86-64.zip
rm: remove regular file ‘p23494992_1915000_Linux-x86-64.zip’? Yes

# rm README.txt
rm: remove regular file ‘README.txt’? yes

Oracle Internal and Approved Partners Only Page 42 of 58 Version 2.1 – Created: 9 Jun 2022
DEPLOYING THE ORACLE DATABASE APPLIANCE
You are now ready to deploy the Oracle Database Appliance. Monitor the progress by:
# tail -f /opt/oracle/dcs/log/dcs-agent.log

If you're not able to have access to the client's network, then a good workaround is to
configure em1 to attach your laptop directly to the NET0 port.
To configure em1:
# ifconfig em1 10.10.10.1 netmask 255.255.255.0

# ifconfig em1
em1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.10.10.1 netmask 255.255.255.0 broadcast 10.10.10.255
ether 00:10:e0:eb:ec:4c txqueuelen 1000 (Ethernet)
RX packets 17352 bytes 1537513 (1.4 MiB)
RX errors 0 dropped 229 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device memory 0x9ca00000-9cafffff
On your laptop, you will also need to configure the network port to 10.10.10.2

• NOTE: em1 gets unconfigured during deployment, so your direct network connection
will get disconnected shortly after you hit submit.
Using the Chrome or IE browser, enter the
following URL and log into the ODA:
• https://<ipaddress-node>:7093/
mgmt/index.html
• The first screen you will see is a screen
asking you to change the password.
• The password must contain at least
two uppercase and two lowercase
letters, two numbers (0-9), and two
special characters (#-_). Password
must have a minimum of 9 characters
and a maximum of 30 characters.

Oracle Internal and Approved Partners Only Page 43 of 58 Version 2.1 – Created: 9 Jun 2022
After the password has been changed, you will see the login screen. Login name is oda-admin

NOTE: If the login fails due to an authentication issue, make sure that network between the
ODA and the installer’s laptop has ports 7070 and 7093 open. Unless these ports are opened
up, you will continue to get authentication errors and will not be able to login until the
customer opens up these ports.
If opening up these ports is not an option, you can connect directly to the ODA by configuring
em1 up and using your laptop to connect directly.

On the first screen of a new system, you will


see this, click on Create Appliance.

Oracle Internal and Approved Partners Only Page 44 of 58 Version 2.1 – Created: 9 Jun 2022
You will now be presented with a one-page web provisioning tool. Mandatory fields have a *.
The host name…
• must begin with an alphabetic character
• can contain alphanumeric characters and dashes (-)
• cannot exceed 30 characters
• cannot start with a number or dash (-)
• cannot end with dash (-)

Even if the ILOM network has been configured, it is good to also input the values here.

Oracle Internal and Approved Partners Only Page 45 of 58 Version 2.1 – Created: 9 Jun 2022
Scroll down to the next section beginning at “Customize Users & Groups”.
If you select Yes, you will need to fill in the following fields:

Creating an initial database is not mandatory for deployment, as this can be done later by the
customer using the odacli create-database command. Additional databases can also be
created with odacli create-database.

If you answer Yes above, you will need to fill in the following fields. The database name
must begin with an alpha character and contain only alphanumeric characters. The database
name must NOT exceed 8 characters.

For customers who prefer extra security during TFA collections. The defaults is No.

Oracle Internal and Approved Partners Only Page 46 of 58 Version 2.1 – Created: 9 Jun 2022
In order to use the External ASR Manager, the
customer will need an existing ASR Manager
IP. While using an External ASR Manager is
highly recommended, it is supported to use the
internal ASR manager.
If Yes, the ASR User Name and Password is required. Make sure this is the customer’s
username/password.
If Yes to HTTP Proxy, the Proxy Server Name and Proxy Port is required.
If Yes to HTTP Proxy Requires Authentication, Proxy User Name and Proxy Password is
required.

It is a good idea to first Save Configuration, so that you have it in case you need to redeploy.

Then press Submit.


Select Yes to confirm:

The very first step is Setting up Network.


You will lose network connectivity for a few
minutes until this is done.

You should see a pop-up like this:

During the initial stages of deployment, network connectivity will be lost. Wait a few minutes before
accessing the hosts again via the network.

Oracle Internal and Approved Partners Only Page 47 of 58 Version 2.1 – Created: 9 Jun 2022
MONITOR DEPLOYMENT PROGRESS
To monitor deployment progress, you can monitor the dcs-agent.log file:
# tail -f /opt/oracle/dcs/log/dcs-agent.log

If there is an error during deployment, we will see it in the dcs-agent.log file.


Furthermore, you can run the odacli command to monitor/check status.
To monitor the deployment progress using command-line, use the job ID identified for
“Provisioning service creation”.
To get the job ID:

[root@oak0 opt]# odacli list-jobs

ID Description
Created Status
---------------------------------------- ---------------------------------------
dc7db71e-e65e-48d5-b672-c317a12d7fe2 Repository Update
May 20, 2022 12:05:03 AM GMT Success
5c9d9baa-0913-40ff-81ad-b3643ab2453f Repository Update
May 20, 2022 12:11:30 AM GMT Success
ab1928cd-ff1c-4e6d-b433-b5c3cd633afa Patch pre-checks for [OS, ILOM, SERVER]
May 20, 2022 12:15:47 AM GMT Success
d26c66ce-5a31-4dc8-b6b9-e412619b7446 Server Patching
May 20, 2022 12:18:14 AM GMT Success
f331acb3-5a95-4238-a059-75a29b0f1160 Storage Firmware Patching
May 20, 2022 12:51:08 AM GMT Success
c2b9e934-40fd-4858-9c57-538eaf1826cc Repository Update
May 20, 2022 1:21:35 PM GMT Success
51c34160-1de8-482e-be75-ae0c77e28ed8 Repository Update
May 21, 2022 2:04:50 PM GMT Created
e5966bbd-0e75-4ce1-b569-c28431a2dd2a User creation(oda-admin)
May 21, 2022 2:07:25 PM GMT Success
3be7c61d-49d1-44ba-b27b-2708fa84d2c2 Provisioning service creation
May 21, 2022 2:18:32 PM GMT Running

To find status of deployment:

[root@oak1 ~]# odacli describe-job -i 3be7c61d-49d1-44ba-b27b-2708fa84d2c2

Job details
----------------------------------------------------------------
ID: 3be7c61d-49d1-44ba-b27b-2708fa84d2c2
Description: Provisioning service creation
Status: Success
Created: May 21, 2022 3:22:43 PM GMT
Message:

Task Name Start Time End


Time Status
---------------------------------------- ----------------------------------- ------
Setting up Network May 21, 2022 3:22:52 PM GMT May 21,
2022 3:22:52 PM GMT Success
Setting up Network May 21, 2022 3:23:09 PM GMT May 21,

Oracle Internal and Approved Partners Only Page 48 of 58 Version 2.1 – Created: 9 Jun 2022
2022 3:23:09 PM GMT Success
Setting up Network May 21, 2022 3:23:20 PM GMT May 21,
2022 3:23:20 PM GMT Success
Setting up Network May 21, 2022 3:23:38 PM GMT May 21,
2022 3:23:38 PM GMT Success
Setting up Vlan May 21, 2022 3:23:56 PM GMT May 21,
2022 3:23:57 PM GMT Success
Setting up Network May 21, 2022 3:24:15 PM GMT May 21,
2022 3:24:15 PM GMT Success
Setting up Network May 21, 2022 3:24:37 PM GMT May 21,
2022 3:24:37 PM GMT Success
Setting up Network May 21, 2022 3:24:53 PM GMT May 21,
2022 3:24:53 PM GMT Success
Setting up Vlan May 21, 2022 3:25:09 PM GMT May 21,
2022 3:25:10 PM GMT Success
Setting up Network May 21, 2022 3:25:26 PM GMT May 21,
2022 3:25:26 PM GMT Success
OS usergroup 'asmdba'creation May 21, 2022 3:25:48 PM GMT May 21,
2022 3:25:48 PM GMT Success
OS usergroup 'asmoper'creation May 21, 2022 3:25:48 PM GMT May 21,
2022 3:25:48 PM GMT Success
OS usergroup 'asmadmin'creation May 21, 2022 3:25:48 PM GMT May 21,
2022 3:25:48 PM GMT Success
OS usergroup 'dba'creation May 21, 2022 3:25:48 PM GMT May 21,
2022 3:25:48 PM GMT Success
OS usergroup 'dbaoper'creation May 21, 2022 3:25:48 PM GMT May 21,
2022 3:25:48 PM GMT Success
OS usergroup 'oinstall'creation May 21, 2022 3:25:48 PM GMT May 21,
2022 3:25:48 PM GMT Success
OS user 'grid'creation May 21, 2022 3:25:48 PM GMT May 21,
2022 3:25:48 PM GMT Success
OS user 'oracle'creation May 21, 2022 3:25:48 PM GMT May 21,
2022 3:25:49 PM GMT Success
Default backup policy creation May 21, 2022 3:25:49 PM GMT May 21,
2022 3:25:49 PM GMT Success
Backup config metadata persist May 21, 2022 3:25:49 PM GMT May 21,
2022 3:25:49 PM GMT Success
Grant permission to RHP files May 21, 2022 3:25:49 PM GMT May 21,
2022 3:25:49 PM GMT Success
Adding USER SSH_EQUIVALENCE May 21, 2022 3:25:49 PM GMT May 21,
2022 3:25:50 PM GMT Success
Adding USER SSH_EQUIVALENCE May 21, 2022 3:25:50 PM GMT May 21,
2022 3:25:51 PM GMT Success
Adding USER SSH_EQUIVALENCE May 21, 2022 3:25:51 PM GMT May 21,
2022 3:25:53 PM GMT Success
Add SYSNAME in Env May 21, 2022 3:25:53 PM GMT May 21,
2022 3:25:53 PM GMT Success
Install oracle-ahf May 21, 2022 3:25:53 PM GMT May 21,
2022 3:26:53 PM GMT Success
Stop DCS Admin May 21, 2022 3:26:57 PM GMT May 21,
2022 3:26:57 PM GMT Success
Stop DCS Admin May 21, 2022 3:26:57 PM GMT May 21,
2022 3:26:57 PM GMT Success
Generate mTLS certificates May 21, 2022 3:26:57 PM GMT May 21,
2022 3:27:12 PM GMT Success
Generate mTLS certificates May 21, 2022 3:26:57 PM GMT May 21,
2022 3:27:11 PM GMT Success
Exporting Public Keys May 21, 2022 3:27:12 PM GMT May 21,
2022 3:27:13 PM GMT Success
Exporting Public Keys May 21, 2022 3:27:12 PM GMT May 21,
2022 3:27:13 PM GMT Success
Copy Public Key to Remote node May 21, 2022 3:27:13 PM GMT May 21,
2022 3:27:14 PM GMT Success
Oracle Internal and Approved Partners Only Page 49 of 58 Version 2.1 – Created: 9 Jun 2022
Copy Public Key to Remote node May 21, 2022 3:27:14 PM GMT May 21,
2022 3:27:15 PM GMT Success
Creating Trust Store May 21, 2022 3:27:15 PM GMT May 21,
2022 3:27:17 PM GMT Success
Creating Trust Store May 21, 2022 3:27:15 PM GMT May 21,
2022 3:27:17 PM GMT Success
Update config files May 21, 2022 3:27:17 PM GMT May 21,
2022 3:27:17 PM GMT Success
Update config files May 21, 2022 3:27:17 PM GMT May 21,
2022 3:27:17 PM GMT Success
Restart DCS Admin May 21, 2022 3:27:17 PM GMT May 21,
2022 3:27:37 PM GMT Success
Restart DCS Admin May 21, 2022 3:27:17 PM GMT May 21,
2022 3:27:37 PM GMT Success
Grid home creation May 21, 2022 3:27:38 PM GMT May 21,
2022 3:30:46 PM GMT Success
Creating GI home directories May 21, 2022 3:27:38 PM GMT May 21,
2022 3:27:38 PM GMT Success
Extract GI clone May 21, 2022 3:27:38 PM GMT May 21,
2022 3:29:12 PM GMT Success
Extract GI clone May 21, 2022 3:29:12 PM GMT May 21,
2022 3:30:46 PM GMT Success
Storage discovery May 21, 2022 3:30:46 PM GMT May 21,
2022 3:41:14 PM GMT Success
Creating wallet for Root User May 21, 2022 3:41:14 PM GMT May 21,
2022 3:41:18 PM GMT Success
Creating wallet for ASM Client May 21, 2022 3:41:18 PM GMT May 21,
2022 3:41:21 PM GMT Success
Grid stack creation May 21, 2022 3:41:22 PM GMT May 21,
2022 4:01:54 PM GMT Success
Provisioning GI with RHP May 21, 2022 3:41:22 PM GMT May 21,
2022 3:57:22 PM GMT Success
Updating GIHome version May 21, 2022 3:57:23 PM GMT May 21,
2022 3:57:26 PM GMT Success
Updating GIHome version May 21, 2022 3:57:23 PM GMT May 21,
2022 3:57:26 PM GMT Success
Post cluster OAKD configuration May 21, 2022 4:01:54 PM GMT May 21,
2022 4:06:40 PM GMT Success
Disk group 'RECO'creation May 21, 2022 4:06:49 PM GMT May 21,
2022 4:07:27 PM GMT Success
Disk group 'FLASH'creation May 21, 2022 4:07:27 PM GMT May 21,
2022 4:07:48 PM GMT Success
Setting ACL for disk groups May 21, 2022 4:07:48 PM GMT May 21,
2022 4:07:54 PM GMT Success
Modify DB file attributes May 21, 2022 4:07:54 PM GMT May 21,
2022 4:08:07 PM GMT Success
Register Scan and Vips to Public Network May 21, 2022 4:08:12 PM GMT May 21,
2022 4:08:17 PM GMT Success
Configure export clones resource May 21, 2022 4:09:32 PM GMT May 21,
2022 4:09:33 PM GMT Success
Volume 'commonstore'creation May 21, 2022 4:09:33 PM GMT May 21,
2022 4:09:52 PM GMT Success
ACFS File system 'DATA'creation May 21, 2022 4:09:52 PM GMT May 21,
2022 4:10:09 PM GMT Success
Database home creation May 21, 2022 4:10:09 PM GMT May 21,
2022 4:17:38 PM GMT Success
Creating ACFS database home May 21, 2022 4:10:09 PM GMT May 21,
2022 4:11:55 PM GMT Success
Validating dbHome available space May 21, 2022 4:11:55 PM GMT May 21,
2022 4:11:55 PM GMT Success
Validating dbHome available space May 21, 2022 4:11:55 PM GMT May 21,
2022 4:11:55 PM GMT Success
Creating DbHome Directory May 21, 2022 4:11:55 PM GMT May 21,
Oracle Internal and Approved Partners Only Page 50 of 58 Version 2.1 – Created: 9 Jun 2022
2022 4:11:55 PM GMT Success
Create required directories May 21, 2022 4:11:55 PM GMT May 21,
2022 4:11:55 PM GMT Success
Extract DB clone May 21, 2022 4:11:55 PM GMT May 21,
2022 4:13:41 PM GMT Success
ProvDbHome by using RHP May 21, 2022 4:13:41 PM GMT May 21,
2022 4:17:11 PM GMT Success
Enable DB options May 21, 2022 4:17:11 PM GMT May 21,
2022 4:17:29 PM GMT Success
Creating wallet for DB Client May 21, 2022 4:17:34 PM GMT May 21,
2022 4:17:38 PM GMT Success
Provisioning service creation May 21, 2022 4:17:38 PM GMT May 21,
2022 4:31:55 PM GMT Success
Database Creation by RHP May 21, 2022 4:17:38 PM GMT May 21,
2022 4:28:37 PM GMT Success
Change permission for xdb wallet files May 21, 2022 4:28:37 PM GMT May 21,
2022 4:28:37 PM GMT Success
SqlPatch upgrade May 21, 2022 4:30:22 PM GMT May 21,
2022 4:30:41 PM GMT Success
Running dbms_stats init_package May 21, 2022 4:30:41 PM GMT May 21,
2022 4:30:43 PM GMT Success
Set log_archive_dest for Database May 21, 2022 4:30:44 PM GMT May 21,
2022 4:30:46 PM GMT Success
Updating the Database version May 21, 2022 4:30:46 PM GMT May 21,
2022 4:30:50 PM GMT Success
Set CPU pool May 21, 2022 4:30:50 PM GMT May 21,
2022 4:30:50 PM GMT Success
Users tablespace creation May 21, 2022 4:31:55 PM GMT May 21,
2022 4:31:58 PM GMT Success
Clear all listeners from Databse {584999fd-4717-4ba8-b4ea-9f31f28433f3} May 21, 2022
4:31:58 PM GMT May 21, 2022 4:31:59 PM GMT Success
Clear local and remote listeners from Database May 21, 2022 4:32:02 PM GMT
May 21, 2022 4:32:03 PM GMT Success
Delete public key May 21, 2022 4:32:07 PM GMT May 21,
2022 4:32:07 PM GMT Success
configure Zk Quorum TLS May 21, 2022 4:32:08 PM GMT May 21,
2022 4:32:09 PM GMT Success
Create Trust Store May 21, 2022 4:32:09 PM GMT May 21,
2022 4:32:10 PM GMT Success
Delete Trust Store May 21, 2022 4:32:10 PM GMT May 21,
2022 4:32:10 PM GMT Success
Create Trust Store May 21, 2022 4:32:10 PM GMT May 21,
2022 4:32:11 PM GMT Success
Provisioning service creation May 21, 2022 4:32:12 PM GMT May 21,
2022 4:32:12 PM GMT Success
persist new agent state entry May 21, 2022 4:32:12 PM GMT May 21,
2022 4:32:12 PM GMT Success
persist new agent state entry May 21, 2022 4:32:12 PM GMT May 21,
2022 4:32:12 PM GMT Success
persist new agent state entry May 21, 2022 4:32:12 PM GMT May 21,
2022 4:32:12 PM GMT Success
persist new agent state entry May 21, 2022 4:32:12 PM GMT May 21,
2022 4:32:12 PM GMT Success
Restart Zookeeper and DCS Agent May 21, 2022 4:32:12 PM GMT May 21,
2022 4:32:13 PM GMT Success

Make sure all components are up-to-date (example below continues to the next page).
Oracle Internal and Approved Partners Only Page 51 of 58 Version 2.1 – Created: 9 Jun 2022
NOTE: If the installed version is higher than the available version, this is OK.

[root@oak1 u01]# odacli describe-component


System Version
---------------
19.15.0.0.0

System node Name


---------------
odax8-2ha-brm06-a-h0

Local System Version


---------------
19.15.0.0.0

Component Installed Version Available Version


---------------------------------------- -------------------- --------------------
OAK
19.15.0.0.0 up-to-date
GI
19.15.0.0.220419 up-to-date
DB[ BroomDB ]
19.15.0.0.220419 up-to-date
DCSCONTROLLER
19.15.0.0.0 up-to-date
DCSCLI
19.15.0.0.0 up-to-date
DCSAGENT
19.15.0.0.0 up-to-date
DCSADMIN
19.15.0.0.0 up-to-date
OS
7.9 up-to-date
ILOM
5.0.2.24.r141466 up-to-date
BIOS
52050300 up-to-date
LOCAL CONTROLLER FIRMWARE {
[c4]
214.2.271.9 up-to-date
[c6]
80000681 up-to-date
}
SHARED CONTROLLER FIRMWARE
16.00.08.00 up-to-date
FIRMWAREEXPANDER
0310 up-to-date
LOCAL DISK FIRMWARE
1120 1132
SHARED DISK FIRMWARE {
[c0d0,c0d1,c0d2,c0d3,c0d4,c0d5,c1d0,
c1d1,c1d2,c1d3,c1d4,c1d5] A959 up-to-date
[c0d6,c0d7,c0d8,c0d9,c0d10,c0d11,c0d12,
c0d13,c0d14,c0d15,c0d16,c0d17,c0d18,
c0d19,c0d20,c0d21,c0d22,c0d23,c1d6,c1d7,
c1d8,c1d9,c1d10,c1d11,c1d12,c1d13,c1d14,
c1d15,c1d16,c1d17,c1d18,c1d19,c1d20,
c1d21,c1d22,c1d23] A144 up-to-date
[c2d0,c2d1]
1120 1132
}
HMP
Oracle Internal and Approved Partners Only Page 52 of 58 Version 2.1 – Created: 9 Jun 2022
2.4.8.0.601 up-to-date
System node Name
---------------
odax8-2ha-brm06-a-h1

Local System Version


---------------
19.15.0.0.0

Component Installed Version Available Version


---------------------------------------- -------------------- --------------------
OAK
19.15.0.0.0 up-to-date
GI
19.15.0.0.220419 up-to-date
DB[ BroomDB ]
19.15.0.0.220419 up-to-date
DCSCONTROLLER
19.15.0.0.0 up-to-date
DCSCLI
19.15.0.0.0 up-to-date
DCSAGENT
19.15.0.0.0 up-to-date
DCSADMIN
19.15.0.0.0 up-to-date
OS
7.9 up-to-date
ILOM
5.0.2.24.r141466 up-to-date
BIOS
52050300 up-to-date
LOCAL CONTROLLER FIRMWARE {
[c6]
80000681 up-to-date
}
SHARED CONTROLLER FIRMWARE
16.00.08.00 up-to-date
FIRMWAREEXPANDER
0310 up-to-date
LOCAL DISK FIRMWARE
1102 up-to-date
SHARED DISK FIRMWARE {
[c0d0,c0d1,c0d2,c0d3,c0d4,c0d5,c1d0,
c1d1,c1d2,c1d3,c1d4,c1d5] A959 up-to-date
[c0d6,c0d7,c0d8,c0d9,c0d10,c0d11,c0d12,
c0d13,c0d14,c0d15,c0d16,c0d17,c0d18,
c0d19,c0d20,c0d21,c0d22,c0d23,c1d6,c1d7,
c1d8,c1d9,c1d10,c1d11,c1d12,c1d13,c1d14,
c1d15,c1d16,c1d17,c1d18,c1d19,c1d20,
c1d21,c1d22,c1d23] A144 up-to-date
[c2d0,c2d1]
1102 up-to-date
}
HMP
2.4.8.0.601 up-to-date

Oracle Internal and Approved Partners Only Page 53 of 58 Version 2.1 – Created: 9 Jun 2022
CHANGING THE ODA CPU CORE COUNT
The ODA is delivered with all 32 cores on each server enabled. This section is only applicable of the
customer requires less than 36 cores.
Set the cores in multiples of 2 (e.g. 12):
NOTE: It is no longer necessary to generate a corekey on MOS. To reduce the number of
cores, use the following command on Node 0 only.
[root@oak1 u01]# odacli update-cpucore --cores 12
Modifying the enabled number of CPU requires a reboot of all nodes in the
ODA system. Are you sure you want to proceed with this operation? (Y/N): y
{
"jobId" : "7d3d8d96-e3ad-497d-a294-fca15dcad853",
"status" : "Created",
"message" : null,
"reports" : [ ],
"createTimestamp" : "May 21, 2022 17:34:59 PM GMT",
"resourceList" : [ ],
"description" : "CPU cores service update",
"updatedTime" : "May 21, 2022 17:34:59 PM GMT"
}
WARNING: Reducing the number of enabled cores may set offline any of the
CPUs used by a CPU Pool. Thus, please review the status of all the CPU Pools
after this operation using the 'describe-cpupool' command. In case of any
CPU from a CPU Pool was offline please run 'remap-cpupools -f' to update the
list of CPUs and use only online CPUs.
WARNING: All the nodes in this ODA system will reboot after this operation
has completed.

After the node reboots, confirm on both nodes that the core count is correct.
Run the following command on both nodes.

# odacli describe-cpucore

Node Cores Modified Job Status


----- ------ ---------------------------------- ---------------
0 12 May 21, 2022 5:37:13 PM GMT CONFIGURED

Oracle Internal and Approved Partners Only Page 54 of 58 Version 2.1 – Created: 9 Jun 2022
POST VALIDATION
Verify the system has been successfully deployed by checking the new system version.
The results should look similar to the output below:
# odacli describe-system

Appliance Information
----------------------------------------------------------------
ID: 0bd94d6d-4973-4c06-8cef-cf6aed93ff8d
Platform: X8-2-HA
Data Disk Count: 25
CPU Core Count: 12
Created: May 21, 2022 3:22:43 PM GMT

System Information
----------------------------------------------------------------
Name: odax8-2ha
Domain Name: us.oracle.com
Time Zone: GMT
DB Edition: EE
DNS Servers: 206.223.27.1 206.223.27.2
NTP Servers: 10.145.227.6 10.145.227.7

Disk Group Information


----------------------------------------------------------------
DG Name Redundancy Percentage
------------------------- ------------------------- ------------
DATA FLEX 80
RECO FLEX 20
FLASH FLEX 100
If there is an error during deployment, it is recommended that you run the cleanup script on
BOTH nodes.

# /opt/oracle/oak/onecmd/cleanup.pl
INFO: Log file is
/opt/oracle/oak/log/odax8-2ha-brm06-a-h0/cleanup/cleanup_2022-05-21_19-03-
42.log

INFO: Log file is


/opt/oracle/oak/log/odax8-2ha-brm06-a-h0/cleanup/dcsemu_diag_precleanup_2022
-05-21_19-03-42.log

INFO: *******************************************************************
INFO: ** Starting process to cleanup provisioned host odax8-2ha-brm06-a-h0
**
INFO: *******************************************************************
INFO: Default mode being used to cleanup a provisioned system.
INFO: It will change all ASM disk status from MEMBER to FORMER
Do you want to continue (yes/no) : yes
INFO:
Running cleanup will delete Grid User - 'grid' and
INFO: DB user - 'oracle' and also the
INFO: groups 'oinstall,dba,asmadmin,asmoper,asmdba'

Oracle Internal and Approved Partners Only Page 55 of 58 Version 2.1 – Created: 9 Jun 2022
INFO: nodes will be rebooted
Do you want to continue (yes/no) : yes
INFO: /u01/app/19.15.0.0/grid/bin/crsctl.bin

INFO: /u01/app/19.15.0.0/grid/.patch_storage/33815596_Apr_6_2022_20_34_59/
files/bin/crsctl.bin

INFO: /u01/app/19.15.0.0/grid/.patch_storage/33987244_Apr_12_2022_06_11_26/
files/bin/crsctl.bin

INFO: *************************************
INFO: ** Checking for GI bits presence
INFO: *************************************
INFO: GI bits /u01/app/19.15.0.0/grid found on system under /u01/app
directory...

INFO: Setting up SSH equivalence from odax8-2ha-brm06-a-h0 to 192.168.16.25

root@192.168.16.25's password:
INFO: Successfully set SSH equivalence from odax8-2ha-brm06-a-h0 to
192.168.16.25
INFO: ** Disabling AFD filtering
SUCCESS: AFD filtering disabled on all devices
INFO: ** Saving disk partition info
INFO: *************************************
INFO: ** Executing Stop Clusterware commands
INFO: *************************************
INFO: *************************************
INFO: ** Remove Oracle Stack
INFO: *************************************
INFO: Stop clusterware
INFO: Oracle High Availability Services online

After reboot from a cleanup.pl script, the GI and RDBMS patch will need to be installed
again:

[root@oak1 u01]# odacli update-repository -f /u01/odacli-dcs-19.15.0.0.0-


220425-DB-19.15.0.0.zip,/u01/odacli-dcs-19.15.0.0.0-220425-GI-19.15.0.0.zip

All passwords will also revert back to the default “welcome1”.

The hostnames will revert back to oak1/oak2.

Oracle Internal and Approved Partners Only Page 56 of 58 Version 2.1 – Created: 9 Jun 2022
SECURING THE ILOM SERVICE PROCESSORS (SP)
Both ILOM SPs must be secured as part of the system's initial installation. Securing requires
the changing of the root password from ‘changeme’. By default connections are secure to the
ILOM either through ssh or https connection through the browser.
Do NOT change the default password until after you have completed software deployment on
the Oracle Appliance.
Change the root password. The password length must between 8-16 characters.
You can use the GUI or the CLI.
Examples below.
Changing the password using the GUI:
1. Pull down the ILOM Administration menu on the left.
2. Select User Management, select the User Accounts sub-tab.
3. Select root user & click on edit.
4. Change the root password.
Changing the password using the CLI:
# ssh -l root <SP-ipaddr>
# set /SP/users/root password=<new password>
Changing password for user /SP/users/root/password...
Enter new password again: *********
New password was successfully set for user /SP/users/root

For further information refer to the Oracle ILOM Administrator's Guide for Configuration and
Maintenance Firmware Release 4.0.x E86149.

HANDOVER
Perform Installation Assessment tests as EISdoc V4 – completed during preparation of
described in the EIS Test Procedures Plan. installation.

Complete documentation and hand over to EISdoc V4:


customer. File EIS-DOCS-Operational Handover-
Document.odt
Short briefing: the configuration.

Oracle Internal and Approved Partners Only Page 57 of 58 Version 2.1 – Created: 9 Jun 2022
For additional help with deployment, you can use global-field-oda-sme-assist_grp@oracle.com
This alias is intended for use is by FEs to reach out to Field resources for help with ODA installs.

Copies of the checklists are available on the EIS web pages or on the EIS-DVD. We recommend that you always check the
web pages for the latest version.

Comments & RFEs welcome. Oracle staff should mail to EIS-SUPPORT_WW_GRP@oracle.com


Partners should contact the PartnerHelp Portal for assistance and feedback.

Oracle Internal and Approved Partners Only Page 58 of 58 Version 2.1 – Created: 9 Jun 2022

You might also like