Get Ready for Operating- ExaCC Gen 2

Hamad Medical
Cristina Ioan
March 2022

1 Copyright © 2021, Oracle and/or its affiliates

Safe harbor statement

The following is intended to outline our general product direction. It is intended for information purposes only, and
may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality,
and should not be relied upon in making purchasing decisions. The development, release, timing, and pricing of any
features or functionality described for Oracle’s products may change and remains at the sole discretion of Oracle
Corporation.

2 Copyright © 2021, Oracle and/or its affiliates

Agenda

• Assurance Team
• Autonomous Health Framework
• Roles and Responsibilities
• Infrastructure Patching
• Working Effectively with Support

3 Copyright © 2021, Oracle and/or its affiliates

Assurance Team

4 Copyright © 2021, Oracle and/or its affiliates

 Assurance Team : Execution Framework

Objective is to sustain Long term Dialogue

through an independent periodic review of their

Op. Monitoring & Opt. Adv.

environment
Synergies with

Objective to provide the most positive experience through the Adoption,

Additional Milestone Implementation and Milestone

Go-Live Assurance
followUp
It is a structured time-boxed assistance provided to critical Milestone can be:

Acceptance
 Provisioning the environment,
 Go-Live Readiness,
 Post Go-Live Stabilization

Proactive SR
Monitoring
Technical Assurance and Remedies CoE Engagement at
Engagement on Critical Scenario with as Due Time of the
Incident Managers Milestone

C@C Gen2
Taskforce
Experience Base
Copyright © 2021, Oracle and/or its affiliates | Confidential: Internal/Restricted/Highly Restricted
 Cloud Assurance
• Technical Assurance & Remedies
• Go-Live Assurance
• Operations Monitoring and Optimization Advisory
• C@C Proactive Services

6 Confidential – Oracle Internal/Restricted/Highly Restricted

 Get Ready for Operating- ExaCC Gen 2
Deploy & Activate Post Activation

• Cloud Specialist Assurance (CSA) provides on-going, post-activation,

support to ExaC@C customers Initial
‘Get Ready’
ExaChk
Operating
output
Workshop
• After Booking, CSA works with Sales to schedule a half-day Get Ready: evaluation

Operating ExaCC Workshop for the Customer

Initial Engagement and cadence of calls established

• The CSA team support includes: Periodic ExaChk Reviews

Assist with escalation of critical SRs

• Regular calls with Customer to check on current health of the ExaC@C
Track level of patching
• Track level of patching for relevant components to ensure they are
always current
• ‘ExaChk’ tool output evaluation for any system health issues
• Assistance with escalation of critical SRs

• Target audience: Technical Architects, DBAs and Infrastructure Engineers

• Cost: Free

Copyright © 2021, Oracle and/or its affiliates | Confidential: Internal/Restricted/Highly Restricted

AHF (Autonomous Health Framework)

9 Copyright © 2021, Oracle and/or its affiliates

Agenda

• AHF Overview
• System Compliance Checking
• Event Monitoring / Fault Detection
• Diagnostic Collection
• Event/Fault Notification
• Upload of Collected Diagnostics to Service Requests
• Integrated Tools for Administrators
• Maintaining AHF

10 Copyright © 2021, Oracle and/or its affiliates

What is AHF and Why AHF?
AHF presents the next generation of tools as components which work together autonomously
24x7 to keep database systems healthy and running while minimizing human reaction time.
What kind of issues can Oracle AHF resolve:
1. Availability issues
• runtime issues that threaten the availability of the software stack either through a software issue (DB, GI, OS) or
the issues with the underlying hardware resources (CPU, Memory, Network, Storage) that can cause a server to be
evicted from the cluster and shut down all the database instances
2. Performance issue
• the runtime issues that threaten the performance of the system as seen by the database client or application either
due to software issues (bugs, configurations.. Etc..) or client issues (demand, query types, connection
management…) etc..

11 Confidential – Oracle Internal/Restricted/Highly Restricted

 AHF Helping Us…Help You!
2 TFA
1 EXAchk/ORAchk
Automatic issue detection,
Automatic compliance checking
and warnings when drifting away
diagnostic collection and
from best practices as well as analysis along with a single
offering pre and post upgrade interface for Database support
advice tools

Compliance Fault Diagnostic Fault Support

Checking Detection Collection Notification Upload

Copyright © 2020 Oracle and/or its affiliates.

TFA and EXAchk Supported Platforms and Versions

• All major Operating Systems are supported • All Oracle Database & Grid versions 10.2+ are
• Linux (OEL, RedHat, SUSE, Itanium & supported
zLinux) • TFA and EXAchk shipped with:
• Oracle Solaris (SPARC & x86-64)
• AIX
• HPUX (Legacy Installer) Oracle Grid Oracle Database
• Windows Infrastructure
• OS versions supported are the same as those 11.2.0.4+
supported by the Database 12.2.0.1+
12.1.0.2+
12.2.0.1+
18.0.0.0+
18.0.0.0+
19.0.0.0+ 19.0.0.0+

Copyright © 2019 Oracle and/or its affiliates.

EXAchk (Compliance Validation)

14 Copyright © 2021, Oracle and/or its affiliates

Customer Experience Before EXAchk

“Hi Oracle Support, my database just fell over”

“ahh yer….we published a note on that a while

ago…. …didn’t you see it?”
Oracle Databases
“hmmmmm”

Copyright © 2019 Oracle and/or its affiliates.

Compliance Checking

EXAchk is for Engineered Systems, ORAchk is for ODA & Everything Else
 Automated Proactive self-service method for customers to perform Health
Checks on their Engineered, RAC and Single Instance systems
 Critical Checks Run daily at 2AM
 Full Checks Run Weekly at 3AM on Sundays
 Automated Runs are only supported on Linux, Solaris and Engineered
Systems (Dom-U and Bare Metal)
 Over 250 Checks Driven by Best Practices and Success Factors generated from
Real-World Customer Experiences
 Repair Functionality for discovered Compliance Drifts (where possible)
 Integration with Cluster Verification Utility (CVU)
 Pre-Upgrade Validation for GI and Database upgrades
 Integrated with CVU Pre/Post Validation and DB AutoUpgrade
 Remote Checks in Cluster Environments performed via TFA SSL
 Ability to Create User Defined Checks
Copyright © 2020 Oracle and/or its affiliates.
 Installation and Update

1. Download latest version: Doc 2550798.1 Execution times vary based on size of
the cluster, installed products being
2. Transfer zip to required machine checked and number of resources!

3. Unzip

4. Execute ahf_setup self extracting install script

• For Continuous Service Mode (Highly Recommended) ahf_setup will be executed as the root user
• For Standalone Mode (generally used in single instance DB environments) ahf_setup will be
executed as the user who owns the database software

Note: No Cluster or DB Downtime is required for Updates!

17 Confidential – Oracle Internal/Restricted/Highly Restricted

 Automatic AHF upgrade

AHF is now capable of automatically upgrading itself.

Upgrade can be trigged a number of different ways:

• Automatically when AHF finds a new version in the specified location
• On demand using ahfctl upgrade

New upgrade versions can be obtained automatically from:

•The local file system
•REST locations
•Object store locations;

You can obtain new version on-demand from My Oracle support and also from the above locations using
ahfctl upgrade
AHF will also prompt you to upgrade when it detects it’s older than 180 days.

18 Confidential – Oracle Internal/Restricted/Highly Restricted

To install as root:

Download the appropriate installer from above, copy to a target machine and unzip
Run the TFA & ORAchk/EXAchk install command ./ahf_setup [-ahf_loc install_dir] [-data_dir data_dir]
Compliance Checking

EXAchk is for Engineered Systems, ORAchk is for ODA & Everything Else
 Automated Proactive self-service method for customers to perform Health
Checks on their Engineered, RAC and Single Instance systems
 Critical Checks Run daily at 2AM
 Full Checks Run Weekly at 3AM on Sundays
 Automated Runs are only supported on Linux, Solaris and Engineered
Systems (Dom-U and Bare Metal)
 Over 250 Checks Driven by Best Practices and Success Factors generated from
Real-World Customer Experiences
 Repair Functionality for discovered Compliance Drifts (where possible)
 Integration with Cluster Verification Utility (CVU)
 Pre-Upgrade Validation for GI and Database upgrades
 Integrated with CVU Pre/Post Validation and DB AutoUpgrade
 Remote Checks in Cluster Environments performed via TFA SSL
 Ability to Create User Defined Checks
Copyright © 2020 Oracle and/or its affiliates.
Compliance Results…

• Easy-to-Read HTML Report via email

• System Health Score
• ALL Findings Documented with Hyper-linked
References
• Proactive Patch Recommendations
• Report Compare and Merge Functionality
• JSON output available for consumption by 3rd party
reporting tools
• Integration with EM via the Compliance Framework
• Centralized Repository and Compliance UI provided
Collection Manager

Copyright © 2019 Oracle and/or its affiliates.

TFA (Trace File Analyzer)

22 Copyright © 2021, Oracle and/or its affiliates

 TFA On-Demand Usage

Real-time Diagnose Perform

2 status 3 4 diagnostic
with DB
summary tools collection
Upload diagnostic
collection to Oracle
5 Support

TFA
Oracle Grid Infrastructure Oracle Support
& Databases
1 Request desired
action on-demand

Copyright © 2019 Oracle and/or its affiliates.

Compliance Fault Diagnostic Fault Support
Checking Detection Collection Notification Upload

Fault Fault Received From

ORA-297(01|02|03|08|09|10|40) Alert Log - Oracle Database

ORA-00600 Alert Log - Oracle ASM
ORA-07445 Alert Log - Oracle ASM Proxy
ORA-04(69|([7-8][0-9]|9([0-3]|[5-8]))) Alert Log - Oracle ASM IO Server
ORA-32701
ORA-00494
ORA-04020
ORA-04021
ORA-01578
ORA-00700
System State dumped
Instance Terminated

CRS-016(07|10|11|12) Alert Log - Oracle Clusterware

Cluster Health Advisor (CHA) Events Clusterware Event Monitoring Daemon (EVMD)
 Fault Detection

 Fault Detection is available from

the following sources:
 Clusterware Event Monitoring
Daemon (EVMD)
 Alert Log Monitoring
 Detected Faults are logged to the
TFA Lucene Index
 Retention of Faults in the Index is
7 Days
 Faults are able to be queried via:
tfactl events

Copyright © 2019 Oracle and/or its affiliates.

Compliance Fault Diagnostic Fault Support
Checking Detection Collection Notification Upload

Database/ASM Auto Collection Events Clusterware Auto Collection Events

ORA-031(13|37) CRS-1602
ORA-00600 CRS-1607
ORA-00603 CRS-1608
CRS-1609
ORA-040(30|31)
CRS-1610
ORA-07445 CRS-1611
ORA-01578 CRS-1612
ORA-297(01|02|03|08|10|40|70|71) CRS-1632
Instance Terminated CRS-1606

Note: The above lists are NOT all inclusive. For the purposes of this presentation we
only covers the most common Auto Collection Faults.
Automatic Diagnostic Collections

 Invoked Automatically by the TFAMain Daemon in Continuous Service Mode when

Known Critical Events are Detected
 Problem Type and Problem Time Identification and Isolation
 Ensures Proper Data is Collected at the Time of the Incident
 Duplicate and Related Events are Flood Controlled to Minimize Resource
Consumption and Avoid Duplicate Collections
 Data Redaction capabilities
 Auto Collections can be queried via: tfactl print collections –last <n>d
 Collected files will be Placed in the TFA Repository of the invoking node

Copyright © 2019 Oracle and/or its affiliates.

Automatic Diagnostic Collections

Copyright © 2019 Oracle and/or its affiliates.

On Demand Diagnostic Collection

Available for 100 problem types

Database areas Other Server Technology

• Errors / Corruption • RMAN • Enterprise Manager
• Performance • Transparent Data Encryption • Data Guard
• Install / patching / upgrade • Storage / partitioning • GoldenGate
• RAC / Grid Infrastructure • Undo / auditing • Exalogic
• Import / Export • Listener / naming services
• Exadata Storage Cells • Spatial / XDB

Run on-demand via:

tfactl diagcollect –srdc <srdc_type> [-sr <sr_number>]

Copyright © 2019 Oracle and/or its affiliates.

On Demand Diagnostics Collection with TFA

 A single command performs complete diagnostic collection for a given

problem
 Problem Type and Problem Time Identification and Isolation
 Built-in Access Control allowing non-root users to collect full diagnostics
 Data Redaction capabilities
 Notification of Auto Collection (if one exists) to avoid redundant work
 Ability to Upload Diagnostic Collections Directly to Service Requests
 REST Interface for Invocation and Query over HTTPS

Copyright © 2020 Oracle and/or its affiliates.

On Demand Use Case – Performance Data Collection
Manual Collection Steps TFA Data Collection
1. Generate ADDM reviewing Document 1680075.1 (multiple 1. Execute the following as the DB software owner:
steps)
2. Identify “good” and “problem” periods and gather AWR tfactl diagcollect –srdc dbperf
reviewing Document 1903158.1 (multiple steps)
3. Generate AWR compare report (awrddrpt.sql) using “good” and
“problem” periods
4. Generate ASH report for “good” and “problem” periods
reviewing Document 1903145.1 (multiple steps)
5. Collect OSWatcher data reviewing Document 301137.1  (multiple
steps)
6. Collect Hang Analyze output at Level 4
7. Run supporting sql scripts – Log File sync diagnostic (lfsdiag.sql)
output using Document 1064487.1  (multiple steps)
8. Check alert.log if there are any errors during the “problem”
period
9. Find any trace files generated during the “problem” period
10. Collate and upload all the above files/outputs to SR

Copyright © 2019 Oracle and/or its affiliates.

On Demand Collections

Copyright © 2020 Oracle and/or its affiliates.

 On Demand Collections

 If an Auto Collection was

already performed for a given
Event:
 A new collection is NOT
performed
 The user is notified of the
Auto Collection location

Copyright © 2020 Oracle and/or its affiliates.

TFA Utilities

Detect and Analyze Problems

Copyright © 2020 Oracle and/or its affiliates.

TFA Utilities

Tool Description
OSWatcher Oswatcher Collects and archives OS metrics. These are useful for instance or node evictions
& performance Issues. See document 301137.1 for more details

oratop Provides near real-time database monitoring. See document 1500864.1 for more details.

alertsummary Provides summary of events for one or more database or ASM alert files from all nodes

ls Lists all files TFA knows about for a given file name pattern across all nodes

pstack Generate process stack for specified processes across all nodes
grep Search alert or trace files with a given database and file name pattern, for a search string.

vi Opens alert or trace files for viewing a given database and file name pattern in the vi editor

tail Runs a tail on an alert or trace files for a given database and file name pattern

Copyright © 2020 Oracle and/or its affiliates.

TFA Utilities

Tool Description
param Shows all database and OS parameters that match a specified pattern
dbglevel Sets and unsets multiple CRS trace levels with one command
history Shows the shell history for the tfactl shell
changes Reports changes in the system setup over a given time period. This includes database
parameters, OS parameters and patches applied

calog Reports major events from the Cluster Event log

events Reports warnings and errors seen in the logs
managelogs Shows disk space usage and purges ADR log and trace files
ps Finds processes
triage Summarize oswatcher/exawatcher data

Copyright © 2020 Oracle and/or its affiliates.

TFA Utilities Execution

[oracle@cehaovmsp1079 ~]$ tfactl

Each tool can be run using TFA Shell tfactl>

Start tfactl shell with tfactl

Run a tool with the tool name tfactl > orachk

1. Where necessary set context with database <dbname> tfactl > database MyDB

2. Then run tool MyDB tfactl > oratop

3. Clear context with database MyDB tfactl > database

Copyright © 2020 Oracle and/or its affiliates.

Use TFA to Review Logs/Traces

42 Copyright © 2019 Oracle and/or its affiliates.

Top Articles and Community Links

 Doc 2550798.1 - Autonomous Health Framework (AHF) - Including TFA

and ORAchk/EXAChk

 AHF MOS Community

Copyright 2020, Oracle and/or affiliates. All rights reserved

 Click icon to add picture

Questions?

Copyright 2020, Oracle and/or affiliates. All rights reserved