Ellipse 8 Patching Guide

Ellipse 8 Patching Guide

Contents
Ellipse 8 Patching Guide 2
Commercial In Confidence 3
Purpose 4
Scope 5
Update Notes 6
Patching the Installation 7
Overview 7
Accessing the Patch 7
Patching non-appliance based installations of Ellipse 8.6 or greater 9
Patching Appliance based installations of Ellipse 8 or greater 9
Load MOR 9
Upgrade the Infrastructure 10
Upgrade the Environment 10
Refresh the Browser 10
Patching the Appliance Operating System - Automatic 11
Overview 11
1. Override Public Yum Server (optional) 11
2. Add Proxy Server (optional) 11
3. Issue the Security Update 11
Patching the Appliance Operating System - Manual 13
Overview 13
1. Configure Yum 13
1.1 Ensure the Oracle Public Yum Server is accessible from the Appliance 13
1.2 Install the Security plugin for Yum 13
1.3 Verify Yum configuration 14
2. Apply patches 14
2.1 Stop Puppet service 14
2.2 Patch a specific errata 14
2.3 Start the Puppet service 14
2.4 Verify patch application 14
3. Roll-back patches 15
3.1 Roll-back the last patch 15
3.2 Verify roll-back 15
4. Patching Command and Control guest 15
4.1 Reboot the Command and Control guest 15
4.2 Verify Command and Control services 15
Patching the Pentaho / ETL System 16
Automated Maintenance Patches 16
Manual Maintenance Patches 16
Commercial In Confidence
Copyright 2016 ABB
All Rights Reserved
Confidential and Proprietary
Legal Disclaimer
The product described in this documentation may be connected to, and/or communicate information and data via, a network
interface, which should be connected to a secure network. It is your sole responsibility to ensure a secure connection to the
network and to establish and maintain appropriate measures (such as but not limited to the installation of firewalls, application
of authentication measures, encryption of data, installation of antivirus programs, etc.) to protect the product, the network,
your systems, and the interface against any kind of security breach, unauthorised access, interference, intrusion, leakage,
damage, or corruption or theft of data. We are not liable for damages or losses related to any such security breach,
unauthorised access, interference, intrusion, leakage, damage, or corruption or theft of data.
Purpose
This document describes the process for applying patches to Ellipse 8.
Scope
This document has sections covering:

• Patching the Installation

• Patching Non-appliance based installations of Ellipse

• Patching Appliance based installations of Ellipse
• Patching the Appliance Operating System

• Configure Yum
• Apply patches for CVEs
• Rollback patches
Update Notes
Please refer to the Ellipse 'Update Notes' and check if there is information that applies.
Patching the Installation

Overview
To install a patch, access and download the MOR file and then follow the relevant patching instructions. The patch name is
communicated to the customer by ABB and referred to here as Ellipse-<patch_number>, for example Ellipse 8.6.1 MOR.

Figure: overview of install to upgrade

Each release comes with an uploaded MOR and release notes, that can be found in the Workspace for that product/version.
Ensure that the MOR file is placed in the directory and that the MOR release selected corresponds to the version indicated in
the release notes.
The Oracle Linux Disk (for example 6.6 for Ellipse 8.6) needs to be present in the drive during any update/patch. For example
from 8.6.1 to 8.6.3.

Note
The MOR file for each release also contains an updated list of OS security advisories (referred to as ELSA, Enterprise Linux
Security Advisory). Application of these security patches is currently an optional step separate from the appliance
infrastructure upgrade. It is envisaged that in a future release, application of OS security patches will be a mandatory step
in the infrastructure upgrade process. This is due to several reasons: the security patches include critical kernel bug fixes;
ABB certifies and tests on patched appliances; and customers face issues when patching is ignored and the appliance OS
becomes out of date. It is highly recommended that OS security patches are applied after each appliance infrastructure
upgrade as a matter of policy.

There are two sets of instructions, one for each type of installation. Refer to the section that applies.
They are:

• Non-appliance based installations of Ellipse 8.6 or greater

• Appliance based installations of Ellipse 8.6 or greater

Accessing the Patch

This section describes the process by which Ellipse 8 patches are downloaded.

1. Download the MOR file (example Ellipse 8.6.6 MOR) from the "ABB Customer Portal"

a. Login to the Customer Portal (https://enterprisesoftware.force.com/customerportal/login)

b. Select "Workspaces" to display "My Workspaces"
c. Select the "Workspace Name" for the Product Version required
 Figure: Customer Product Portal - Select the Ellipse version

d.
1. Select the Icon next to the file name to display the Open window, then (2) Select Open

Figure: Customer Product Portal - Select the Icon/File to Open

e. Select the Save action to save the MOR zip file.

Figure: Customer Product Portal - Select Save to download the zip file
 Note
If a customer does not have a login to the ABB Customer Portal they should contact their account manager and
request one.

For 8.6 Appliance Manager installations:

2. Place the file in the appliance host:

/appliance/data/dist

Patching non-appliance based installations of Ellipse 8.6 or

greater
Use these instructions when patching Ellipse 8.6 versions that have been installed using the Ellipse 8 Manual Installation Guide.

Note
Use the Ellipse 8 Manual Installation guide to assist in installing the patch.

1. Follow the instructions for downloading MOR (Accessing the Patch above).
2. Update the properties files to ensure that the ELLIPSEEAR.BASELINE.VERSION refers to the baseline indicated in the
Release Notes.
3. Remove the application servers that need to be replaced using the instructions Un-install Ellipse 8 Components from
the E8_Install_Manual.pdf.
4. Install the latest version of MOR by following the Steps to install MOR section from the Ellipse 8 Manual Installation
Guide (E8_Install_Manual.pdf). Use the MOR associated with the release notes.
5. Install the application servers that need to be updated, using the instructions from the Ellipse 8 Manual Installation Guide.

Patching Appliance based installations of Ellipse 8 or greater

Use these instructions when patching Ellipse 8.6 versions that have been installed using the Enterprise Appliance Manager
(VEAM), also called Appliance Manager.
The steps for a release upgrade include:

1. Upload the offline software repository or MOR (see Load MOR below)
2. Upgrade the Appliance Infrastructure which is part of the VEAM (see Upgrade the Infrastructure below)
3. Upgrade the Appliance Environment(s), that is each environment hosted in that appliance (see Upgrade the Environment
below)

Note
Do not attempt to upgrade an environment release/version unless the appliance infrastructure upgrade is first performed.

Load MOR
Assumption:

• The new MOR file (.mor) has been uploaded to the directory (/appliance/data/dist) on the Appliance server that is to
be upgraded
1. Access the Appliance Manager at the following URL
http://cmdctl.(fully_qualified_hostname)
2. Click on Upload MOR from the Operations drop down list
3. Enter the file name (no path) of the MOR (.mor) file
4. Click on Execute to load the MOR file into the Appliance Manager
The version shall now be selectable from the version (Add) and new version (Upgrade/Downgrade) drop down lists. This
means that the uploaded MOR can be used for a new environment or an upgrade to an environment
Upgrade the Infrastructure

1. Access the Appliance Manager at the following URL

http://cmdctl.(fully_qualified_hostname)
2. Click on Upgrade Infrastructure from the Help link located in the top left of the window
3. Select the target Appliance Infrastructure release from the dropdown list
4. Click on OK to perform the upgrade of the infrastructure.

Note
Upgrades to the infrastructure will not impact deployed environments.

Upgrade the Environment

1. Go to the Manage Environments tab and select the environment to be upgraded.

2. Upgrade the environment (Upgrade/Downgrade button)

Note
The properties should not need to be changed.

Refresh the Browser

Following any Environment upgrade, users will be required to do one of the following to ensure that updated versions of
browser side components related to the upgraded application are being used:

• Use F5 to reload the login screen page; or

• Close and reopen the browser tab or window and reload the login screen page
There should be no need for end users to clear their browser cache or manipulate cache settings in any way.
Patching the Appliance Operating System - Automatic

Overview
Each MOR file installed in the system contains a list of security advisories (referred to as ELSA, Enterprise Linux Security
Advisory).
Related software patches addressing these security advisories are tested by ABB before shipping the MOR, thus supported for
installation on a targeted appliance. Once the Infrastructure is upgraded with contents from a specific MOR, the operation
"Security Update" operation can be selected in VEAM to install the patches.
Patches will be downloaded and installed on the Appliance and the Command and Control guest only by the standard utility
'yum'.
The amount of time required to apply these patches may vary and depends on factors such as network speed and number of
patches already installed on the system.

1. Override Public Yum Server (optional)

By default, the Security Update operation tries to access Oracle's Public Yum repositories, http://public-yum.oracle.com
Customer may set up a local mirror and override Oracle's address by adding a new property to
/appliance/data/conf/etc/appliance.properties

#------------------------------------------------------------------------------#
#
# LINUX UPDATE SERVER
#
#------------------------------------------------------------------------------#

# Linux Update Server

#
# * DEFAULT: public-yum.oracle.com
# * Allows the overriding of the Public Yum Oracle repository used to download
# * security patches for the appliance operating system.
# *
#
# [yum_server] = Yum server containing a Yum - compatible OEL distribution tree
#
# Example:
# linux.update.server=[yum_server]

Note
Any change to appliance.properties will require restarting services 'puppetmaster' (on the Appliance) and 'puppet' (on the
Appliance and Command and Control).

2. Add Proxy Server (optional)

To configure a HTTP proxy for use by Yum, edit /etc/yum.conf:

proxy=http://<host>:<port>
proxy_username=<username>
proxy_password=<password>

Note
Changes to /etc/yum.conf must be done on both the Appliance and the Command and Control Virtual Server.

3. Issue the Security Update

Assumption:

• The new MOR file (.mor) has been uploaded to the directory (/appliance/data/dist) on the Appliance server that is to
be upgraded
• The Infrastructure level has been upgraded to the level supplied by the uploaded MOR
1. Access the Appliance Manager at the following URL
 http://cmdctl.(fully_qualified_hostname)

2. Click on 'Security Update' from the Operations drop down list. Type the word 'reboot' in the text field if an appliance
reboot is to be performed right after the patches are applied. Check progress report in
'/appliance/data/dist/sec_update.log' and '/appliance/data/dist/sec_update.err'.

Note
Given that ELSAs may affect several Operating System components, it is almost impossible to predict when a reboot is
required as a result of the application of a patch. ABB recommend rebooting as soon as it's practicable after performing the
update. This is especially true when certain components are involved (i.e. Linux kernels, stdlib, glib, ssh).
Patching the Appliance Operating System - Manual

Note
Patching an appliance with this manual system may lead to a system running an untested OS configuration and is therefore
no longer supported by ABB. This section is intentionally left in the document to help the customer installing manual
patches if directed to do so by ABB.

This sections covers:

• Configure Yum
• Apply patches for CVEs
• Rollback patches

Overview
This section will outline the procedures required to patch an Ellipse 8.6 3rd generation Appliance based on Oracle Linux 7.2 OS
for Common Vulnerabilities and Exposures (CVE).
The use of the term "patch" in this document represents the change of existing OS packages to address a specific CVE, the
standard identifier as defined by http://cve.mitre.org.
There will be no new OS functionality introduced as part of this procedure and the OS major and minor version will remain
constant.

1. Configure Yum

1.1 Ensure the Oracle Public Yum Server is accessible from the Appliance
The two specific Yum repositories required are "ol7_latest" and "ol7_UEK_latest".
Using the Public Yum Server, define the two repositories in the following file /etc/yum.repos.d/public-yum-ol7.repo:

[ol7_latest]
name=Oracle Linux $releasever Latest ($basearch)
baseurl=http://public-yum.oracle.com/repo/OracleLinux/ol7/latest/$basearch/
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
gpgcheck=1
enabled=0

[ol7_UEK_latest]
name=Latest Unbreakable Enterprise Kernel for Oracle Linux $releasever ($basearch)
baseurl=http://public-yum.oracle.com/repo/OracleLinux/ol7/UEK/latest/$basearch/
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
gpgcheck=1
enabled=0

Note
Disabling both Yum repositories with "enabled=0" is critical as Puppet will utilise available repositories in order to ensure
the latest version of packages it administers. Another option is to specify the Yum config file at execution time with the
"--config" flag, this may be a URL (http://public-yum.oracle.com/public-yum-ol7.repo) or a file system path outside of
"/etc/yum.repos.d/".

To configure a HTTP proxy for use by Yum, edit /etc/yum.conf:

proxy=http://<host>:<port>
proxy_username=<username>
proxy_password=<password>

1.2 Install the Security plugin for Yum

Install the following plug-in for Yum which will provide security specific package listing and information options.

yum -y install yum-plugin-security.noarch

1.3 Verify Yum configuration
Determine the specific security errata that are applicable using the previously defined Yum configuration:

yum --disablerepo=* --enablerepo=ol7_latest,ol7_UEK_latest --security updateinfo list

The Yum query returns a list of errata identifiers, severity and package version.

2. Apply patches
It is recommended issued patches only be applied no later that one month prior to the specific Deployment Infrastructure
release date. For example the release date of di-3.10.6 was 2015-10-14, the last patch issued priort to 2015-10-01 was
ELSA-2015-1840 - issued 2015-09-29.
Issue date can be determined by inspecting the detailed information for the patch:

yum --disablerepo=* --enablerepo=ol7_latest,ol7_UEK_latest updateinfo info --advisory ELSA-2013-2576

Loaded plugins: security

===============================================================================
unbreakable enterprise kernel security update
===============================================================================
Update ID : ELSA-2013-2576
Release : Oracle Linux 7
Type : security
Status : final
Issued : 2013-10-18
CVEs : CVE-2013-4299
Description : [2.6.39-400.209.2]
: - dm snapshot: fix data corruption (Mikulas
: Patocka) [Orabug: 17618492] {CVE-2013-4299}
Severity : Moderate
updateinfo info done

2.1 Stop Puppet service

Stop the Puppet service on the Appliance to ensure there are no conflicts with Yum:

service puppet stop

2.2 Patch a specific errata

Determine the list of available errata, sort most recent to oldest and start from the last recommended patch:
yum --disablerepo=* --enablerepo=ol7_latest,ol7_UEK_latest --security updateinfo list | cut -f 1 -d ' ' | egrep ^ELSA-[0-9]*-[0-9]* | sort -r | uniq

Apply the last recommeded patch, for an Ellipse 8.6.1 Appliance running di-3.10.6 this would be "ELSA-2015-1840".

yum --disablerepo=* --enablerepo=ol7_latest,ol7_UEK_latest -y update --advisory ELSA-2015-1840

Note
the Yum transaction id, this will be required in the event of a roll-back:

yum history info | egrep '^Transaction ID' | cut -f 4 -d ' '

For kernel patching, specifically the "kernel-uek", ensure the Grub configuration is updated and an outage schedule to reboot
the Appliance. Unless Ksplice is configured, "/etc/grub.conf" will need to be manually copied to "/boot/grub/grub.conf" for the
new kernel to take effect at boot.

2.3 Start the Puppet service

Start the Puppet service by compiling the Puppet catalog for the Appliance:

puppet agent -t

There should be no errors returned from the catalog compile, else a roll-back will be required (return code 0 or 2).

2.4 Verify patch application

To verify the patches have been applied, execute the same query to list the security errata for the current OS packages. This is
recommended after each patch application to determine the next errata to apply:
yum --disablerepo=* --enablerepo=ol7_latest,ol7_UEK_latest --security updateinfo list | cut -f 1 -d ' ' | egrep ^ELSA-[0-9]*-[0-9]* | sort -r | uniq
This list will be significantly smaller than the initial list prior to installing the patch.

3. Roll-back patches
In the event the patch application is unsuccessful, a roll-back is required in order to reset the versions of OS packages. This is
achieved using Yum history and transaction rewinding.

3.1 Roll-back the last patch

Using the transaction id determined in the previous section roll-back the changes using Yum history.

service puppet stop

yum history undo <transaction_id>
puppet agent -t

If there have been no Yum transactions since the patch application, the keyword 'last' maybe substituted:
service puppet stop
yum history undo last
puppet agent -t

Again the manual Puppet catalog compile from both examples should complete without error.

3.2 Verify roll-back

In addition to the clean execution of a manual Puppet catalog compile, query the list of security errata for the newly reset OS
package versions:

yum --disablerepo=* --enablerepo=el5_latest,ol5_UEK_latest --security updateinfo list

The list returned will represent the patches available for the current versions of OS packages.

4. Patching Command and Control guest

To patch the Command and Control guest, the same procedures for the Appliance may be applied. Ensure an appropriate
outage has been scheduled as this will directly impact accessibility of deployed environments.
In the event a patch introduces a new kernel, the reboot process is obviously different with a guest than the physical Appliance:

4.1 Reboot the Command and Control guest

Identify the OpenNebula id which will be the integer in the first column of the output:

su -c "onevm list | grep cmdctl" oneadmin

Append the id to the string "one-" and reset the guest using "virsh":

virsh reset one-<id>

Optionally connect to the guest console to view the boot process:

virsh console one-<id>

4.2 Verify Command and Control services

Once the guest has finished booting, ensure Puppet, Veam and HTTPD services have started without error by inspecting the
respective logs:

/var/log/messages
/opt/veam/current/server.log
/var/log/httpd/error_log

In addition to this, navigate to the Command and Control URL to exercise HTTPD and Veam.
Patching the Pentaho / ETL System

Automated Maintenance Patches

This process will handle the Standard Maintenance Patches applied on the Appliance System. When this is done the appliance
will be rebuilt with the standard installation. After this has completed these additional steps will need to be done

1. JINDI Database Connection Configuration (refer to Ellipse Operations and Configuration Guilde > Pentaho Server >
JINDI...)
2. CRON Job to Schedule /opt/datamart/pentaho/data-integration/load_all_for_SITE1.sh
3. Changes(If any) that you have made in "opt/datamart/pentaho/datamart/Star" will need to be made again.

Manual Maintenance Patches

Manual Releases and Patches will be applied using this approach.
The Maintenance Patch and information will exist within these components

• Patch.tar.bz2

• This contains the Pentaho Star Schema

• This file will be used to update the Pentaho ETL Logic
• ReleaseNotes.docx

• This will detail the Changes included in this Maintenance Patch

• PatchNotes.docx

• This will detail any other steps required in the installation

This process will cover every action required unless more information is contained in the PatchNotes.docx document. This
document should be read prior to starting this step.

1. Copy Tar file to /opt/datamart/pentaho/datamart/Stars

2. Unwind this tar file like : tar jxvf Patch.tar.gz
3. Apply and Steps contained in PatchNotes.docx
4. Run the Monthly Update

• The update ETL layer will make the required changes to the Datamart Schema and Data the first time it is run