Professional Documents
Culture Documents
The information in this document applies solely to the hardware/software product (“Product”) specified herein, and only as specified herein.
This document is intended for use by Nokia' customers (“You”) only, and it may not be used except for the purposes defined in the agreement
between You and Nokia (“Agreement”) under which this document is distributed. No part of this document may be used, copied, reproduced,
modified or transmitted in any form or means without the prior written permission of Nokia. If you have not entered into an Agreement
applicable to the Product, or if that Agreement has expired or has been terminated, You may not use this document in any manner and You
are obliged to return it to Nokia and destroy or delete any copies thereof.
The document has been prepared to be used by professional and properly trained personnel, and You assume full responsibility when using
it. Nokia welcome Your comments as part of the process of continuous development and improvement of the documentation.
This document and its contents are provided as a convenience to You. Any information or statements concerning the suitability, capacity,
fitness for purpose or performance of the Product are given solely on an “as is” and “as available” basis in this document, and Nokia reserves
the right to change any such information and statements without notice. Nokia has made all reasonable efforts to ensure that the content of
this document is adequate and free of material errors and omissions, and Nokia will correct errors that You identify in this document. But,
Nokia' total liability for any errors in the document is strictly limited to the correction of such error(s). Nokia does not warrant that the use of
the software in the Product will be uninterrupted or error-free.
This document is Nokia’ proprietary and confidential information, which may not be distributed or disclosed to any third parties without the
prior written consent of Nokia.
Nokia is a registered trademark of Nokia Corporation. Other product names mentioned in this document may be trademarks of their
respective owners, and they are mentioned for identification purposes only.
Nokia is continually striving to reduce the adverse environmental effects of its products and services. We would like to encourage you
as our customers and users to join us in working towards a cleaner, safer environment. Please recycle product packaging and follow the
recommendations for power use and proper disposal of our products and their components.
If you should have questions regarding our Environmental Policy or any of the environmental services we offer, please contact us at Nokia for
any additional information.
EdenNet Software Update Guide DN1000008332 1-2 Table of Contents
Contents
1 Summary of changes...................................................................................................................................... 4
7 Troubleshooting............................................................................................................................................. 17
7.1 Software update failure as Apps are not started within the stipulated time............................................17
7.2 GUI is unresponsive when accessed from an external IP address........................................................ 17
7.2.1 Add IP addresses to nginx configuration....................................................................................... 18
7.3 Rollback failure at task........................................................................................................................... 19
7.4 Software update failure as SON App Manager API is down..................................................................19
8 Appendices.....................................................................................................................................................20
8.1 Appendix A: Automation of pre-update tasks......................................................................................... 20
8.2 Appendix B: Automation of backup tasks...............................................................................................22
8.3 Appendix C: Automation of post-update tasks....................................................................................... 23
8.4 Appendix D: Backup, pre and post-update report description................................................................25
8.5 Appendix E: Rollback the software update.............................................................................................29
8.5.1 Rolling back for APP Nodes...........................................................................................................29
8.5.2 Rolling back DB changes............................................................................................................... 29
8.5.2.1 Restoring the database snapshot using the MySQL Enterprise backup................................30
8.5.2.2 Restoring the Database virtual machine (DB VM) using AVE............................................... 30
8.5.3 Rollback support............................................................................................................................. 34
8.5.4 Rollback limitations......................................................................................................................... 34
8.6 Appendix F: Rollback after a successful update.................................................................................... 34
8.7 Appendix G: Generating the sudoers file............................................................................................... 35
8.8 Appendix H: Copying the sudoers file to the target servers using ansible............................................. 36
8.9 Appendix I: Setup the control server sudoers for non-root user.............................................................36
8.10 Appendix J: Manually adding more nodes to the existing setup.......................................................... 37
1 Summary of changes
Release Description
EdenNet 21 No change.
Updated sections:
Deleted section:
EdenNet 20 No change.
Release Description
• Appendix J: Importing certificates to Automatic Site Creation from Net-
Act
• Restoring the database snapshot using the MySQL Enterprise backup
• Restoring the Database virtual machine (DB VM) using AVE
• Rolling back for APP Nodes
Updated sections:
Updated sections:
• Rollback support
Deleted section:
Updated section:
Release Description
Deleted section:
• Updating the software - updated the note with secure storage informa-
tion
EdenNet 19 No change.
Release Description
• Troubleshooting - Software update failure as Apps are not started with-
in the stipulated time
EdenNet 18 SP1 1811 This is a new document which describes the procedures to be performed to
update the EdenNet software.
This document describes the tasks to be performed to update the software in an existing EdenNet in-
stance. An outline of the update process is:
1. Obtaining the software update archive and placing it in the control server.
2. Editing the inventory file.
3. Starting the ansible playbook in the control server.
4. Ansible execution of the software update steps in the target EdenNet instance using SSH.
5. Verifying the software update installation.
Before starting the software update process, ensure that the following prerequisites are met:
• Inventory file for the previous EdenNet instance must be available in the control server.
• It is recommended to update the Selfmon node if the Selfmon is installed during EdenNet 20 in-
stallation.
If Selfmon installation or update is required, set the selfmon_enabled = True and uncomment
all the other Selfmon related parameters in the inventory file by filling valid details.
• Sufficient storage space must be available for the software update archive (minimum 1GB must be
free) in the control server.
• Minor software update archive must be downloaded and be available in the control server.
– For information on DB snapshot using MySQL Enterprise backup, see Mounting of external
disk for taking backup of MySQL section in the EdenNet Backup and Restore document.
– An online backup of the database can be taken while the database is running. This is the usu-
al way to take a backup of the database. Online database backup mode can be configured
to full with compression. To do the database backup operation in the required database VMs,
see Taking a full online backup of the database section in the EdenNet Backup and Restore
document.
OR
– For information on DB VM backup using AVE (Avamar Virtual Edition), see Backups of virtual
machines section in the EdenNet Backup and Restore document.
• Backup of configuration files under Central VM must be performed. For backup procedure, see
Backing up of configuration files under Central VM.
• Ensure all the customized equations are the same after the software update. for more information,
see Ensuring all the customized equations are the same after the software update.
• Check the system configurations and verify that the prerequisites are met before upgrading Eden-
Net. For more information see, Appendix A: Automation of pre-update tasks.
• Backup the global module configurations, LMS files, and other configurations to the control server.
For more information see, Appendix B: Automation of backup tasks.
• If there are any modules deprecated in the release, then the respective schedules for such mod-
ules must be deleted and the active instances must be stopped.
• For more information on errata installation, see EdenNet Errata installation guide.
• If the user wants to execute the system acceptance test cases, see SAT Automation Deployment
and Execution Manual.
• If the operator is using Automated Site Creation (ASC) and one or more NetActs are integrated to
ASC, then the RESTful Web Service Data Access API (RESTDA) must be enabled on each Net-
Act to monitor alarms in ASC.
# cp -f /opt/keycloak/standalone/configuration/standalone.xml /home/
vson/standalone.xml.bkp
# cp -f /etc/kong/kong.conf /home/vson/kong.conf.bkp
Expected outcome
3.2 Ensuring all the customized equations are the same after the
software update
To ensure all the customized equations are the same after the software update, do the following.
1. Log in to the Central VM and KPI supplier nodes as a vson user and search for the latest
kpi_equations folder by entering:
# sudo -i -u vson
# mkdir -p /var/tmp/kpi_equations_backup
• Equations.txt
• known_pegs.json
• metadata.json
• peg_aliases.json
As a part of the EdenNet installation, kpi_equations folder is available at the default path /
tmp/config/framework/service/regionService/files/ on App VM (2VM configuration)
and Central VM (5VM/8VM configuration). If the custom KPIs are located at any other path/VM, it
is required to take a back up from the specific path/VM to the kpi_equations_backup folder.
There are three inventory parameters introduced for Fault Management (FM) service:
1. Run the following command to enter the maintenance mode as a vson user in the GUI server
(5VM configuration) or in the App server (2VM configuration):
# enterMaintenanceMode
Note: When the system enters the maintenance mode, all the running or idle modules
will be stopped automatically.
# showMaintenanceMode
For example,
# showMaintenanceMode
System in maintenance mode
3. Take a backup of the existing inventory file from the previous EdenNet installer.
Note: The inventory file will be available at <directory of the Control server>/
installer/ansible_files/
This operation will create a new installer folder in the current directory.
Note: Ensure that the previous directory does not exist before this action.
5. If the <installation_user> is a non-root user, see Appendix I: Setup the control server
sudoers for non-root user before proceeding
7. Update the existing variables and IPs in the new inventory file located at <directory of the
Control server>/installer/ansible_files/ by referring to the backed up inventory file
from step 3.
Note:
• Ensure that you do not copy the content from the old inventory file directly.
• Ensure that force_install, cluster_upgrade, and
force_regenerate_certs variables are set to False in the inventory file.
• If the configuration involves multiple IPs for any node’s interface, the first IP address
given in the inventory file for that node, must be the same as that in /etc/hosts
from the same node. This is required to ensure that nginx, redis, and ldap
services bind to the correct IP address. This check is needed only if any node in the
system has multiple IP interfaces.
• The enable_audit_logging_and_hardening=True/False parameter does
not have any impact on the software update.
CAUTION!
If uid_offset is not configured in the inventory file of previous installation, then configure
the value as uid_offset=0.
8. If the installation user is a non-root user, see Appendix G: Generating the sudoers file
before proceeding.
# export LD_LIBRARY_PATH=/usr/local/lib
Note:
10. To exit the maintenance mode as a vson user in the GUI server (5VM configuration) or the App
server (2VM configuration), enter:
# sudo -i -u vson
# exitMaintenanceMode
Note: When the system exits from maintenance mode, all the iterative modules which
were running or idle before entering the maintenance mode and had completed one it-
eration start automatically. Modules may not abide by the trigger rule (set before the up-
grade) for the next iteration. Non-iterative modules that were stopped, won't start auto-
matically.
# showMaintenanceMode
For example,
# showMaintenanceMode
System not in maintenance mode
Note:
• EdenNet software update is successful if all the listed services are Active.
• If the EdenNet software update is unsuccessful:
7 Troubleshooting
This section describes the error scenarios that might occur during the EdenNet software update and
how they can be resolved.
7.1 Software update failure as Apps are not started within the stipulated
time
Problem:
The EdenNet software update failed at task: launch.enet : Restart enet applications
Cause:
Solution:
The EdenNet software update must be re-triggered by entering the following command:
The GUI is not reachable from external IP addresses but is accessible from internal IP addresses.
Possible cause:
Solution:
With this feature, nginx service running on GUI VM is bound to listen to specific IP addresses. This is
done as part of a security enhancement. Disabling services on interfaces which do not require system
accessibility or by limiting the reachability greatly reduces the potential vulnerabilities offered to an
attacker.
To bind nginx to specific interfaces, the following default configurations must be included in the
gui.conf file located at /home/vson/enet/etc/nginx/conf.d/servers/gui.conf as:
server {
listen <GUI_SERVER_IP>:80 default_server;
listen <GUI_SERVER_IP>:443 default_server ssl;
....
}
Here, the <GUI_SERVER_IP> is the IP address of the GUI VM as mentioned in the inventory file.
If the customer has multiple IP addresses configured on the GUI VM, and the GUI is accessible only
from a limited set of internal IP addresses, it might be due to this service binding. It is recommended to
keep this configuration unchanged.
However, if the customer needs to access the GUI from external IP addresses, and the GUI VM has
multiple IP addresses configured, other IP addresses must be added to the nginx configuration. For
more information, see Add IP addresses to nginx configuration .
ifconfig
3. If the ifconfig command shows more than one IP address, update the configuration in the
gui.conf file located at /home/vson/enet/etc/nginx/conf.d/servers/gui.conf to
include other IP addresses as:
server {
listen <GUI_SERVER_IP>:80 default_server;
listen <GUI_SERVER_IP>:443 default_server ssl;
listen <customer_public_ip_1>:443 default_server ssl;
listen <customer_public_ip_2>:443 default_server ssl;
....
}
Note: It is recommended to expose public IP addresses over port 443 only. (If required,
similar lines can be added for port 80.)
4. Save the changes in gui.conf file and enter the following command:
The EdenNet software Rollback failed at task: Enable the OSS interface.
Cause:
Solution:
The EdenNet software update fails at task: launch.enet : Install SON modules
Cause:
SON App Manager or SON App Manager API is down when module is being imported from SON Mod-
ule Manager component.
Solution:
The EdenNet software update must be re-triggered by entering the following command:
8 Appendices
The user should provide the following parameters in the [all:vars] section of the respective
2VM/5VM/8VM inventory file.
Table 1: Pre-update task execution parameters describes all the pre-update task execution parame-
ters.
Name Description
Name Description
For example,
To check for certain error strings in the log files on the target servers, the following code snippet
must be updated at ../installer/ansible_files/roles/perform-pre-checks/vars/
main.yml to automate the process.
Snippet:
For example,
- name: CENTRAL_VM_SERVERS
log_path: /home/vson/log/enet/appManager.log
genreic_regex: Traceback (most recent call last)
This content is in YAML format. User can add further Groups, log paths, and generic regex to search.
If the regex matches, the respective string will be printed in the ansible logs and the pre-update CSV
report as:
Note:
# ANSIBLE_STDOUT_CALLBACK=skippy ANSIBLE_LOG_PATH="<log_directory>/
ansible_precheck_$(date +%Y_%m_%d_%H_%M_%S).log" ansible-playbook
automate_upgrade.yml --tags "perform_pre_check" -i inventoryfile.2VM
# ANSIBLE_STDOUT_CALLBACK=skippy ANSIBLE_LOG_PATH="<log_directory>/
ansible_precheck_$(date +%Y_%m_%d_%H_%M_%S).log" ansible-playbook
automate_upgrade.yml --tags "perform_pre_check" -i inventoryfile.5VM
Note:
Note:
ANSIBLE_LOG_PATH="<log_directory>/ansible_backup_$(date +%Y_%m_
%d_%H_%M_%S).log" ansible-playbook automate_upgrade.yml --tags
"perform_back_up_of_user_files" -i inventoryfile.2VM
ANSIBLE_LOG_PATH="<log_directory>/ansible_backup_$(date +%Y_%m_
%d_%H_%M_%S).log" ansible-playbook automate_upgrade.yml --tags
"perform_back_up_of_user_files" -i inventoryfile.5VM
After the command is executed, the backed up module global configuration INI and LMS files will be
available on the control server at <path_for_automated_reports_on_control_server>/
backup_of_user_files.
The report containing the mail server configuration, CM schedules, and PM schedule counts will be
available on the control server at <path_for_automated_reports_on_control_server>/
backup_of_user_report.
Note:
The user should update the following parameter in the [all:vars] section of the respective
2VM/5VM/8VM inventory file.
Table 2: Post update parameters describes all the post upgrade parameters.
Name Value
For example,
Note:
ANSIBLE_LOG_PATH="<log_directory>/ansible_post_eden_upgrade_$(date +
%Y_%m_%d_%H_%M_%S).log" ansible-playbook automate_upgrade.yml --tags
"post_upgrade" -i inventoryfile.2VM --extra-vars "selfmon_enabled=False
license_expiry_time=<number_of_days_until_license_expires>"
• For 5VM configuration:
ANSIBLE_LOG_PATH="<log_directory>/ansible_post_eden_upgrade_$(date +
%Y_%m_%d_%H_%M_%S).log" ansible-playbook automate_upgrade.yml --tags
"post_upgrade" -i inventoryfile.5VM --extra-vars "selfmon_enabled=False
license_expiry_time=<number_of_days_until_license_expires>"
• For 8VM configuration:
ANSIBLE_LOG_PATH="/ansible_post_eden_upgrade_$(date + %Y_%m_
%d_%H_%M_%S).log" ansible-playbook automate_upgrade.yml --tags
"post_upgrade" -i inventoryfile.8VM --extra-vars "selfmon_enabled=False
license_expiry_time=<number_of_days_until_license_expires>"
After the command is executed, a report containing the mail server configuration,
CM schedules, and PM schedule counts will be available on the control server at
<path_for_automated_reports_on_control_server>/post_upgrade_reports.
the details of the execution as described in Table 3: Pre-update CSV file values. This generated CSV
report contains all the relevant details of each target server which is listed under the target hostname.
Value Description
Total Shared RAM Memory used by the tmpfs file system in the target
server
Total SWAP used Total used SWAP memory in the target server
Total SWAP free Total free SWAP memory in the target server
System Uptime Days/(HH:MM) Time for which the target server has been active
Java version check Checks whether the JAVA version is higher than 2 on
the App servers
OpenSSL version check Checks whether the OpenSSL version is higher than
1.0.1e on the GUI Server
NSS Version check status on Central DB VM Checks whether the NSS version is higher than 3.20
on Central DB VM
Value Description
NSS Version check status on Central APP Checks whether the NSS version is higher than 3.20
VM on Central APP VM
Table 4: Optional pre-update CSV file values describes the optional values for pre-update reports
which is available only when their respective condition is satisfied.
Disk usage summary for /var This is available when the /var directory has
less than 5 GB available in the target server.
Generic error search status This is available when there are user provided
errors in the ../installer/ansible_files/
roles/perform-pre-checks/vars/main.
yml file.
Disk usage summary for user provided threshold This is available when any of the drive partitions
exceed the user allocated max_resource_us-
age_threshold.
Disk usage summary for /home This is available when the /home directory has
less than 2.5GB available in the target server.
Disk usage summary for site_packages_tmp_ This is available when the site_packages_
dir provided in the inventory file tmp_dir has less than 7GB available in the tar-
get server.
Table 5: Backup and post-upgrade report sheets describes the back up and post update reports that
are generated as an excel file with different sheets containing information as follows.
There will be one sheet for every available EdenNet App server.
It consists of two columns the Application name running on the
host and its status (ACTIVE or INACTIVE or FAILED).
Son Exclusion List This reports cell labels and cluster names that were SON exclud-
ed.
Email Configuration This reports the different settings of the configured email service.
User Accounts This provides all the details of the LDAP user accounts config-
ured in EdenNet.
Oss Access Data This provides details regarding OSS which are integrated to
EdenNet and the current status of OSS (Enabled or Disabled).
Expiring License Data This lists out all the licenses which are about to expire in a fixed
amount of days provided by user.
Enet Version This contains information about the current EdenNet version
available along with its the build details.
# export LD_LIBRARY_PATH=/usr/local/lib
Note:
WARNING! Do not re-attempt the EdenNet software update unless the rollback is
successful.
Note: This procedure is required and supported only for Central DB, GUI DB, and KPI DBs.
Any reference to the DB nodes provided in the following section is for Central DB, GUI DB,
and KPI DB nodes.
8.5.2.1 Restoring the database snapshot using the MySQL Enterprise backup
This section describes the procedure for restoring the database snapshot using the MySQL Enterprise backup.
1. Restore the MySQL database by following the instructions in Restoring the database from backups
section in the EdenNet Backup and Restore document.
# cd /opt/keycloak/standalone/configuration/
5. Edit and replace the keycloak_db_password in the standalone.xml file with the password
retrieved from the /home/vson/standalone.xml.bkp file. For reference, the password will be
available under this section:
<security>
<user-name>keycloak</user-name>
<password>keycloak_db_password</password>
</security>
7. Once the EdenNet system is operational, remove the temporary certificate directory on all DB
nodes by entering:
# rm -rf /opt/nokia/mysql_cert_backups
Expected outcome
8.5.2.2 Restoring the Database virtual machine (DB VM) using AVE
This section describes the procedure for restoring the Databse VM using AVE.
1. Back up the following files from the Central VM DB to a non DB node. For example, control server
or any of the APP VMs.
• /opt/cassandra-for-kong/conf/cassandra_kong.p12
• /opt/cassandra-for-kong/conf/cassandra.yaml
• /opt/nokia/mysql/certs/mysqlclient-keystore.jks
• /opt/nokia/mysql/certs/mysqlclient-keystore.p12
• /opt/nokia/mysql/certs/mysql_client.crt
• /opt/nokia/mysql/certs/mysql_client.key
• /opt/nokia/mysql/certs/ca_cert.crt
• /home/data/mysql/mysql_server.crt
• /home/data/mysql/mysql_server.key
2. Restore the DB VMs by following the Restoring virtual machine backups section in the EdenNet
Backup and restore document.
Note: After the DB VM restore is completed and virtual machine is powered on, continue
even if you see any errors while checking the enet app status.
3. Stop all the apps on the APP nodes by entering the following command as a vson user:
4. Stop the enet service on all APP nodes by entering the following command as root user:
6. Stop the mysqld service on all DB nodes by entering the following command as root user:
7. Stop the cassandra_kong service on Central VM DB node by entering the following command as
root user:
8. Restore the files which were backed up in step 1. See Table 6: File backup details for more
information.
Note: ca_cert.crt file has to be copied to two locations on all the DB Nodes.
# cd /opt/cassandra-for-kong/conf/
b) Edit and replace the keystore_password at two places in the cassandra.yaml file
with the password retrieved from the backed up file /opt/cassandra-for-kong/conf/
cassandra.yaml in step 1.
server_encryption_options:
internode_encryption: none
keystore: /opt/cassandra-for-kong/conf/cassandra_kong.p12
keystore_password: <password>
…
…
client_encryption_options:
enabled: true
# If enabled and optional is set to true encrypted and
unencrypted connections are handled.
optional: true
keystore: /opt/cassandra-for-kong/conf/cassandra_kong.p12
keystore_password: <password>
10. Start the mysqld service on all DB nodes by entering the following command as root user:
11. Log in to Central VM as root user and perform the following steps:
a) Navigate to the keycloak configuration directory by entering:
# cd /opt/keycloak/standalone/configuration/
b) Edit and replace the keycloak_db_password in the standalone.xml file with the
password retrieved from the /home/vson/standalone.xml.bkp file. For reference, the
password will be available under this section:
<security>
<user-name>keycloak</user-name>
<password>keycloak_db_password</password>
</security>
# cd /etc/kong
d) Edit and replace the cassandra_password value in the kong.conf file with the password
retrieved from the /home/vson/kong.conf.bkp file. For reference, see:
cassandra_password = <password>
12. Start the enet service on all APP nodes by entering the following command as root user:
13. Start all the apps on APP nodes by entering the following command as vson user:
Expected outcome
The user must import the module manually after a successful rollback.
# sudo -i -u vson
3. Rename the file as venv_backup.gz while retaining the same read-write permissions.
4. If the application server has the /home/custom directory, log out from vson by entering:
# exit vson
# sudo -i -u custom
6. Rename the file as venv_backup.gz while retaining the same read-write permissions.
7. Trigger the rollback by following the steps described in Appendix E: Rollback the software update.
Note: Based on which sudoers file needs to be generated, <tags> can be replaced in
the command with the required tag/tags as per the following table:
For example,
Multiple tags can be given at once, as comma separated values, to generate the
respective sudoers file.
8.8 Appendix H: Copying the sudoers file to the target servers using
ansible
8.9 Appendix I: Setup the control server sudoers for non-root user
1. Log in to the control server as an <installation_user>.
2. Navigate to ansible_files directory by entering:
3. Execute the create_cs_sudoers script file to generate the sudoers file for control server by
entering:
# sh create_cs_sudoers.sh
4. The eden_cs_<installation_user>_sudoers file is generated in your current working
directory. This file must be copied to the /etc/sudoers.d/ directory.
2. For manually adding task or custom task nodes, see Manually adding task and custom task nodes
on demand section in the EdenNet Installation and Upgrade Guide.