Hortonworks upgrade

Prerequisites:

1. Ensure all services are up & running in the Cluster without any issues
2. Run each Service Check (found under the Service Actions menu) and confirm they execute
successfully.
3. Clear all alerts, or understand why they are being generated. Remediate as necessary.
4. Get all custom jars from all tools like Sqoop, Pig & Hive from /var/lib/ to not to fail jobs after
upgrade.
5. Make a note of any nodes to bring down/bring up before upgrade if possible.
6. Make sure enough space under RPM directory for new HDP RPMS.

At the time of upgrade process:

1. Send Communication mail to all developers & business users about downtime window.
2. Perform checkpoint before starting Cluster upgrades.

sudo -u hdfs hdfs dfsadmin -safemode enter

sudo -u hdfs hdfs dfsadmin -saveNamespace
sudo -u hdfs hdfs dfsadmin -safemode leave

3. Run FSCK
clush -g all -b "sudo mkdir /var/www/html/upgrade_backups"
clush -g all -b "sudo chmod 777 /var/www/html/upgrade_backups"

Run below in NameNode machine,

sudo -u hdfs hdfs fsck / -files -blocks -locations -racks >

/var/www/html/upgrade_backups/dfs-fsck-old.txt

4. Get Cluster Status Report

sudo -u hdfs hdfs dfsadmin -report > /var/www/html/upgrade_backups/dfs-report-old.txt

5. Get All Files Report on HDFS

sudo -u hdfs hdfs dfs -ls -R / > /var/www/html/upgrade_backups/dfs-lsr-old.txt

6. Take screenshot for Ambari, NameNode, Hive, Yarn, etc., for future reference
7. Don’t execute this at this time. Stop cluster services. (Note: If you are doing rolling upgrade,
no need to stop cluster services)
8. Backup all MySQL databases (Hive Metastore, Oozie, etc..)
 sudo mysqldump -h ham.hadoop.com -u root -pwelcome1 --all-databases --add-drop-
database --add-drop-table > /var/www/html/upgrade_backups/ambari-mysql-backup.sql

9. NameNode Metadata Directory backup

clush -g nn -b "sudo mkdir -p /var/www/html/upgrade_backups/nn_data"

clush -g nn -b "sudo cp -p -r /mnt/disk1 /var/www/html/upgrade_backups/nn_data"

This includes fsimage and journal nodes data.

10. DataNode VERSION file back up

clush -g nn -b "sudo mkdir -p /var/www/html/upgrade_backups/dn_data"

clush -g dn -b "sudo cp -p /mnt/disk1/data/current/VERSION
/var/www/html/upgrade_backups/"

11. Zookeeper Data Backup

clush -g zk -b "sudo cp -p -r /hadoop/zookeeper /var/www/html/upgrade_backups"

12. Perform the upgrade for Ambari and HDP

clush -g am -b "sudo /usr/sbin/ambari-server stop"

clush -g hadoop -b "sudo /usr/sbin/ambari-agent stop"

Copy Hortonworks new version RPMs to /var/www/html directory and extract them.
cd /var/www/html/hortonworks-2.6.2.0
sudo tar -zxvf ambari-2.5.2.0-centos7.tar.gz -C /var/www/html/hortonworks-2.6.2.0/
sudo tar -zxvf HDP-2.6.2.0-centos7-rpm.tar.gz -C /var/www/html/hortonworks-2.6.2.0/
sudo mkdir -p /var/www/html/hortonworks-2.6.2.0/HDP-UTILS
sudo tar -zxvf HDP-UTILS-1.1.0.21-centos7.tar.gz -C /var/www/html/hortonworks-2.6.2.0/HDP-UTILS
sudo rm -rf /var/www/html/hortonworks-2.6.2.0/*.tar.gz
sudo chown -R root:root /var/www/html/hortonworks-2.6.2.0

Remove any other repos if we have.

clush -ab "sudo mkdir -p /etc/yum.repos.d/baserepos"

clush -ab "sudo mv -f /etc/yum.repos.d/* /etc/yum.repos.d/baserepos/"
clush -ab "sudo ls /etc/yum.repos.d/"

Create REPO files:

sudo vi /etc/yum.repos.d/ambari.repo

add below content to the file:

 [Updates-ambari-2.5.2.0]
name=ambari-2.5.2.0 – Updates
baseurl=http://192.168.56.121/hortonworks-2.6.2.0/ambari/centos7/
gpgcheck=1
gpgkey=http://192.168.56.121/hortonworks-2.6.2.0/ambari/centos7/RPM-GPG-KEY/RPM-GPG-KEY-
Jenkins
enabled=1
priority=1
Create/Change HDP.repo as below
sudo vi /etc/yum.repos.d/HDP.repo

add below content to the file:

[HDP-2.6.2.0]
name=HDP Version - HDP-2.6.2.0
baseurl=http://192.168.56.121/hortonworks-2.6.2.0/HDP/centos7
gpgcheck=1
gpgkey=http://192.168.56.121/hortonworks-2.6.2.0/HDP/centos7/RPM-GPG-KEY/RPM-GPG-KEY-
Jenkins
enabled=1
priority=1

[HDP-UTILS-1.1.0.21]
name=HDP Utils Version - HDP-UTILS-1.1.0.21
baseurl=http://192.168.56.121/hortonworks-2.6.2.0/HDP-UTILS/
gpgcheck=1
gpgkey=http://192.168.56.121/hortonworks-2.6.2.0/HDP-UTILS/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

Clean old Repos

sudo yum clean all

Upgrade Ambari UI

sudo yum list ambari-server

sudo yum upgrade -y ambari-server

Check whether ambari-server is upgraded with new version

ll /usr/lib/ambari-server | grep ambari

sudo rpm -qa | grep ambari-server
Setup new version Ambari Server:
sudo ambari-server upgrade -j /usr/java/default

Using python /usr/bin/python

Upgrading ambari-server
 Updating properties in ambari.properties ...

WARNING: Original file ambari-env.sh kept

Fixing database objects owner

Ambari Server configured for MySQL. Confirm you have made a backup of the Ambari Server
database [y/n] (y)? y

Upgrading database schema

Adjusting ambari-server permissions and ownership...

Ambari Server 'upgrade' completed successfully.

Upgrade Ambari Agents

clush -g hadoop --copy /etc/yum.repos.d/ambari.repo /etc/yum.repos.d/HDP.repo --dest

/tmp/
clush -g hadoop -b "sudo cp -r /tmp/ambari.repo /tmp/HDP.repo /etc/yum.repos.d/"
clush -g hadoop -b "ls /etc/yum.repos.d"
clush -g hadoop -b "sudo yum upgrade -y ambari-agent"

Check whether ambari-agents are upgraded with new version

clush -g hadoop -b "sudo rpm -qa | grep ambari-agent"

Start Ambari Server

sudo /usr/sbin/ambari-server start

This will start CM with MySQL. To see the logs,

sudo tailf /var/log/ambari-server/ambari-server.log

Start Ambari Agents

clush -g hadoop -b "sudo /usr/sbin/ambari-agent start"

Login to CM Server UI.

http://192.168.56.111:7180

username: admin
password: admin

Start Cluster Services.

Clear conflict RMPs from CentOS.

clush -g all -b "sudo systemctl restart network"

clush -g all -b "sudo cp /etc/yum.repos.d/baserepos/CentOS-* /etc/yum.repos.d/"
clush -g all -b "sudo yum remove -y ipa-server-common-4.4.0-14.el7.centos.7.noarch"
 clush -g all -b "sudo yum remove -y ipa-common-4.4.0-14.el7.centos.7.noarch"
clush -g all -b "sudo yum remove -y ipa-client-common-4.4.0-14.el7.centos.7.noarch"
clush -g all -b "sudo rm -rf /etc/yum.repos.d/CentOS-*"
clush -g all -b "sudo yum clean all"
clush -g all -b "sudo ls -l /etc/yum.repos.d"

Go to admin -> Manage Ambari -> Versions

Click on +Register Version

Enter 3.0 in Name Text box

Enter the location of local repositories as below:

For HDP-2.6 http://192.168.56.121/hortonworks-2.6.2.0/HDP/centos7/

For HDP-UTILS-1.1.0.20 http://192.168.56.121/hortonworks-2.6.2.0/HDP-UTILS

Click on Save

You will get below screen.

Click on Install on… and select the name of your cluster.

Click on Install.
This will start installing on all nodes.
Click on Upgrade.

After Upgrade, select “Show Details”

Click on “Perform Upgrade”

Make sure there are no required checks for Express Upgrade.
Click on “Finalize” to complete the Hortonworks Upgrade.
Click on Admin -> Stack and Versions. You should see HDP-2.6.2.0 version as current here.