sys_WordMark_AT_Pag

e1
RANGER HA RUNBOOK FOR SLB
PROD/QA/DEV ENVIRONMENTS

© Copyright 2019, ATOS PGS sp. z o.o. All rights reserved. Reproduction in whole or in part is prohibited without the prior
written consent of the copyright owner. For any questions or remarks on this document, please contact Atos Poland Global
Services, +48 22 4446500.

AUTHOR(S) : Patryk Palimaka

DOCUMENT NUMBER :
VERSION : 0.1
STATUS : Draft
SOURCE : Atos Poland Global Services
DOCUMENT DATE : 20 November 2019RELEASED FOR TRAININGRELEASED FOR
OPERATIONSREVIEW BEFORE
NUMBER OF PAGES : 9

OWNER
HDFS HA Runbook for SLB PROD/QA/DEV environments

sys_WordMark_AT_Continued
version: 0.1

Public

document number:

Contents
1 Audience and document purpose...................................................................
2 Components in scope...................................................................................
3 Ranger.......................................................................................................
3.1 Ranger database ackup and restoration....................................................
3.2 Ranger HA configuration.........................................................................
4 Ranger KMS................................................................................................
5 Apache SOLR..............................................................................................

Atos Poland Global Services

20 November 2019
2 of 9
HDFS HA Runbook for SLB PROD/QA/DEV environments

sys_WordMark_AT_Continued
version: 0.1

Public

document number:

List of changes
version Date Description Author(s)
0.1 20.11.2019 Initial document structure created Patryk
Palimaka

Atos Poland Global Services

20 November 2019
3 of 9
 HDFS HA Runbook for SLB PROD/QA/DEV environments

sys_WordMark_AT_Continued
version: 0.1

Public

document number:

1 Audience and document purpose

The document has been prepared for the SLB HDP platform administrators and ATOS team
responsible for maintaining the PROD/QA and DEV environments. End-user/business team was
not meant as a participant in the process nor the document recipient.
Scope of the document describes the current (for the date of document creation) configuration,
processes and detailed steps leading to archive, update and restore the services functionality in
case of HA/DR drill or real-life issue.
Processes described in this document were based on the vendor (HortonWorks) best practices
and/or documentation. Links to them are the integral part of the knowledge required to operate
the runbook.
During this runbook creation – authors followed the suggestions brought together in the
following articles:
Part one:
https://community.cloudera.com/t5/Community-Articles/Disaster-recovery-and-Backup-best-
practices-in-a-typical/ta-p/246641
and part two:
https://community.cloudera.com/t5/Community-Articles/Disaster-recovery-and-Backup-best-
practices-in-a-typical/ta-p/246651
For the knowledge systematization – the following vendor support case was created by Arshad
Amir Jamadar:
https://my.cloudera.com/cases/639712/comments

Atos Poland Global Services

20 November 2019
4 of 9
 HDFS HA Runbook for SLB PROD/QA/DEV environments

sys_WordMark_AT_Continued
version: 0.1

Public

document number:

2 Components in scope
To cover the HA/DR processes for Ranger service = the following components were described:

1. Ranger

2. Ranger KMS

3. Apache Solr

Atos Poland Global Services

20 November 2019
5 of 9
 HDFS HA Runbook for SLB PROD/QA/DEV environments

sys_WordMark_AT_Continued
version: 0.1

Public

document number:

3 Ranger
Ranger is a framework to enable, monitor and manage comprehensive data security across
the Hadoop platform. The vision with Ranger is to provide comprehensive security across the
Apache Hadoop ecosystem.

https://docs.cloudera.com/HDPDocuments/HDP2/HDP-2.3.2/bk_Ranger_User_Guide/content/
ch09.html

3.1 Ranger database backup and restoration

The dedicated PostgreSQL runbook covers all the technical steps and proceedings related to
HA/DR in general. This part of the Ranger runbook describes the basic Ranger DB backup.

Vendor documentation proposes the following:

https://docs.cloudera.com/HDPDocuments/Ambari-2.1.2.1/bk_upgrading_Ambari/content/
_perform_backups_mamiu.html

https://docs.cloudera.com/HDPDocuments/Ambari-2.1.1.0/bk_ambari_reference_guide/content/
_back_up_current_data.html

1. Backup Ranger and Ranger-audit DBs

psqldump $dbname > $outputfilename.sql

2. (new server) Install PostgreSQL

yum -y install postgresql-server

3. Install Ranger via Ambari

4. Stop Ranger

5. Enter psql cli and drop the ranger db

DROP DATABASE IF EXISTS ranger;

6. Recreate Ranger dbase

CREATE DATABASE ranger;

GRANT ALL ON DATABASE ranger TO ranger;

7. Restore ranger dbase via psql command

psql $dbname < $inputfilename.sql

8. Update Ranger admin pass (if you change your admin pass) on Ambar/Ranger

Atos Poland Global Services

20 November 2019
6 of 9
 HDFS HA Runbook for SLB PROD/QA/DEV environments

sys_WordMark_AT_Continued
version: 0.1

Public

document number:
9. Start Ranger.

3.2 Ranger HA configuration

Special Requirements for High Availability Environments In a HA environment, primary and
secondary NameNodes must be configured as described in the HDP System Administration Guide.

To enable Ranger in the HDFS HA environment, the HDFS plugin must be set up in each
NameNode, and then pointed to the same HDFS service set up in the Security Manager. Any
policies created within that HDFS service are automatically synchronized to the primary and
secondary NameNodes through the installed Apache Ranger plugin. That way, if the primary
NameNode fails, the secondary namenode takes over and the Ranger plugin at that NameNode
begins to enforce the same policies for access control.

https://docs.cloudera.com/HDPDocuments/HDP2/HDP-2.3.2/bk_Ranger_User_Guide/
content/ch09.html

Step by step tutorial below:

https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.4/fault-tolerance/configuring-
fault-tolerance.pdf

Atos Poland Global Services

20 November 2019
7 of 9
 HDFS HA Runbook for SLB PROD/QA/DEV environments

sys_WordMark_AT_Continued
version: 0.1

Public

document number:

4 Ranger KMS
Ranger Key Management Server (KMS) is based on the Hadoop KMS developed by the Apache
community. It extends the native Hadoop KMS functions by letting you store keys in a secure
database.

https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/
TransparentEncryption.html#Key_Management_Server_KeyProvider_EDEKs

1. Backup Ranger KMS and DBs

psqldump $dbname > $outputfilename.sql

2. (new server) Install PostgreSQL

yum -y install postgresql-server

3. Install Ranger KMS via Ambari

4. Stop Ranger KMS

5. Enter psql cli and drop the Ranger KMS db

DROP DATABASE IF EXISTS rangerkms;

6. Recreate Ranger KMS dbase

CREATE DATABASE rangerkms;

GRANT ALL ON DATABASE rangerkms TO rangerkms;

7. Restore Ranger KMS dbase via psql command

psql $dbname < $inputfilename.sql

8. Update Ranger KMS admin pass (if you change your admin pass) on Ambar/Ranger KMS

9. Start Ranger KMS.

Atos Poland Global Services

20 November 2019
8 of 9
 HDFS HA Runbook for SLB PROD/QA/DEV environments

sys_WordMark_AT_Continued
version: 0.1

Public

document number:

5 Apache SOLR
Solr is an open-source enterprise-search platform, written in Java, from the Apache Lucene
project. Its major features include full-text search, hit highlighting, faceted search, real-time
indexing, dynamic clustering, database integration, NoSQL features and rich document handling.

https://blog.cloudera.com/how-to-backup-and-disaster-recovery-for-apache-solr-part-i/

Collections backup:

1. Create a snapshot.

solrctl collection --create-snapshot [my-snap] -c [collection-name]

2. Place snapshot in HDFS in backup file.

sudo -u solr solrctl collection --export-snapshot [my-snap] -c [collection-name] -d

/solr-backups

3. Restore from the backup.

sudo -u solr solrctl collection --restore books_restored -l /solr-backups -b [my-snap] -

i req_0

Atos Poland Global Services

20 November 2019
9 of 9