HBase Monitoring Plugin for Oracle Enterprise Manager

Administration and Installation Guide

v12.1.0.3.0
Prerequisites  .....................................................................................................................................................................................................................................................................................  3  
Plug-­‐in  installation  (OMS)  ...........................................................................................................................................................................................................................................................  5  
Plug-­‐in  installation  (Agent)  .......................................................................................................................................................................................................................................................  14  
Target  Addition  ..............................................................................................................................................................................................................................................................................  23  
Setting  OS  credentials  ..................................................................................................................................................................................................................................................................  32  
Component  Control  ......................................................................................................................................................................................................................................................................  35  
APPENDIX  A    -­‐  EXAMPLE  SSL  CONFIGURATION  .............................................................................................................................................................................................................  37  

2
Prerequisites

• Oracle Enterprise Manager 12c R3 or above (including 13c).

• The agent side host should be an EM supported Linux 32 or 64 bit platform

• HBase version 0.95 and above

• An Oracle EM 12.1.0.3.0 or higher agent deployment is required on each monitoring host. This can be remote to the HBase installation.

• Access should be given to the agent OS user to run remote commands for component start/stop operations.

• JMX publishing of org.apache.hadoop.metrics2 should be enabled on the HBase side with appropriate username/password security.

• If SSL is used, please follow the SSL configuration guide to provide the relevant keystores/certificates on the agent side.

3
4
Plug-in installation (OMS)

The plug-in should be installed into the EM environment using emcli.

Following download of the opar file from http://www.aidev.uk please import using the following command as SYSMAN:

emcli import_update -file={full path to opar file} -omslocal

This imports the plug-in into the EM environment and makes it visible within the EM console

eg:

$ emcli import_update -file=/home/oracle/ 12.1.0.3.0_aidev.hbase.xdbh_2000_0.opar -omslocal

Processing update: Plug-in - Aidev HBase system monitoring plugin for Oracle Enterprise Manager
Successfully uploaded the update to Enterprise Manager. Use the Self Update Console to manage this update.

5
Within Oracle Enterprise Manager, deploy the plug-in.

Navigate to "Setup" , “Extensibility”, "Plugins" then expand the "Databases" section:

Select the HBase Database plugin.

6
left click on "Deploy on" then "Management Servers"

7
Ensure the correct version is shown

Click Next

8
ensure the prerequisites succeed then click "Next"

9
Enter the repository SYS password or use an existing named credential

Click Next

10
Review the information shown

If all is correct, click Deploy

NB. No outage is required for this operation

11
Click Show Status:

12
after a few minutes, this should succeed

13
Plug-in installation (Agent)

Each monitoring agent requires the HBase plugin to be deployed to it before it can monitor HBase targets.

This should be undertaken prior to adding targets.

From the EM homepage, Setup ->Extensibility -> Plugins

14
expand the "Databases" section:

Select the HBase plugin.

15
Select Deploy On -> Management Agent

16
Click Continue

17
Select the agents to deploy to, click Continue

18
Ensure the prerequisite checks succeed, click Next

19
Review and click Deploy:

20
Click Show Status to see the job:

21
This should complete after a period of time:

22
Target Addition

Targets are added from the EM user interface.

Setup->Add Target-> Add Targets Manually

23
Choose Add Non-Host Targets Using Declarative Process:

24
Click spyglass to select the host to add the target to:

25
Select the host then click Select

26
Select HBase Database in the target type box:

Click Add to view the Target Properties screen

27
The target properties screen will be shown:

28
Enter the target properties as follows:

Target:

Target Name : enter the target name, eg. hbase01

HBase Monitoring Credentials:

Username : enter jmx monitoring username, eg. monitorRole

Password : enter password for this account

Confirm Password : confirm the password

Properties:

Admin Port number : enter jmx port number

Custom lookup provider class : do not set

HBase Environment Identifier : set this attribute to link HBase targets togerther in an environment

Machine Name : enter IP address or hostname of HBase machine

Use SSL : true or false

Click OK

29
The target will be added:

NOTE: FOR SSL ENABLED ENVIRONMENTS, PLEASE FOLLOW APPENDIX A FOR GUIDANCE ON THE REQUIRED CONFIGURATION
TO ENABLE TARGET MONITORING AND CONTROL

30
The targets should now be visible:

31
Setting OS credentials

To be able to control HBase components from with EM, OS credentials are required for each target.

To do this, Setup-> Security -> Preferred Credentials

32
Select the target type and then Manage Preferred Credentials

33
For the Host Credentials credential set, you can set for each target or as a default for all targets of this type:

34
Component Control

HBase components can be controlled from within the EM user interface.

35
A prerequisite of this functionality is the setting of Host Credentials for the target in question.

Additionally, users should configure site specific standard access for the EM agent to call the relevant scripts.

The agent side location of {agent base}/plugins/aidev.hbase.xdbh.agent.plugin_12.1.0.3.0/scripts holds the files that call the operation.

• stop_hbase.sh

• start_hbase.sh

36
APPENDIX A - EXAMPLE SSL CONFIGURATION

This section details the steps required when configuring SSL for HBase -> Oracle EM. This should serve as an example and should be used in
conjunction with HBase/Hadoop vendor recommendations.

Please refer to the Apache HBase documentation as the primary reference for JMX configuration.

In this example we have a directory structure as so:

$HOME
/ssl
/client
/server
/ca

37
Step 1 - Create a CA key and certificate

(only required if no site CA is available to sign the certificates)

openssl genrsa –out rootCA.key 4096

Example:

hbase@hbase1:~/ssl/ca$ openssl genrsa -out rootCA.key 4096

Generating RSA private key, 4096 bit long modulus
...........++
.....++
e is 65537 (0x10001)

Generate the root certificate

openssl req -x509 -new -nodes -key rootCA.key -days 3650 -out rootCA.pem

Example:

hbase@hbase1:~/ssl/ca$ openssl req -x509 -new -nodes -key rootCA.key -days 3650 -out rootCA.pem
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.

38
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:UK
State or Province Name (full name) [Some-State]:SCO
Locality Name (eg, city) []:FK
Organization Name (eg, company) [Internet Widgits Pty Ltd]:AIDEV
Organizational Unit Name (eg, section) []:AIDEV UK
Common Name (e.g. server FQDN or YOUR name) []:ROOTCA
Email Address []:info@aidev.uk

39
Step 2 - Create Client keystore, request and certificate:

This is used by the client (jconsole (for testing) and the EM agent and agent-side scripts)

cd ../client

keytool -genkeypair -alias hbaseclient -keyalg RSA -keysize 4096 -keypass welcome1 -keystore
hbaseclient.keystore -storepass welcome1 -validity 3650

Example:
What is your first and last name?
[Unknown]: HBASECLIENT
What is the name of your organizational unit?
[Unknown]: AIDEV UK
What is the name of your organization?
[Unknown]: AIDEV
What is the name of your City or Locality?
[Unknown]: FK
What is the name of your State or Province?
[Unknown]: SCO
What is the two-letter country code for this unit?
[Unknown]: UK
Is CN=HBASECLIENT, OU=AIDEV UK, O=AIDEV, L=FK, ST=SCO, C=UK correct?
[no]: yes

40
Create the request:

keytool -certreq -file hbaseclient.req -keystore hbaseclient.keystore -alias hbaseclient -storepass

welcome1 -keypass welcome1

Sign the client certificate request

cp hbaseclient.req ../ca

cd ../ca

openssl x509 -req -CA rootCA.pem -CAkey rootCA.key -in hbaseclient.req -out hbaseclient.cer -days 1000 -
CAcreateserial

Output:

Signature ok
subject=/C=UK/ST=SCO/L=FK/O=AIDEV/OU=AIDEV UK/CN=HBASECLIENT
Getting CA Private Key

41
Step 3 - Create Server keystore, request and certificate

cd ../server

keytool -genkeypair -alias hbaseserver -keyalg RSA -keysize 4096 -keypass welcome1 -keystore
hbaseserver.keystore -storepass welcome1 -validity 3650

Output:

What is your first and last name?

[Unknown]: HBASESERVER
What is the name of your organizational unit?
[Unknown]: AIDEV UK
What is the name of your organization?
[Unknown]: AIDEV
What is the name of your City or Locality?
[Unknown]: FK
What is the name of your State or Province?
[Unknown]: SCO
What is the two-letter country code for this unit?
[Unknown]: UK
Is CN=HBASESERVER, OU=AIDEV UK, O=AIDEV, L=FK, ST=SCO, C=UK correct?
[no]: yes

42
Create the request:

keytool -certreq -file hbaseserver.req -keystore hbaseserver.keystore -alias hbaseserver -storepass

welcome1 -keypass welcome1

Sign the server certificate request:

cp hbaseserver.req ../ca

cd ../ca

openssl x509 -req -CA rootCA.pem -CAkey rootCA.key -in hbaseserver.req -out hbaseserver.cer -days 1000 -
CAcreateserial

Output:

Signature ok
subject=/C=UK/ST=SCO/L=FK/O=AIDEV/OU=AIDEV UK/CN=HBASESERVER
Getting CA Private Key

43
Step 4 - Import the rootCA, client and server certificates (client side keystore):

cd ../client

Copy the certificates to the client directory:

cp ../ca/rootCA.pem .

cp ../ca/hbaseclient.cer .

cp ../ca/hbaseserver.cer .

Import the root CA certificate:

keytool -importcert -trustcacerts -file rootCA.pem -alias rootCA –keystore hbaseclient.keystore -

storepass welcome1

Import the client certificate:

keytool -keystore hbaseclient.keystore -storepass welcome1 -alias hbaseclient -import -file

hbaseclient.cer -keypass welcome1

44
Output:

Certificate reply was installed in keystore

Import the server cert as a trusted certificate:

keytool -importcert -trustcacerts -file hbaseserver.cer -alias hbaseserver -keystore hbaseclient.keystore

-storepass welcome1

45
Step 5 - Import the rootCA, client and server certificates (server side keystore):

cd ../server

Copy the certificates to the client directory:

cp ../ca/rootCA.pem .

cp ../ca/hbaseclient.cer .

cp ../ca/hbaseserver.cer .

Import the root CA certificate:

keytool -importcert -trustcacerts -file rootCA.pem -alias rootCA –keystore hbaseserver.keystore -

storepass welcome1

Import the server certificate:

keytool -keystore hbaseserver.keystore -storepass welcome1 -alias hbaseserver -import -file

hbaseserver.cer -keypass welcome1

46
Output:

Certificate reply was installed in keystore

Import the client cert as a trusted certificate:

keytool -importcert -trustcacerts -file hbaseclient.cer -alias hbaseclient -keystore hbaseserver.keystore

-storepass welcome1

47
Step 6 – Configure HBase to require SSL for remote JMX connections

On the HBASE side, an example configuration:

hbase-env.sh:

export HBASE_JMX_OPTS="-Dcom.sun.management.jmxremote.password.file=$HBASE_HOME/conf/jmxremote.passwd"
export HBASE_JMX_OPTS="$HBASE_JMX_OPTS -Djavax.net.ssl.keyStorePassword=welcome1 -
Djavax.net.ssl.keyStore=/home/hbase/ssl/server/hbaseserver.keystore"
export HBASE_JMX_OPTS="$HBASE_JMX_OPTS -
Dcom.sun.management.jmxremote.access.file=$HBASE_HOME/conf/jmxremote.access"
export HBASE_JMX_BASE="-Dcom.sun.management.jmxremote.ssl=true -
Dcom.sun.management.jmxremote.authenticate=true"

jmxremote.passwd :

monitorRole monitorpass
controlRole controlpass

jmxremote.access :

monitorRole readonly
controlRole readwrite

Restart HBase for the changes to take effect

48
Step 7 – Test remote JMX connectivity through jconsole

Copy the client keystore to the jconsole client machine

Run jconsole:

jconsole -J-Djavax.net.ssl.trustStore=hbaseclient.keystore -J-Djavax.net.ssl.trustStorePassword=welcome1

Enter host, port and username, password.

The connection should succeed and mbeans should be visible.

49
Step 8 – Oracle EM agent side configuration

The EM agent requires the addition of the server certificate and rootCA certificate to be added as trusted certs.

Copy the rootCA.pem and hbaseserver.cer files to the agent host

cd {location of certificate files copied over}

as the agent OS user, add the root certificate:

emctl secure add_trust_cert_to_jks -trust_certs_loc ./rootCA.pem -alias rootCA

The default password is welcome

Output:

Oracle Enterprise Manager Cloud Control 12c Release 3

Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved.
Password:

Message : Certificate was added to keystore

ExitStatus: SUCCESS

50
add the server certificate:

emctl secure add_trust_cert_to_jks -trust_certs_loc ./hbaseserver.cer -alias hbaseserver_hbase01

Output:

Oracle Enterprise Manager Cloud Control 12c Release 3

Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved.
Password:

Message : Certificate was added to keystore

ExitStatus: SUCCESS

Restart the EM agent:

<AGENT_HOME>/bin>./emctl stop agent

<AGENT_HOME>/bin>./emctl start agent

51
Copy the client keystore to the HBase monitoring plugin scripts directory, for example

/u01/OracleHomes/agent/plugins/aidev.hbase.xdbs.agent.plugin_12.1.0.3.0/scripts

Amend the status monitoring script:

vi hbase_resp.sh

uncomment and amend the following line:

export SSLOPTS="-Djavax.net.ssl.trustStore=$dir/hbaseclient.keystore -
Djavax.net.ssl.trustStorePassword=welcome1"

The target should now show as Up in EM and all metrics should be captured by the agent.

52
NEED FURTHER INFORMATION? contact info@aidev.uk for more details on this product and how to join up with us.

Copyright © 2016 Aidev [A trading name of Wardrop Consulting Limited]. All rights reserved.
Oracle and Oracle Enterprise Manager are trademarks or registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their
respective owners.

Specifications and product offerings subject to change without notice.

Please ensure you have appropriate licenses for running this product. Aidev takes no responsibility whatsoever re. site licensing of Oracle EM, HBase or
any other software used with this plugin – this responsibility lies solely with the Client.

53