You are on page 1of 18

Copyright © 2010 EMC Corporation. Do not Copy - All Rights Reserved.

Module 4: Avamar Node Replacement
Upon completion of this module, you will be able to:

• List the requirements to replace a storage node in an
Avamar system

• Describe the Avamar node replacement process
• Perform Avamar node replacement

© 2010 EMC Corporation. All rights reserved.

Avamar Node Replacement 4 - 1

The objectives for this module are shown here. Please take a moment to read them.

Avamar Node Replacement, Module 4 - 1

Avamar Node Replacement 4 .All Rights Reserved. you can replace the node and populate the replacement node (spare node) using parity data DISCLAIMER: The node replacement procedures in this module are provisional and should not be used at a customer site.2 Avamar Node Replacement. Node Replacement – What is it? • In an Avamar multi-node RAIN system.Copyright © 2010 EMC Corporation. Module 4 .2 . if a data node fails. Do not Copy . © 2010 EMC Corporation. All rights reserved.

3 . Node Replacement – Before you Begin • Ensure that: – A spare node is available – Operating System ISO is installed on spare node (non-data store only) – Networking information is configured for spare node – Passwords on spare node reflect the passwords for the existing production Avamar system © 2010 EMC Corporation. Avamar Node Replacement 4 . Module 4 .3 Avamar Node Replacement. All rights reserved.Copyright © 2010 EMC Corporation.All Rights Reserved. Do not Copy .

Copyright © 2010 EMC Corporation.4 Avamar Node Replacement. Node Replacement . All rights reserved. Avamar Node Replacement 4 . Do not Copy .xml file on the utility node to remove the failed node and update the spare node to a storage node • Synchronize the clocks on all hosts • Suspend Avamar client activities • Suspend Avamar replication cron job • Populate the node using parity data and bring it online • Ensure new node comes online • Resume client activities • Resume Avamar replication cron job © 2010 EMC Corporation.4 .All Rights Reserved. Module 4 .Steps • Update the hosts file on all nodes to add spare node’s IP Address • Ensure that you have a validated checkpoint • Update the probe.

6) 10.3 © 2010 EMC Corporation.127.100.All Rights Reserved.127.3 0.100.2) Physical Node Logical Node Number Number 10.4) 10.4 10.2 0. Physical Node 0.100.4 Data Node 0.0 (10.5) Data Node 0.100.3 0.5 Avamar Node Replacement. All rights reserved.100.5 Data Node 0.100.0 MAC address 00:13:72:50:E6:7D 00:13:72:4E:55:B9 00:13:72:4E:56:99 00:13:72:4E:56:81 Avamar Node Replacement 4 .3 IP address 10.0 0.2 0.127.127.1 0.3 Data Node Spare (10.0 0.100.Copyright © 2010 EMC Corporation.5 0.5 .127.2 10.1 0.127.127. Do not Copy .127.2 0.127.127.2 0.100.100.2 Utility Node (10.100.3 (10.100.2 (10.3 10.100.127.100. Module 4 .100. Node Replacement Example ( 1of 3) Data Node Data Node 0.127.1 0.1 0.127.1) Output of nodenumbers command Logical node 0.1 (10.127.3) 10.0 0.

100.Copyright © 2010 EMC Corporation.127.0 (10.1) How to achieve this: 1.2 fails ( 2of 3) Data Node 0.127.xml file: Use nodedb command.100.127. Use rebuild.100. Node Replacement Example – Node 0.3) Data Node 0. Assign the same physical and logical node number to the spare node as the failed data node and rebuild the node Utility Node (10.2) Data Node Spare (10.100. Module 4 .6 .1 (10.100. promote spare node 2. Replace failed node with spare node 2. Do not Copy .100.node command with the –nodes=<physical node number of new node> --TAG=<logical node number of failed node> options © 2010 EMC Corporation.2 (10.127.4) Fails Goals: 1. Remove failed node.127.All Rights Reserved.3 (10. Update probe.4) Data Node 0.127.6 Avamar Node Replacement.2 (10.5) Data Node 0.100.6) Situation: Data node 0.127. Avamar Node Replacement 4 . All rights reserved.

127.3 (10.127.3 © 2010 EMC Corporation.127.6) Utility Node (10.1 0.100.100.7 Avamar Node Replacement. Physical Node 0.1 (10.100.6 0.127.2 0.127.100.127.2 10.2 0.1) Output of nodenumbers command Logical node 0.2 (10.3 10.3) 10.3 IP address 10.Copyright © 2010 EMC Corporation.0 0.2 0.2 10.5 MAC address 00:13:72:50:E6:7D 00:13:72:4E:55:B9 00:13:72:4E:56:6C 00:13:72:4E:56:81 Avamar Node Replacement 4 .All Rights Reserved.1 0. All rights reserved. Do not Copy .127.100.2 0.127.127.7 .5 0.100.3 Data Node 0.1 10.127.100.100.1 0.100.127.127.100. Node Replacement Example Desired Solution – (3 of 3) Data Node 0.6 < 10.127.3 0.2) 10.0 (10.100. Module 4 .100.100.0 0.0 Data Node 0.5) Data Node Physical Node Logical Node Number Number Data Node 0.3 0.0 0.

Avamar Node Replacement 4 . Do not Copy .8 Avamar Node Replacement. Module 4 .Copyright © 2010 EMC Corporation. All rights reserved. Update Hosts File on All Nodes • Update the hosts file on all nodes to add spare node’s IP Address Comment out the entry for the failed node Add a new line for the replacement node (spare) © 2010 EMC Corporation.All Rights Reserved.8 .

All rights reserved.9 . Avamar Node Replacement 4 . Module 4 .Copyright © 2010 EMC Corporation. Ensure You Have a Validated Checkpoint • It is very important to have a validated checkpoint you can roll back to if required • Checkpoints shown in Avamar Administrator .Server window Validated checkpoint © 2010 EMC Corporation.All Rights Reserved.9 Avamar Node Replacement. Do not Copy .

1 --type=storage © 2010 EMC Corporation.spare.xml File • To maintain the same physical node number for the replacement node as the failed node replace the failed node with the spare node using the nodedb command nodedb delete node --node=0.Copyright © 2010 EMC Corporation. Module 4 .10 . All rights reserved.0 nodedb add node --addr=192. Update the probe. Avamar Node Replacement 4 .All Rights Reserved.100.7 --replace --node=0.168. Do not Copy .10 Avamar Node Replacement.

Synchronize Clocks Across All Nodes • • • Avamar requires that clocks on all nodes are synchronized The Avamar utility to perform time synchronization is asktime asktime – must be run as user dpn with ssh keys loaded – Must be run on the utility node Running asktime on the utility node as user dpn Desired output © 2010 EMC Corporation. Avamar Node Replacement 4 . Do not Copy . Module 4 .11 Avamar Node Replacement. All rights reserved.All Rights Reserved.Copyright © 2010 EMC Corporation.11 .

Do not Copy .All Rights Reserved. Module 4 .Copyright © 2010 EMC Corporation. you must suspend all client activities on the Avamar system – Utility to perform this task: avmaint – Syntax: avmaint suspend • avmaint suspend halts existing client activities and denies any new client requests • Run this command from the utility node – Must be run by user admin with ssh keys loaded © 2010 EMC Corporation.12 . All rights reserved. Suspend Avamar Client Activities • Before re-building a replacement node. Avamar Node Replacement 4 .12 Avamar Node Replacement.

All rights reserved. Suspend Replication Cron Job • Before re-building a replacement node.13 Avamar Node Replacement. Module 4 .Copyright © 2010 EMC Corporation. Avamar Node Replacement 4 .13 .All Rights Reserved. Do not Copy . you must suspend the Avamar replication cron job – Script to perform this task: suspend_crons • suspend_crons suspends activity of any of the Avamar replication cron job: repl_cron • Run this command from the utility node – Must be run by user admin with ssh keys loaded © 2010 EMC Corporation.

All rights reserved. Do not Copy .node script • rebuild.Copyright © 2010 EMC Corporation. Avamar Node Replacement 4 .14 Avamar Node Replacement.All Rights Reserved. Module 4 . Populate the Node Using Parity Data and Bring It Online • To populate the replacement node using parity data and bring it online. use the rebuild. allows you to assign the same logical node number to the replacement node as the failed node © 2010 EMC Corporation. the replacement node is assigned the logical node number specified Note: The --tag option.node – Used to rebuild and restart a node – Must be run as admin – The --nodes=<physical node number of node to be rebuilt> must be specified – If the --clean option is specified. all existing data on the node is deleted – If the --tag=<logical node number to assign to this node> is specified.14 .

Avamar Node Replacement 4 . All rights reserved. the node status is refreshed every 5 seconds © 2010 EMC Corporation.  For example.All Rights Reserved. using the status.15 .15 Avamar Node Replacement. status.Copyright © 2010 EMC Corporation. Do not Copy . Ensure that the New Node is Online • • Check node status.dpn – script used to return status of Avamar nodes – Interval can be specified (optional). Module 4 .dpn command status.dpn 5.

All Rights Reserved. type: – Script to perform this task: resume_crons © 2010 EMC Corporation. Do not Copy .16 . Module 4 . Avamar Node Replacement 4 . All rights reserved.Copyright © 2010 EMC Corporation.16 Avamar Node Replacement. Resume Avamar Client Activities and Replication Cron Job • Resume Avamar client activities: – Utility to perform this task: avmaint – Syntax: avmaint resume • Resume Avamar replication cron job.

17 .Copyright © 2010 EMC Corporation.All Rights Reserved. Lab 4-1: In this lab you will: • Fail a node • Perform recovery when the Avamar server is running in degraded mode • Perform node replacement Avamar Node Replacement. Do not Copy .17 © 2010 EMC Corporation. Exploration Lab 4-1: Node Replacement Note: • Avamar identifies nodes with physical and logical node numbers • Therefore. All rights reserved. it is important to ensure that the replacement node has the same physical and logical node numbers as the failed data node Avamar Node Replacement 4 . Module 4 .

Avamar Node Replacement 4 .All Rights Reserved. Module Summary Key points covered in this module: • Requirements to replace a storage node in an Avamar system • Avamar Node Replacement process © 2010 EMC Corporation.Copyright © 2010 EMC Corporation.18 . Do not Copy . Please take a moment to review them. All rights reserved. Avamar Node Replacement.18 These are the key points covered in Module 4. Module 4 .