<Insert Picture Here>

Administering ASM DiskGroups with Exadata CellGrid
Disks.
Author: Esteban Bernal.

© 2008 Oracle Corporation – Proprietary and Confidential

Agenda

• Administering ASM DiskGroups Using Exadata
Cell.

• Administering Grid Disks with ASM.

© 2008 Oracle Corporation – Proprietary and Confidential – 2–

Administering ASM DiskGroups with
Exadata CellGrid Disks.
Introduction
Automatic Storage Management (ASM)
ASM is the cluster volume manager and
file system used to manage Exadata Cell resources. ASM provides
enhanced storage management by:

Striping database files evenly across all available storage cells and disks
for optimal performance.
Using mirroring and failuregroups to avoid any single point of failure.
Enabling dynamic add and drop capability for nonintrusive cell and disk
allocation, deallocation, and reallocation.
Enabling multiple databases to share storage cells and disks.

© 2008 Oracle Corporation – Proprietary and Confidential – 3–

© 2008 Oracle Corporation – Proprietary and Confidential – 4– . Exadata Cell exposes grid disks to ASM as individual disks available for membership in ASM disk groups. Whenever possible. the diskgroups must contain only Exadata Cell grid disks. Administering ASM DiskGroups with Exadata CellGrid Disks. To take advantage of Exadata Cell features. such as predicate processing offload. ASM DiskGroup is the primary storage abstraction within ASM and is composed of one or more disks. grid disk names should correspond closely with ASM diskgroup names to assist in problem diagnosis between ASM and Exadata Cell. and the tables must be fully inside these diskgroups.

Failure groups for Exadata Cell grid disks are set by default so that the disks on a single cell are in the same failure group. Because of this scenario. making correct failure group configuration simple for Exadata Cell. For Exadata Cell. Administering ASM DiskGroups with Exadata CellGrid Disks. © 2008 Oracle Corporation – Proprietary and Confidential – 5– . all griddisks (ASM disk group members and candidates) can effectively fail together if the storage cell fails. ASM FailGroup is a subset of disks in an ASM diskgroup that can fail together because they share the same hardware and guarantees data redundancy. all ASM disks (griddisks) sourced from a given storage cell should be assigned to a single failuregroup representing the cell.

Administering ASM DiskGroups with Exadata CellGrid Disks. © 2008 Oracle Corporation – Proprietary and Confidential – 6– . For example. then all grid disks on storage cell A are designated as one failure group. if all grid disks from two storage cells (A and B) are added to a single ASM diskgroup with normal redundancy. and all grid disks on storage cell B are designated as another failure group. This enables Exadata Cell and ASM to tolerate the failure of either storage cell.

Administering ASM DiskGroups Using Exadata Cell. © 2008 Oracle Corporation – Proprietary and Confidential – 7– .  Determine the GridDisks.  The default failure group name is the cell name (recommended). Note: If a cell is renamed. To ensure all the disks from the cell are in one failure group. Administering ASM DiskGroups with Exadata CellGrid Disks. a) Creating an Exadata Cell ASM DiskGroup:  Using the CREATE DISKGROUP SQL command. and a disk from that cell is added to an existing diskgroup that has disks from that cell.  Choose similar names for the ASM diskgroup and its griddisks. add the disk to the diskgroup and specify the original failure group name.  The default ASM disk name is the grid disk name (recommended). then ASM will add the new disk to a failure group using the new cell name. 1.

168.130:5042/data_CD_disk08_cell2 CANDIDATE o/192.168.129:5042/data_CD_disk07_cell1 CANDIDATE o/192. o/192.130:5042/data_CD_disk10_cell2 CANDIDATE .22.168.22. header_status STATUS FROM V$ASM_DISK WHERE path LIKE 'o/%'.22.------------ o/192.168. . Administering ASM DiskGroups with Exadata CellGrid Disks. 1.130:5042/data_CD_disk11_cell2 CANDIDATE o/192.22.129:5042/data_CD_disk01_cell1 CANDIDATE o/192.168. . Determine which Exadata Cell grid disks are available by querying the V$ASM_DISK view on the ASM instance: SQL> SELECT PATH.168.22. PATH STATUS -----------------------------------------------------------.129:5042/data_CD_disk09_cell1 CANDIDATE . © 2008 Oracle Corporation – Proprietary and Confidential – 8– .22. 24 rows selected.

rdbms'='11. © 2008 Oracle Corporation – Proprietary and Confidential – 9– .These grid disks are on cell2 FAILGROUP cell02 DISK 'o/*/data_CD_disk*_cell2‘ -. Use the SQL CREATE DISKGROUP command to create the ASM diskgroup. 2.smart_scan_capable'='TRUE'.0.0'.Note that this diskgroup is set for cell only ATTRIBUTE 'compatible.These disk group attributes must be set for cell access -.These grid disks are on cell1 FAILGROUP cell01 DISK 'o/*/data_CD_disk*_cell1‘ -.7.1. The next diskgroup attributes must be set: SQL> CREATE DISKGROUP data NORMAL REDUNDANCY -.1.7.asm'='11. 'compatible. 'cell. Administering ASM DiskGroups with Exadata CellGrid Disks.0'.0. 'au_size'='4M'.

View the ASM disk groups and associated attributes with the next SQL query on V$ASM dynamic views: SQL> SELECT dg. DISKGROUP NAME VALUE -----------------------------.1.0 © 2008 Oracle Corporation – Proprietary and Confidential – 10 – .0.7.6h DATA au_size 4194304 DATA cell.------------------ DATA disk_repair_time 3.value. SUBSTR(a. SUBSTR(a.1.group_number.group_number = a.24) AS name.name.1.1.24) AS value FROM V$ASM_DISKGROUP dg. Administering ASM DiskGroups with Exadata CellGrid Disks.smart_scan_capable TRUE DATA compatible.rdbms 11. 3.asm 11.name AS diskgroup.0. V$ASM_ATTRIBUTE a WHERE dg.-----------------------.0 DATA compatible.7.

TABLESPACE_NAME PREDICATE_EVALUATION -----------------------------. The PREDICATE_EVALUATION column of the DBA_TABLESPACES view indicates whether predicates are evaluated by host (HOST) or by storage (STORAGE): SQL> SELECT tablespace_name. predicate_evaluation FROM dba_tablespaces WHERE tablespace_name = 'SALES'.--------------------- SALES STORAGE © 2008 Oracle Corporation – Proprietary and Confidential – 11 – . Create a tablespace in the disk group to take advantage of Exadata Cell features. Verify that the tablespace is in an Exadata Cell disk group. This tablespace should contain the tables that you want to query with offload processing: SQL> CREATE TABLESPACE SALES DATAFILE '+DATA'. 4. such as offload processing. 5. Administering ASM DiskGroups with Exadata CellGrid Disks.

If the header status is set to CANDIDATE that means the disk is a candidate for a diskgroup: SQL> SELECT PATH. b) Adding a Disk to an ASM DiskGroup: 1.129:5042/data_CD_disk13_cell1 CANDIDATE o/192. Note: Do not add Exadata Cell grid disks to a non-Exadata Cell ASM diskgroup unless you are planning to migrate the diskgroup to an Exadata Cell diskgroup. Determine which Exadata Cell grid disks are available by querying the V$ASM_DISK view on the ASM instance. Administering ASM DiskGroups with Exadata CellGrid Disks.130:5042/data_CD_disk13_cell2 CANDIDATE 2 rows selected. © 2008 Oracle Corporation – Proprietary and Confidential – 12 – .22. PATH STATUS -----------------------------------------------------------.------------ o/192.168.22. header_status STATUS FROM V$ASM_DISK WHERE path LIKE 'o/%' and STATUS like 'CANDIDATE' .168.

You can query the V$ASM_OPERATION view for the status of the rebalance operation. ASM will rebalance the diskgroup. SQL> SELECT * FROM V$ASM_OPERATION. When the disk is added. © 2008 Oracle Corporation – Proprietary and Confidential – 13 – . 'o/*/data_CD_disk13_cell2'. Administering ASM DiskGroups with Exadata CellGrid Disks. Use the SQL ALTER DISKGROUP command with the ADD DISK clause to add the disk to the ASM diskgroup: SQL> ALTER DISKGROUP DATA ADD DISK 'o/*/data_CD_disk13_cell1'. 2.

which results in a loss of full redundancy. SQL> ALTER DISKGROUP DATA MOUNT FORCE. Administering ASM DiskGroups with Exadata CellGrid Disks. SQL> ALTER DISKGROUP DATA MOUNT. © 2008 Oracle Corporation – Proprietary and Confidential – 14 – . c) Mounting or Dismounting an ASM DiskGroup: To mount or dismount a diskgroup.. use the SQL ALTER DISKGROUP statement with the MOUNT or DISMOUNT option: SQL> ALTER DISKGROUP DATA DISMOUNT. Note: You can use the FORCE option of the ALTER DISKGROUP statement MOUNT clause to mount diskgroups if their components are unavailable.

Use the SQL ALTER DISKGROUP statement with the OFFLINE DISK or ONLINE DISK clause to change the disk in the ASM disk group to offline or online: SQL> ALTER DISKGROUP DATA OFFLINE DISK DATA_CD_DISK16_CELL1. failgroup from v$asm_disk order by 3. d) Changing a Disk to Offline or Online: 1. © 2008 Oracle Corporation – Proprietary and Confidential – 15 – . path. 2. SQL> ALTER DISKGROUP DATA ONLINE DISK DATA_CD_DISK16_CELL1. Administering ASM DiskGroups with Exadata CellGrid Disks. Determine which disk you want offline or online in the ASM disk group by querying the V$ASM_DISK and V$ASM_DISKGROUP views on the ASM instance: SQL> select name.

Note: As soon as the disk is online. Administering ASM DiskGroups with Exadata CellGrid Disks. © 2008 Oracle Corporation – Proprietary and Confidential – 16 – . You can query the V$ASM_OPERATION view for the status of the rebalance operation. ASM will rebalance the disk group. SQL> SELECT * FROM V$ASM_OPERATION.

e) Dropping a Disk from an ASM DiskGroup: 1. Determine which disks you want to drop from the ASM diskgroup by querying the V$ASM_DISK and V$ASM_DISKGROUP views on the ASM instance: SQL> SELECT d. V$ASM_DISKGROUP dg WHERE dg.name as asmdisk. dg.name LIKE 'DATA%' AND d.name as diskgroup FROM V$ASM_DISK d.------------------------------ DATA_CD_DISK12_CELL2 DATA DATA_CD_DISK16_CELL1 DATA DATA_CD_DISK04_CELL2 DATA DATA_CD_DISK07_CELL2 DATA © 2008 Oracle Corporation – Proprietary and Confidential – 17 – .group_number. Administering ASM DiskGroups with Exadata CellGrid Disks. ASMDISK DISKGROUP -----------------------------.group_number = dg.

ensure that you identify the grid disks that are mapped to each ASM diskgroup. You can query the V$ASM_OPERATION view for the status of the rebalance operation. Use the SQL ALTER DISKGROUP statement with the DROP DISK clause to drop the disks from the ASM diskgroup: SQL> ALTER DISKGROUP DATA DROP DISK DATA_CD_DISK16_CELL1. Note: If you are removing an Exadata Cell grid disk. ASM will rebalance the diskgroup. SQL> SELECT * FROM V$ASM_OPERATION. Administering ASM DiskGroups with Exadata CellGrid Disks. Note: When the disk is dropped from the ASM diskgroup. 2. © 2008 Oracle Corporation – Proprietary and Confidential – 18 – .

© 2008 Oracle Corporation – Proprietary and Confidential – 19 – . Note: After an Exadata Cell grid disk is dropped from the ASM diskgroup. you can drop the grid disk from a cell. Administering ASM DiskGroups with Exadata CellGrid Disks.

Use the SQL DROP DISKGROUP statement to drop the ASM disk group. state from v$asm_diskgroup. © 2008 Oracle Corporation – Proprietary and Confidential – 20 – . Administering ASM DiskGroups with Exadata CellGrid Disks. Determine the disk group that you want to drop by querying the V$ASM_DISKGROUP view on the ASM instance: SQL> select name. f) Dropping an ASM DiskGroup: 1.----------- DATA MOUNTED 2. NAME STATE -----------------------------. SQL> DROP DISKGROUP DATA.

you can use the FORCE option of the DROP DISKGROUP statement: SQL> DROP DISKGROUP DATA FORCE. Administering ASM DiskGroups with Exadata CellGrid Disks. © 2008 Oracle Corporation – Proprietary and Confidential – 21 – . Note: If you cannot mount a diskgroup but must drop it.

smart_scan_capable to true. Administering ASM DiskGroups with Exadata CellGrid Disks.1. true To optimize the operation.  Requires ASM to be upgraded to release 11.7. then you can migrate the entire diskgroup with no downtime by adding the Oracle Exadata Storage to the diskgroup.0. removing the legacy storage from the diskgroup. and setting the diskgroup attribute cell. remove and add storage as one command in the ASM instance. g) Migrating an Existing ASM DiskGroup to an Exadata Cell DiskGroup.  Necessity to use the rebalance POWER setting to control the effect on applications. The advantages and disadvantages for this strategy are as follows:  Zero downtime. © 2008 Oracle Corporation – Proprietary and Confidential – 22 – . If ASM normal or high redundancy is used for the legacy storage.

1. Administering ASM DiskGroups with Exadata CellGrid Disks.----------- DATA MOUNTED © 2008 Oracle Corporation – Proprietary and Confidential – 23 – . state from v$asm_diskgroup. SQL> select name. To migrate an existing ASM diskgroup to an Exadata Cell-only diskgroup. Determine the diskgroup that you want to migrate by querying the V$ASM_DISKGROUP view on the ASM instance. perform the following steps. NAME STATE -----------------------------. The ASM diskgroup must be mounted to complete this procedure.

2.130:5042/data_CD_disk02_cell4 CANDIDATE o/192.168. SQL> SELECT PATH. Administering ASM DiskGroups with Exadata CellGrid Disks.168.130:5042/data_CD_disk03_cell4 CANDIDATE 2 rows selected.22.130:5042/data_CD_disk01_cell4 CANDIDATE o/192. header_status STATUS FROM V$ASM_DISK WHERE path LIKE 'o/%' and STATUS like 'CANDIDATE' . PATH STATUS -----------------------------------------------------------.168.129:5042/data_CD_disk01_cell3 CANDIDATE o/192.168.168.129:5042/data_CD_disk02_cell3 CANDIDATE o/192.22.168. Determine which Exadata Cell grid disks are available by querying the V$ASM_DISK view on the ASM instance. © 2008 Oracle Corporation – Proprietary and Confidential – 24 – .22.22.22.22.129:5042/data_CD_disk03_cell3 CANDIDATE o/192.------------ o/192.

smart_scan_capable' = 'FALSE'  'compatible. the cell.1.smart_scan_capable attribute must be set to FALSE.7.asm' = '11.0' SQL> ALTER DISKGROUP DATA SET ATTRIBUTE 'cell.1.asm' 'compatible.1.rdbms' = '11.rdbms' = '11.asm' = '11.smart_scan_capable' = 'FALSE' SQL> ALTER DISKGROUP DATA SET ATTRIBUTE 'compatible.rdbms' 'compatible.1. Administering ASM DiskGroups with Exadata CellGrid Disks.0.0' Note: When you alter the diskgroup to add Exadata Cell grid disks.0' SQL> ALTER DISKGROUP DATA SET ATTRIBUTE 'compatible.7.0.7.0. Use the SQL ALTER DISKGROUP statement to set the following for the existing diskgroup:  'cell. FALSE © 2008 Oracle Corporation – Proprietary and Confidential – 25 – . 3.0'  'compatible.0.7.

168.129:5042/data_CD_disk02_cell8'. 'o/192.22.22.22. 'o/192. 'o/192. 'o/192.22.168.130:5042/data_CD_disk03_cell9'.168.168.168. Use the SQL ALTER DISKGROUP statement with the ADD DISK clause to add all the Exadata Cell grid disks to the ASM diskgroup: SQL> ALTER DISKGROUP DATA ADD DISK 'o/*/data_CD_disk13_cell8'.22.130:5042/data_CD_disk03_cell9' Note: Allow time for ASM to rebalance the diskgroup.22.168. 4.130:5042/data_CD_disk02_cell9'. © 2008 Oracle Corporation – Proprietary and Confidential – 26 – .129:5042/data_CD_disk01_cell8'. 'o/*/data_CD_disk13_cell9' SQL> ALTER DISKGROUP DATA ADD DISK 'o/192.130:5042/data_CD_disk01_cell9'. 'o/*/data_CD_disk13_cell8' 'o/*/data_CD_disk13_cell9'.168. Administering ASM DiskGroups with Exadata CellGrid Disks.129:5042/data_CD_disk03_cell8'. 'o/192. 'o/192.22.

use the SQL ALTER DISKGROUP statement to set the cell. Administering ASM DiskGroups with Exadata CellGrid Disks.DATA5. Note: Allow time for ASM to rebalance the diskgroup.smart_scan_capable' = ‘TRUE TRUE‘ © 2008 Oracle Corporation – Proprietary and Confidential – 27 – .DATA3.DATA6 REBALANCE POWER 11. 6. 5. Use the SQL ALTER DISKGROUP statement with the DROP DISK clause to drop all the non-Exadata Cell disks from the ASM diskgroup: SQL> ALTER DISKGROUP DATA DROP DISK DATA1. After the disk drop operation is completed.DATA2.DATA4.smart_scan_capable disk group attribute to be TRUE: TRUE SQL> ALTER DISKGROUP DATA SET ATTRIBUTE 'cell.

a subset of the grid disk name should match all or part of the name of the ASM disk group to which the grid disk will belong. Administering ASM DiskGroups with Exadata CellGrid Disks. © 2008 Oracle Corporation – Proprietary and Confidential – 28 – . To also help determine which grid disks belong to an ASM diskgroup. then the grid disk name should contain data_CD_disk01_cell1 and data. if a grid disk is created on the cell disk CD_disk01_cell1 and that grid disk will belong to an ASM diskgroup named DATA. For example. a) Naming Conventions for Exadata Cell Grid Disks: The name of the grid disk should contain the cell disk name to make it easy to determine which grid disks belong to a cell disk. The grid disk name should be data_CD_disk01_cell1. Administering Grid Disks with ASM. 2.

Administering ASM DiskGroups with Exadata CellGrid Disks.129:5042/data_CD_disk01_cell1 © 2008 Oracle Corporation – Proprietary and Confidential – 29 – . PATH ---------------------------------------------------------------- o/192. CellCLI> list celldisk CD_disk01_cell1 normal CellCLI> list griddisk CellCLI> list celldisk data_CD_disk01_cell1 active CD_disk01_cell1 normal SQL> select path from v$asm_disk order by 1.168.22.

Administering ASM DiskGroups with Exadata CellGrid Disks. If you do not use the default generated name when creating grid disks. data_CD_disk10_cell1 active data_CD_disk11_cell1 active data_CD_disk12_cell1 active © 2008 Oracle Corporation – Proprietary and Confidential – 30 – . . it might not be possible to add the grid disk to an ASM diskgroup. CellCLI> create griddisk all prefix=data CellCLI> list griddisk data_CD_disk01_cell1 active data_CD_disk02_cell1 active data_CD_disk03_cell1 active . If the disk name is not unique. a unique grid disk name is automatically generated that includes the prefix and cell name. you must ensure that the grid disk name is unique across all cells. When you use the ALL PREFIX option with CREATE GRIDDISK.

Administering ASM DiskGroups with Exadata CellGrid Disks. First. you must be aware of how that might affect the ASM diskgroup to which the grid disk belongs. b) Changing a Grid Disk that Belongs to an ASM DiskGroup: When you change a grid disk that belongs to an ASM diskgroup. • Name To change attributes of a grid disk. use the DESCRIBE GRIDDISK command to determine which grid disk attributes can be modified. use the Changing a Grid Disk Name: CellCLI ALTER GRIDDISK command. CellCLI> describe griddisk name modifiable availableTo modifiable cellDisk comment modifiable creationTime errorCount id offset size modifiable status © 2008 Oracle Corporation – Proprietary and Confidential – 31 – .

ensure that the corresponding ASM disk is offline. CellCLI> ALTER GRIDDISK data11 name='data_CD_3_cell04_11‘ Caution: Before changing the name of a grid disk that belongs to an ASM diskgroup. © 2008 Oracle Corporation – Proprietary and Confidential – 32 – . Administering ASM DiskGroups with Exadata CellGrid Disks.

ASM will discover all active disks. © 2008 Oracle Corporation – Proprietary and Confidential – 33 – . then ASM takes the corresponding ASM disk offline when I/Os to the disk fail. Inactive To activate or deactivate a grid disk. Making a Grid Disk Active or Inactive: use the CellCLI ALTER GRIDDISK command. then the grid disk should be changed to inactive. A grid disk should be made inactive before dropping the grid disk to ensure that the grid disk is not in use. 2. To make the disk usable again. Administering ASM DiskGroups with Exadata CellGrid Disks. However. or from inactive to active. Note 2: There is usually no need to change the grid disk state from active to inactive. first make the grid disk active in the cell and then bring the corresponding ASM disk back online in ASM. CellCLI> ALTER GRIDDISK data0_CD_3_cell04_31 ACTIVE CellCLI> ALTER GRIDDISK data0_CD_3_cell04_31 INACTIVE Note 1: When a grid disk in use by a database client is made inactive. if a grid disk should not be available to ASM.

and a grid disk in that group is made inactive. Note 3: If a diskgroup is created with external redundancy. © 2008 Oracle Corporation – Proprietary and Confidential – 34 – . Administering ASM DiskGroups with Exadata CellGrid Disks. then the diskgroup cannot be mounted.

CellCLI> ALTER GRIDDISK data_CD_16_cell01 INACTIVE CellCLI> ALTER GRIDDISK ALL PREFIX=data INACTIVE © 2008 Oracle Corporation – Proprietary and Confidential – 35 – . use the CellCLI DROP GRIDDISK command. Administering ASM DiskGroups with Exadata CellGrid Disks. Make the grid disk inactive. i. 3. Make the grid disk inactive before dropping the grid disk to ensure that the grid disk is not in use. The FORCE option can be used to force the grid disk that is in use to be dropped.  Caution 2: Before dropping a grid disk using the FORCE option. ensure that the ASM disk was dropped from the diskgroup. Dropping a Grid Disk:Disk To drop a grid disk.  Caution 1: Before dropping a grid disk that belongs to an ASM diskgroup. ensure that the corresponding ASM disk was dropped from the diskgroup.

SQL> ALTER DISKGROUP DATA DROP DISK data_CD_16_cell01. Drop the corresponding ASM disk from the diskgroup. data_CD_15_cell01. ii. SQL> ALTER DISKGROUP DATA DROP DISK data_CD_01_cell01. iii. Drop the grid disk. . data_CD_16_cell01. CellCLI> DROP GRIDDISK data_CD_16_cell01 CellCLI> DROP GRIDDISK ALL PREFIX=data CellCLI> DROP GRIDDISK data_CD_16_cell01 FORCE © 2008 Oracle Corporation – Proprietary and Confidential – 36 – . . Administering ASM DiskGroups with Exadata CellGrid Disks.

dg. You can also use SQL*Plus on the ASM instance to find the ASM diskgroup that matches part of the specific grid disk name. V$ASM_DISKGROUP dg WHERE dg. This can help you to determine which ASM diskgroup contains a specific grid disk. then you can determine the ASM diskgroup to which the grid disk belongs.name LIKE 'DATA%' AND d.name as asmdisk. ASMDISK DISKGROUP -----------------------------.name as diskgroup FROM V$ASM_DISK d.group_number. SQL> SELECT d. © 2008 Oracle Corporation – Proprietary and Confidential – 37 – . c) Determining Which ASM DiskGroup Contains a Grid Disk: If a grid disk name matches the ASM disk name and the name contains the ASM diskgroup name.group_number = dg. Administering ASM DiskGroups with Exadata CellGrid Disks.------------------------- DATA_CD_DISK12_CELL2 DATA DATA_CD_DISK04_CELL2 DATA DATA_CD_DISK07_CELL2 DATA . .

. © 2008 Oracle Corporation – Proprietary and Confidential – 38 – . SQL> SELECT name FROM V$ASM_DISKGROUP.*data. NAME ------------------------------ FRA DATA CellCLI> LIST GRIDDISK WHERE name LIKE '. then you can use SQL*Plus on the ASM instance to list the ASM diskgroup names and use the CellCLI utility to search for specific grid disk names. d) Determining Which Grid Disks Belong to an ASM Disk Group: If a grid disk name contains the ASM diskgroup name. Administering ASM DiskGroups with Exadata CellGrid Disks.*' data_CD_disk01_cell1 active data_CD_disk02_cell1 active data_CD_disk03_cell1 active data_CD_disk04_cell1 active .

© 2008 Oracle Corporation – Proprietary and Confidential – 39 – .