You are on page 1of 43

Exadata: from Beginner to Advanced in 3 Hours

Arup Nanda Longtime Oracle DBA (and now DMA)

Why this Session?
• If you are
– an Oracle DBA
• Familiar with RAC, 11gR2 and ASM

– about to be a Database Machine Administrator (DMA)

• How much do you have to learn?
• • • • How much of you own prior knowledge I can apply? What’s different in Exadata? What makes it special, fast, efficient? Do you have to go through a lot of training?
Exadata: from Beginner to Advanced in 3 Hours 2

What is Exadata
• It is like an appliance containing
– Storage, Flash Disks, Database Servers, Infiniband Switches, Ethernet Switches, KVM (some models)

• But it is not an appliance. Why?
– additional software to make it a better database machine – Components can be managed independently

• That’s why Oracle calls it a Database Machine (DBM) • And DMA – Database Machine Administrator
Exadata: from Beginner to Advanced in 3 Hours 3

Anatomy of an Oracle Database
Instance Combination of •Memory Areas •Background Processes

SELECT NAME FROM CUSTOMERS WHERE STATUS ='ANGRY' UPDATE CUSTOMERS SET BONUS = 1M WHERE STATUS ='ANGRY'

datafile1 datafile2
Exadata: from Beginner to Advanced in 3 Hours

Storage

4

RAC Database datafile1 datafile2 Exadata: from Beginner to Advanced in 3 Hours Storage 5 .

Query Processing JILL SELECT NAME FROM CUSTOMERS WHERE STATUS ='ANGRY' datafile1 datafile2 Exadata: from Beginner to Advanced in 3 Hours Database Block Storage 6 .

Components for Performance CPU Memory Network I/O Controller Disk Exadata: from Beginner to Advanced in 3 Hours 7 Less I/O = better performance .

not data • Most database systems – are way bigger than caches – need to get the data to the memory to process --> I/O at the disk level is still high • Caches are excellent for filesystems .What about SAN Caches? • Success of SAN caches is built upon predictive analytics • They work well. if a small percentage of disk is accessed most often – The emphasis is on disk. or very small databases Exadata: from Beginner to Advanced in 3 Hours 8 .

What about In-Memory DBs • Memory is still more expensive • How much memory is enough? • You have a 100 MB database and 100 MB buffer cache • The whole database will fit in the memory.blogspot.html Exadata: from Beginner to Advanced in 3 Hours 9 .com/2011/04/can-i-fit-80mb-database-completely-in. right? • NO! • Oracle database fills up to 7x DB size buffer cache http://arup.

The Solution • A typical query may: – Select 10% of the entire storage – Use only 1% of the data it gets • To gain performance. The storage must be cognizant of the data CPU Memory SELECT NAME FROM CUSTOMERS WHERE STATUS ='ANGRY' Filtering should be Applied Here Exadata: from Beginner to Advanced in 3 Hours Network I/O Controller Disk 10 . Filtering at the storage level . the DB needs to shed weight • It has to get less from the storage .

The Magic #1 CPU Memory Network I/O Controller Disk Exadata: from Beginner to Advanced in 3 Hours iDB The communication between CPU and Disk carries the information on the query – columns and predicates. 11 . This occurs as a result of a special protocol called iDB.

Magic #2 Storage Cell Server iDB • • • Cells are Sun Blades Run Oracle Enterprise Linux Software called Exadata Storage Server (ESS) which understands iDB Disk1 Disk2 Disk3 Exadata: from Beginner to Advanced in 3 Hours 12 .

SELECT … FROM TABLE WHERE COL1 = 1 Disk1 MIN = 4 MAX = 5 Disk2 MIN = 1 MAX = 2 Disk3 MIN = 3 MAX = 5 MIN = 3 MAX = 5 Storage Index Disk4 Exadata: from Beginner to Advanced in 3 Hours 13 .Magic #3 Storage Indexes Storage Indexes store in memory of the Cell Server the areas on the disk and the MIN/MAX value of the column and whether NULL exists. They eliminate disk I/O.

value/1024/1024 as stat_value from v$mystat s.name in ( 'cell physical IO bytes saved by storage index'. v$statname n where s.---------SI Savings 5120.statistic# = n. 'cell physical IO interconnect bytes returned by smart scan’) Output STAT_NAME STAT_VALUE ---------.00 Exadata: from Beginner to Advanced in 3 Hours 14 .45 Smart Scan 1034.statistic# and n.Checking Storage Index Use select name.

decode(io_cell_offload_eligible_bytes. buffer_gets/decode(nvl(executions.0.100*(io_cell_offload_eligible_bytesio_interconnect_bytes) /decode(io_cell_offload_eligible_bytes.0).0.1. executions)) avg_elapsed_time_in_secs. (elapsed_time/1000000)/decode(nvl(executions.0.0. decode(io_cell_offload_eligible_bytes.1.io_cell_offload_eligible_bytes)) "%age IO Saved".0).0).0.0. px_servers_executions/decode(nvl(executions.0.executions) avg_lio from v$sql where sql_text like <SQL Statement Comes Here>%' Exadata: from Beginner to Advanced in 3 Hours 15 .executions) avg_par_deg.0. child_number child#.1.1. executions execs.executions)/ decode(px_servers_executions.'Yes') Offloaded.px_servers_executions/decode(nvl(executions.1.0).1.Checking Offloading of an SQL select sql_id.0. plan_hash_value plan_hash.'No'.

smart_scan_capable set to FALSE. – Not on clustered tables. – – – – • The diskgroup attribute cell. _kcfis_storageidx_disabled = • Other Reasons – Cell is not offload capable true.Why Not? • Pre-requisite for Smart Scan Direct Path Full Table or Full Index Scan > 0 Predicates Disabling Smart Scans cell_offload_processing = Simple Comparison Operators false. IOTs. Exadata: from Beginner to Advanced in 3 Hours 16 . etc.

but Oracle controls what goes on there and how long it stays. They are similar to SAN cache.Magic #4 Flash Cache These are flash cards presented as disks. not memory to the Storage Cells. datafile1 datafile2 Storage Exadata: from Beginner to Advanced in 3 Hours 17 .

Magic #5 Process Offloading • Bloom Filters • Functions Offloading – Get the functions that can be offloaded • V$SQLFN_METADATA • Decompression – (Compression handled by Compute Nodes) • Virtual Columns Exadata: from Beginner to Advanced in 3 Hours 18 .

Flash Exadata: from Beginner to Advanced in 3 Hours 19 . OEL) Oracle 11gR2 RAC InfiniBand Switch Storage Cell Exadata Storage Server Disks.Components CPU Memory Network I/O Controller Disk Database Node (Sun Blade.

Network Switch Cell 1 Cell 1 Cell 1 Cell 1 Cell 14 Exadata: from Beginner to Advanced in 3 Hours 20 .Put Together: One Full Rack Database Database Node 1 Database Node 1 Database Node 1 Node 1 InfiniBand Switch RAC Cluster Database Node 8 Clients connect to the database nodes.

HC Number of InfiniBand Switches 2 24 X2-2 Half 4 48 X2-2 Full 8 96 X2-8 Full 2 160 196 GB 3 36 21.8 TB 4 TB 14 168 100.6 TB 384 GB 7 84 50.How it Looks X2-2 Qtr Number of Compute Nodes Total Compute Node Processor Cores Total Compute Node Memory Number of Storage Servers Number of SAS Disks in Storage Storage Capacity .8 TB 108 TB 2 252 TB 3 504 TB 3 504 TB 3 Source: upcoming book Exadata Recipes by Clarke from Apress Exadata: from Beginner to Advanced in 3 Hours 21 .HP Storage Capacity .4 TB 768 GB 14 168 100.

Disk Layout Compute Nodes • • Storage Cell • • • Disks (hard and flash) are connected to the cells. The disks are partitioned at the cell Some partitions are presented as filesystems The rest are used for ASM diskgroups All these disks/partitions are presented to the compute nodes 22 Exadata: from Beginner to Advanced in 3 Hours .

Disk Presentation Node filesystem filesystem Cell Exadata: from Beginner to Advanced in 3 Hours 23 .

Linux Commands – vmstat. CellCLI – command line tool to manage the Cell 5-part Linux Commands article series http://bit. SRVCTL. etc. etc. mpstat.ly/k4mKQS 4-part Exadata Command Reference article series http://bit. fdisk. alter database. ASM Commands – SQL*Plus. Clusterware Commands – CRSCTL. etc.Command Components Compute Nodes Linux Commands – vmstat. ASMCMD. fdisk. mpstat. ASMCA Database Commands – startup. etc.ly/lljFl0 24 Storage Cell Exadata: from Beginner to Advanced in 3 Hours .

Administration Skills Skill System Administrator Storage Administrator Network Administrator Database Administrator Cell Administration Needed 15% 0% 5% 60% 20% DBA Sys Admin Network Admin Exadata: from Beginner to Advanced in 3 Hours Cell Admin 25 .

One Cluster? One Cluster QA1 QA1 QA1 QA2 QA2 QA2 QA3 Prod4 QA3 Prod1 Prod1 Prod1 Prod2 Prod2 Prod2 Prod3 Prod3 Prod3 Dev1 Dev1 Dev1 Int1 Int1 Int1 Dev2 Exadata: from Beginner to Advanced in 3 Hours 26 .

Many Clusters? QA Cluster Prod Cluster Dev Int QA1 QA2 QA3 Prod1 Prod2 Prod3 Dev1 Int1 QA Cluster Prod Cluster Exadata: from Beginner to Advanced in 3 Hours Dev Int 27 .

Disk Failures Datafile block1 Cell 1 block1 Cell 2 block1 Exadata: from Beginner to Advanced in 3 Hours 28 .

Disk Failures Datafile block1 Cell 1 block1 Cell 2 block1 Cell 3 block1 High Redundancy Exadata: from Beginner to Advanced in 3 Hours 29 .

Playing Nice • Database Resource Manager • I/O Resource Manager • Cell Fencing QA Prod Compute Nodes Storage Cells Exadata: from Beginner to Advanced in 3 Hours 30 .

Divide and Conquer Database Machine Network Storage DBA System Admin Network Admin ?? DBA Sys Admin Network Admin Cell Admin Exadata: from Beginner to Advanced in 3 Hours 31 .

Combined Skills Database Machine New Role Network Storage Database Machine Administrator (DMA) DBA Sys Admin Network Admin Cell Admin Exadata: from Beginner to Advanced in 3 Hours 32 .

On nodes – yes Q: How do I monitor it? A: Enterprise Manager. CellCLI.Other Questions Q: Do clients have to connect using Infiniband? A: No. Ethernet is also available Q: How do you back it up? A: Normal RMAN Backup. just like an Oracle Database Q: How do you create DR? A: Data Guard is the only solution Q: Can I install any other software? A: Nothing on Cells. SQL Commands Exadata: from Beginner to Advanced in 3 Hours 33 .

pdf • Golden Gate Exadata: from Beginner to Advanced in 3 Hours 34 .oracle.Backup and DR • No SAN connectivity • Only NAS – Infiniband – Tape .com/technetwork/database/features/availa bility/maa-wp-dr-dbm-130065. Disk Pool Exadata Infiniband • DR – – – – – No Storage Level Replication Backup Only Data Guard Supplemental Logging Device Force Logging http://www.

ETL and Reporting Exadata OLTP Golden Gate Infiniband Informatica MicroStrategy Exalytics Exadata: from Beginner to Advanced in 3 Hours 35 .

Overall Activities • Physical Aspects – Delivery. • • • • Layout Planning Installation and Configuration Data Migration Administration – Who manages it – Backup and Disaster Recovery • Application Development Exadata: from Beginner to Advanced in 3 Hours 36 . network components. etc. power.

Power Delivery Units • Over or Under the unit • Power Requirement – Single-Phase Low Voltage Americas / Japan / Taiwan) – Single-Phase High Voltage (EMEA & APAC (excluding Japan / Taiwan) – Three-Phase Low Voltage (Americas / Japan / Taiwan) – Three-Phase High Voltage (EMEA & APAC (excluding Japan / Taiwan) Exadata: from Beginner to Advanced in 3 Hours 37 .

Half Rack: 11.Network Ports • NET0 – Admin Interface • NET1. NET2 – Network Access to Nodes • NET3 – Backup Network • IB – Infiniband Network – IP Addr: Qtr Rack: 5. Full rack: 22 Exadata: from Beginner to Advanced in 3 Hours 38 .

6. 3. 8. 5. 7. Configuration Worksheet Pre-delivery Survey Generate config files Run checkip. 4.Installation Activities 1.sh Power on and validate components Configure KVM Configure IB Configure Cisco Switch Exadata: from Beginner to Advanced in 3 Hours Oracle HW 39 ACS . 2.

9. Validate Storage Cells 11. contd.sh 14. Config files from USB 13. Validate Compute Nodes 12.Installation. Run OneCommand Oracle HW ACS 40 Exadata: from Beginner to Advanced in 3 Hours . Configure IP to PDUs 10. Firstboot and applyconfig. Stage Oracle Software on Node 1 15.

reduces I/O at cells even for full table scans • Cell is controlled by CellCLI commands • DMA skills = 60% RAC DBA + 15% Linux + 20% CellCLI + 5% miscellaneous Exadata: from Beginner to Advanced in 3 Hours 41 .Summary • Exadata is an Oracle Database running 11. when possible.2 • The storage cells have added intelligence about data placement • The compute nodes run Oracle DB and Grid Infra • Nodes communicate with Cells using iDB which can send more information on the query • Smart Scan.

oracle.com/groups?home=&gid=918317 Exadata: from Beginner to Advanced in 3 Hours 42 .oracle.com/technetwork/tutorials/index.jspa?forumID=829 • Exadata SIG – http://www.com/technetwork/database/exadata/index.oracle.html • OTN Exadata Forum – https://forums.ly/k4mKQS – 4-part Exadata Reference article series http://bit.com/forums/forum.Resources • My Articles – 5-part Linux Commands article series http://bit.ly/lljFl0 • OTN Page on Exadata – http://www.linkedin. html • Tutorials – http://www.

Thank You! My Blog: arup.blogspot.com My Tweeter: arupnanda Exadata: from Beginner to Advanced in 3 Hours 43 .