Step by step how to build Exadata Virtual Machine - Cell Node | Oracle Exadata Architec...

Page 1 of 15

Oracle Exadata Architecture & Best practices

Blog everything I know about Oracle Linux Exadata Storage & Architecture
Home
Archives
Site Map
Disclaimer
About me
Step by step how to build
Exadata Virtual Machine - Cell
Node
4 replies

Step by step how to build Exadata Virtual

Machine - Cell Node

Original link:

http://www.dbaleet.org/how_to_build_an_exadata_simulator_step_by_step_1_build_a_cell_node

overmars students in early April to ask me how to build a set

of Exadata specific virtual machine, then my answer is that I
will write in May before the Exadata how to build a virtual
machine's articles, please go to my attention when the blog.
Here the students have to apologize to overmars because due
to some personal reasons stood me up. Anyway, Just hope it
is not too late.

I know that many Oracle DBA Exadata are interested in

learning, but has been struggling to learn there are not a test
Exadata environment. To know the Oracle Exadata machine is
a combination of hardware and software, simply through
their own PC will never be simulated real Exadata
environments. So here's Exadata virtual machine says it
plainly, then just follow the cat out of the tiger. Inside a virtual
machine on Oracle Exadata has existed, but only in Oracle
University or Oracle Internal Exadata to train or study
purposes within Oracle's Web site identifies this virtual
machine "Internal Use Only, Strict Confidential" message. I

http://translate.googleusercontent.com/translate_c?depth=1&hl=en&ie=UTF8&prev=_t&r... 7/15/2013
Step by step how to build Exadata Virtual Machine - Cell Node | Oracle Exadata Architec... Page 2 of 15

unintentionally violate O remember policy, so they need to

start building your own from start to finish.

Well, ado, to build a set of Exadata virtual environments

requires at least two virtual machines, one for Cell nodes, one
for DB nodes.

First you need a higher machine configuration:

• CPU Intel Core i3 above (or AMD Athlon II X4 above), it

is recommended Core i5 (AMD Phenom II X4);
• RAM (Memory) at least more than 4G, recommended
configuration 8G;
• Disk (Harddisk) spare at least 40G or more, of course, if
there is a better SSD
• Installed a virtual machine, it is recommended to use
Oracle Virtualbox ( https://www.virtualbox.org/ );
• Oracle Linux 5.7 installation media. Can go
https://edelivery.oracle.com/ download, download
needed before registration, registration is free. Oracle
Linux 5.7 media called V27570-01.zip, after extracting
the file named OracleLinux-
OracleLinux-R5-
R5-U7-
U7-Server-
Server-x86_64-
x86_64-dvd.iso
• Exadata 11.2.3.2 Cell installation media. Can go
https://edelivery.oracle.com/ download, download
needed before registration, registration is free. Exadata
11.2.3.2's Cell medium called V33693-01.zip unzip the
file name cellImageMaker_11.2.3.2.0_LINUX.X64_120713-
cellImageMaker_11.2.3.2.0_LINUX.X64_120713-
1.x86_64.tar;

• Oracle Clusterware 11.2.0.3 and Oracle database

11.2.0.3 for Linux x86_64 installation media, file name:
p10404530_112030_Linux-
p10404530_112030_Linux-x86-
x86-64_1of7.zip
p10404530_112030_Linux-
p10404530_112030_Linux-x86-
x86-64_2of7.zip
p10404530_112030_Linux-
p10404530_112030_Linux-x86-
x86-64_3of7.zip
• The latest patch tool Opatch. Patch Number: 6880880:
OPatch patch of version 11.2.0.3.4 for Oracle software
releases 11.2.0.x (APRIL 2013)
• Exadata RDBMS Bundle Patch 17 Patch ID: 16474946

http://translate.googleusercontent.com/translate_c?depth=1&hl=en&ie=UTF8&prev=_t&r... 7/15/2013
Step by step how to build Exadata Virtual Machine - Cell Node | Oracle Exadata Architec... Page 3 of 15

Then you can officially begin our tour of Exadata.

You first need to install in the virtual machine Oracle Linux

5.7, (Red Hat Enterprise Linux theory can, but I have not
tested), 1GB memory allocation is usually sufficient. The
installation process is very simple, you need to pay attention
to is the need to choose the software development kit, such
as gcc / aio and the like, graphical interface (GUI) from time to
dress. Recommended to use a static IP address, my network
configuration is as follows:

[Root @ cell ~] # cat / etc/sysconfig/network-scripts

# Intel Corporation 82540EM Gigabit Ethernet Controlle
DEVICE = eth0
BOOTPROTO = static
BROADCAST = 192.168.56.255
HWADDR = 08:00:27: B0: 39:02
IPADDR = 192.168.56.101
NETMASK = 255.255.255.0
NETWORK = 192.168.56.0
ONBOOT = yes
[Root @ cell ~] # cat / etc / hosts
# Do not remove the following line, or various program
# That require network functionality will fail.
127.0.0.1 cell localhost.localdomain localhost
:: 1 localhost6.localdomain6 localhost6

Note: After installation is complete, Oracle Linux default UEK,

if used here UEK, in a later step fails to start cellsrv services.
You can modify the grub configuration will modify its default
boot kernel redhat-compatible kernel:

[Root @ cell ~] # vi / etc / grub.conf

Modify the default = 0 to default = 1, then restart.

http://translate.googleusercontent.com/translate_c?depth=1&hl=en&ie=UTF8&prev=_t&r... 7/15/2013
Step by step how to build Exadata Virtual Machine - Cell Node | Oracle Exadata Architec... Page 4 of 15

Because the default Oracle Linux boot a lot we do not need

the service, in order to save resources, we recommend the
following services stopped and disabled.

chkconfig - level 2345 auditd off && service auditd s

chkconfig - level 2345 autofs off && service autofs st
chkconfig - level 2345 avahi-daemon off && service ava
chkconfig - level 2345 bluetooth off && service blueto
chkconfig - level 2345 cups off && service cups stop
chkconfig - level 2345 ip6tables off && service ip6tab
chkconfig - level 2345 iptables off && service iptable
chkconfig - level 2345 isdn off && service isdn stop
chkconfig - level 2345 kudzu off && service kudzu stop
chkconfig - level 2345 mcstrans off && service auditd s
chkconfig - level 2345 netfs off && service netfs stop
chkconfig - level 2345 pcscd off && service pcscd stop
chkconfig - level 2345 restorecond off && service rest
chkconfig - level 2345 rhnsd off && service rhnsd stop
chkconfig - level 2345 sendmail off && service sendmai
chkconfig - level 2345 setroubleshoot off && service s
chkconfig - level 2345 smartd off && service smartd st
chkconfig - level 2345 xinetd off && service xinetd st
chkconfig - level 2345 yum-updatesd off && service yum

Of course, these services can also be disabled via ntsysv-level

2345 in the graphical interface to select, cancel unneeded
services, and then restart to take effect.

Then we will Exadata Cell Image V33693-01.zip uploaded to a

virtual machine, unpack, get
cellImageMaker_11.2.3.2.0_LINUX.X64_120713-1.x86_64.tar,
continue decompression get a folder named dl180.

[Root @ cell ~] # unzip V33693-01.zip

Archive: V33693-01.zip
inflating: README.txt
inflating: cellImageMaker_11.2.3.2.0_LINUX.X64_12071

http://translate.googleusercontent.com/translate_c?depth=1&hl=en&ie=UTF8&prev=_t&r... 7/15/2013
Step by step how to build Exadata Virtual Machine - Cell Node | Oracle Exadata Architec... Page 5 of 15

[Root @ cell ~] # tar-pxvf cellImageMaker_11.2.3.2.0_L

dl180 ......

In dl180/boot/cellbits to find cell.bin file. The bin file is actually

a zip archive, we use unzip to decompress it:

[Root @ cell ~] # unzip cell.bin

Archive: cell.bin
warning [cell.bin]: 6408 extra bytes at beginning or w
(Attempting to process anyway)
inflating: cell-11.2.3.2.1_LINUX.X64_130109-1.x86_64
inflating: jdk-1_5_0_15-linux-amd64.rpm

Obtained after extracting cell-11.2.3.2.1_LINUX.X64_130109-

1.x86_64.rpm and jdk-1_5_0_15-linux-amd64.rpm two rpm
packages

Let's install jdk:

[Root @ cell ~] # rpm-ivh jdk-1_5_0_15-linux-amd64.rp

Then install the cell:

[Root @ cell ~] # rpm-ivh cell-11.2.3.2.1_LINUX.X64_1

Errors when installing, suggesting LWP package

dependencies, this is because not installed by default perl-
libwww-perl lead, but this package depends more
recommended to use yum to install.

Configure yum source directly using yum install LWP:

http://translate.googleusercontent.com/translate_c?depth=1&hl=en&ie=UTF8&prev=_t&r... 7/15/2013
Step by step how to build Exadata Virtual Machine - Cell Node | Oracle Exadata Architec... Page 6 of 15

[Root @ cell ~] # yum install perl-libwww-perl

Install again cell, an error again, provided conditions are not

met.

But what specific prerequisite conditions are not met without

prompting, rpm package management is this inconvenient.
Can only be generated script specific inspection criteria and
then analyze what conditions is satisfied:

[Root @ cell ~] # rpm - scripts-qp cell-11.2.3.2.1_LI

Open diag.log, should soon see / var / log / oracle directory

does not exist causes a prerequisite checks pass, then
manually create this directory, and modify the permissions to
775.

[Root @ cell ~] # mkdir-p / var / log / oracle

[Root @ cell ~] # chmod-R 775 / var / log / oracle

Install cell again this time without error.

The next step should be to create a virtual machine in the cell

corresponding virtual disks and flash drives:

[Root @ cell ~] # mkdir-p / opt / oracle / cell / dis

[Root @ cell ~] cd / opt / oracle / cell / disks / raw
[Root @ cell ~] vi dd.sh
[Root @ cell ~] cat dd.sh
dd if = / dev / zero of = disk01 bs = 1M count = 1000
dd if = / dev / zero of = disk02 bs = 1M count = 1000

http://translate.googleusercontent.com/translate_c?depth=1&hl=en&ie=UTF8&prev=_t&r... 7/15/2013
Step by step how to build Exadata Virtual Machine - Cell Node | Oracle Exadata Architec... Page 7 of 15

dd if = / dev / zero of = disk03 bs = 1M count = 1000

dd if = / dev / zero of = disk04 bs = 1M count = 1000
dd if = / dev / zero of = disk05 bs = 1M count = 1000
dd if = / dev / zero of = disk06 bs = 1M count = 1000
dd if = / dev / zero of = disk07 bs = 1M count = 1000
dd if = / dev / zero of = disk08 bs = 1M count = 1000
dd if = / dev / zero of = disk09 bs = 1M count = 1000
dd if = / dev / zero of = disk10 bs = 1M count = 1000
dd if = / dev / zero of = disk11 bs = 1M count = 1000
dd if = / dev / zero of = disk12 bs = 1M count = 1000
dd if = / dev / zero of = FLASH01 bs = 1M count = 1000
dd if = / dev / zero of = FLASH02 bs = 1M count = 1000
dd if = / dev / zero of = FLASH03 bs = 1M count = 1000
dd if = / dev / zero of = FLASH04 bs = 1M count = 1000

Executive dd.sh flash disks and create the corresponding:

wherein the disk 12, the size of each block 1GB, flash 4, not
the block size is 1GB.

[Root @ cell raw] # chmod 660 *

[Root @ cell raw] # ls-ltr
total 16400068
-Rw-rw ---- 1 root root 692 May 16 16:24 dd.sh
-Rw-rw ---- 1 root root 1048576000 May 16 16:24 disk01
-Rw-rw ---- 1 root root 1048576000 May 16 16:24 disk02
-Rw-rw ---- 1 root root 1048576000 May 16 16:24 disk03
-Rw-rw ---- 1 root root 1048576000 May 16 16:24 disk04
-Rw-rw ---- 1 root root 1048576000 May 16 16:25 disk05
-Rw-rw ---- 1 root root 1048576000 May 16 16:25 disk06
-Rw-rw ---- 1 root root 1048576000 May 16 16:25 disk07
-Rw-rw ---- 1 root root 1048576000 May 16 16:26 disk08
-Rw-rw ---- 1 root root 1048576000 May 16 16:26 disk09
-Rw-rw ---- 1 root root 1048576000 May 16 16:27 disk10
-Rw-rw ---- 1 root root 1048576000 May 16 16:27 disk11
-Rw-rw ---- 1 root root 1048576000 May 16 16:27 disk12
-Rw-rw ---- 1 root root 1048576000 May 16 16:27 FLASH0
-Rw-rw ---- 1 root root 1048576000 May 16 16:27 FLASH0
-Rw-rw ---- 1 root root 1048576000 May 16 16:27 FLASH0
-Rw-rw ---- 1 root root 1048576000 May 16 16:28 FLASH0

http://translate.googleusercontent.com/translate_c?depth=1&hl=en&ie=UTF8&prev=_t&r... 7/15/2013
Step by step how to build Exadata Virtual Machine - Cell Node | Oracle Exadata Architec... Page 8 of 15

Then delete dd script, switch to celladmin users, restart celld

services.

[Root @ cell ~] # su - celladmin

[Celladmin @ cell ~] $ cellcli-e alter cell restart se

Found cellsrv service fails to start, check /

opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/log/diag/asm/cell/cell/trace/alert.log
found error message similar to the following:

CELLSRV version = 11.2.3.2.1, label = OSS_11.2.3.2.1_

Non critical error DIA-48913 caught while writing to t
Error message: DIA-48913: Writing into trace file fail

Can be judged from the error number should be less than the
maximum number of files, so the need to modify the
operating system limits the maximum number of files:

In the / etc / sysctl.ctl added last line: fs.file-max = 65536, then

refresh the effect:

[Root @ cell ~] # sysctl-p

net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 2
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
fs.file-max = 65536

http://translate.googleusercontent.com/translate_c?depth=1&hl=en&ie=UTF8&prev=_t&r... 7/15/2013
Step by step how to build Exadata Virtual Machine - Cell Node | Oracle Exadata Architec... Page 9 of 15

In the / etc / security / limit.conf file, add two lines:

* Soft nofile 65536

* Hard nofile 65536

Then log out, switch to celladmin, use the ulimit-a to view take
effect:

[Root @ cell ~] # ulimit-a

core file size (blocks,-c) 0
data seg size (kbytes,-d) unlimited
scheduling priority (-e) 0
file size (blocks,-f) unlimited
pending signals (-i) 11999
max locked memory (kbytes,-l) 32
max memory size (kbytes,-m) unlimited
open files (-n) 65536
pipe size (512 bytes,-p) 8
POSIX message queues (bytes,-q) 819200
real-time priority (-r) 0
stack size (kbytes,-s) 10240
cpu time (seconds,-t) unlimited
max user processes (-u) 11999
virtual memory (kbytes,-v) unlimited
file locks (-x) unlimited

Cell all the services to start again:

[Celladmin @ cell ~] $ cellcli-e alter cell restart s

The discovery cell under cellsrv, ms, rs services can be

properly started.

Next you need to add in cellinit.ora card information:

http://translate.googleusercontent.com/translate_c?depth=1&hl=en&ie=UTF8&prev=_t&r... 7/15/2013
Step by step how to build Exadata Virtual Machine - Cell Node | Oracle Exadata Archit... Page 10 of 15

[Celladmin @ cell ~] $ cellcli-e create cell cell1 in

After successful execution, you can see cellinit.ora file, add a

line ipaddress1 = 192.168.56.101/24 similar information.

[Root @ cell config] # cat / opt / oracle / cell / ce

# CELL Initialization Parameters
version = 0.0
DEPLOYED = TRUE
HTTP_PORT = 8888
RMI_PORT = 23791
SSL_PORT = 23943
JMS_PORT = 9127
BMC_SNMP_PORT = 162
ipaddress1 = 192.168.56.101/24

Next create celldisk, griddisk, flashcache, flashlog:

[Celladmin @ cell ~] $ cellcli

CellCLI: Release 11.2.3.2.1 - Production on Thu May 16

Copyright (c) 2007, 2012, Oracle. All rights reserved.

Cell Efficiency Ratio: 1

CellCLI> alter cell restart services all

Stopping the RS, CELLSRV, and MS services ...

The SHUTDOWN of services was successful.
Starting the RS, CELLSRV, and MS services ...
Getting the state of RS services ... running
Starting CELLSRV services ...
The STARTUP of CELLSRV services was successful.
Starting MS services ...
The STARTUP of MS services was successful.

CellCLI> create celldisk all

http://translate.googleusercontent.com/translate_c?depth=1&hl=en&ie=UTF8&prev=_t&r... 7/15/2013
Step by step how to build Exadata Virtual Machine - Cell Node | Oracle Exadata Archit... Page 11 of 15

CellDisk FD_00_cell1 successfully created

CellDisk FD_01_cell1 successfully created
CellDisk FD_02_cell1 successfully created
CellDisk FD_03_cell1 successfully created
CellDisk CD_disk01_cell1 successfully created
CellDisk CD_disk02_cell1 successfully created
CellDisk CD_disk03_cell1 successfully created
CellDisk CD_disk04_cell1 successfully created
CellDisk CD_disk05_cell1 successfully created
CellDisk CD_disk06_cell1 successfully created
CellDisk CD_disk07_cell1 successfully created
CellDisk CD_disk08_cell1 successfully created
CellDisk CD_disk09_cell1 successfully created
CellDisk CD_disk10_cell1 successfully created
CellDisk CD_disk11_cell1 successfully created
CellDisk CD_disk12_cell1 successfully created

CellCLI> create flashcache all size = 2G

Flash cache cell1_FLASHCACHE successfully created

CellCLI> create flashlog all

Flash log cell1_FLASHLOG successfully created

CellCLI> list flashcache detail

name: cell1_FLASHCACHE
cellDisk: FD_00_cell1, FD_03_cell1, FD_02_cel
creationTime: 2013-05-16T17: 11:57 +08:00
degradedCelldisks:
effectiveCacheSize: 2G
id: 33020341-ba55-4b35-9b3a-4030b5085475
size: 2G
status: normal

CellCLI> list flashlog detail

name: cell1_FLASHLOG
cellDisk: FD_01_cell1, FD_03_cell1, FD_02_cel
creationTime: 2013-05-16T17: 12:10 +08:00
degradedCelldisks:
effectiveSize: 512M
efficiency: 100.0
id: f10e1ac7-5e3f-4c1e-8f3b-8e9ab19fffeb
size: 512M
status: normal

http://translate.googleusercontent.com/translate_c?depth=1&hl=en&ie=UTF8&prev=_t&r... 7/15/2013
Step by step how to build Exadata Virtual Machine - Cell Node | Oracle Exadata Archit... Page 12 of 15

CellCLI> list cell

cell1 online

CellCLI> list celldisk

CD_disk01_cell1 normal
CD_disk02_cell1 normal
CD_disk03_cell1 normal
CD_disk04_cell1 normal
CD_disk05_cell1 normal
CD_disk06_cell1 normal
CD_disk07_cell1 normal
CD_disk08_cell1 normal
CD_disk09_cell1 normal
CD_disk10_cell1 normal
CD_disk11_cell1 normal
CD_disk12_cell1 normal
FD_00_cell1 normal
FD_01_cell1 normal
FD_02_cell1 normal
FD_03_cell1 normal

CellCLI> list griddisk

data_CD_disk01_cell1 active
data_CD_disk02_cell1 active
data_CD_disk03_cell1 active
data_CD_disk04_cell1 active
data_CD_disk05_cell1 active
data_CD_disk06_cell1 active
data_CD_disk07_cell1 active
data_CD_disk08_cell1 active
data_CD_disk09_cell1 active
data_CD_disk10_cell1 active
data_CD_disk11_cell1 active
data_CD_disk12_cell1 active

CellCLI> list celldisk CD_disk01_cell1 detail

name: CD_disk01_cell1
comment:
creationTime: 2013-05-16T16: 40:29 +08:00
deviceName: / opt/oracle/cell11.2.3.2.1_LINUX
devicePartition: / opt/oracle/cell11.2.3.2.1_

http://translate.googleusercontent.com/translate_c?depth=1&hl=en&ie=UTF8&prev=_t&r... 7/15/2013
Step by step how to build Exadata Virtual Machine - Cell Node | Oracle Exadata Archit... Page 13 of 15

diskType: HardDisk
errorCount: 0
freeSpace: 0
id: ecc913eb-5f74-4ad6-9d05-f811af986921
interleaving: none
lun: / opt/oracle/cell11.2.3.2.1_LINUX.X64_13
physicalDisk: / opt/oracle/cell11.2.3.2.1_LIN
raidLevel: "RAID 0"
size: 992M
status: normal

CellCLI> list celldisk FD_00_cell1 detail

name: FD_00_cell1
comment:
creationTime: 2013-05-16T16: 40:25 +08:00
deviceName: / opt/oracle/cell11.2.3.2.1_LINUX
devicePartition: / opt/oracle/cell11.2.3.2.1_
diskType: FlashDisk
errorCount: 0
freeSpace: 304M
freeSpaceMap: offset = 688M, size = 304M
id: c9488ae4-d3b9-4aa2-a4e5-d3539e44b417
interleaving: none
lun: / opt/oracle/cell11.2.3.2.1_LINUX.X64_13
physicalDisk: / opt/oracle/cell11.2.3.2.1_LIN
raidLevel: "RAID 0"
size: 992M
status: normal

CellCLI> list griddisk data_CD_disk01_cell1 detail

name: data_CD_disk01_cell1
availableTo:
cachingPolicy: default
cellDisk: CD_disk01_cell1
comment:
creationTime: 2013-05-16T16: 49:51 +08:00
diskType: HardDisk
errorCount: 0
id: 7a36bc8a-1611-474d-85fc-fa730e73176d
offset: 48M
size: 944M
status: active

http://translate.googleusercontent.com/translate_c?depth=1&hl=en&ie=UTF8&prev=_t&r... 7/15/2013
Step by step how to build Exadata Virtual Machine - Cell Node | Oracle Exadata Archit... Page 14 of 15

At this point the basic cell node virtual machine has been
created.

Share to:

Related posts:

1. Step by step how to build Exadata Virtual Machine - DB

node
2. How do the Exadata cell node systemstate dump
3. Exadata Cell command line tool CellCLI

This entry posted on May 16, 2013 . Belong to Exadata ,

Exadata Configuration classification was posted Exadata
Simulator , Exadata Virtuabox , Exadata VM , Exadata virtual
machine label. Author Steven Lee .
← Oracle DRM technology step by step how to build
changes (five) Exadata Virtual Machine - DB
node →
0    0

     

http://translate.googleusercontent.com/translate_c?depth=1&hl=en&ie=UTF8&prev=_t&r... 7/15/2013
Step by step how to build Exadata Virtual Machine - Cell Node | Oracle Exadata Archit... Page 15 of 15

[?]

http://translate.googleusercontent.com/translate_c?depth=1&hl=en&ie=UTF8&prev=_t&r... 7/15/2013