RAC Lab Part 10 - Installing Grid Infrastructure

U a
RAC lab Part 10 – Installing Grid

Infrastructure
by Maciej Tokar | Dec 6, 2016 | Enterprise Edition, Installation, Linux |
This is tenth part of my RAC lab series. I will show you step by step how to prepare for
and how to install Oracle 12c Grid Infrastructure
RAC lab Part 1 – Installing the Ubuntu 16.04 desktop
RAC lab Part 2 – Virtualbox installation and configuration
RAC lab Part 3 – VMs configuration
RAC lab Part 4 – Installing the Openfiler software
RAC lab Part 5 – Installing OEL 7.x on VM
RAC lab Part 6 – OEL 7.x configuration for Oracle 12c GI & RAC DB installation
RAC lab Part 7 – Openfilers configuration and cloning
RAC lab Part 8 – OEL 7.x prepare storage
RAC lab Part 9 – Clone first RAC node as 2nd node and prepare config.
RAC lab Part 10 – Installing Grid Infrastructure
RAC lab Part 11 – Installing Database Software
RAC lab Part 12 – Creating a Container Database
Oracle 12c official documentation
Please refer to the official 12.1 documentation and CLUVFY documentation:
Grid Infrastructure Installation Guide

https://docs.oracle.com/database/121/CWLIN/toc.htm
Clusterware Administration and Deployment Guide – Cluster Verification Utility Reference
https://docs.oracle.com/database/121/CWADD/cvu.htm#CWADD1100
Prepare installation files for grid and database software
I need shared directory between host and VMs to to keep oracle installation files there. From
RAC nodes I can then grab those files and unzip them into /u01/app/grid/grid-software and
/u01/app/oracle/oracle-software. VboxAddition software will help to establish thos shared
dirs.
Prepare directory for shared dir
1 maciek@macieksrv.dba24.pl /home/maciek/Downloads # sudo mkdir -p /opt/vboxshared/

Oracle12.1_install
Move files downloaded from Oracle to that directory
1 maciek@macieksrv.dba24.pl /home/maciek/Downloads # sudo mv /home/maciek/Downl

oads/V46*.zip /opt/vboxshared/Oracle12.1_install
Set this folder as shared with the vms
1 vboxuser1@macieksrv.dba24.pl ~ $ vboxmanage sharedfolder add oel7rac1n1 --name S

2 haredWithHost --hostpath /opt/vboxshared --automount
3 vboxuser1@macieksrv.dba24.pl ~ $ vboxmanage sharedfolder add oel7rac1n2 --name S
4 haredWithHost --hostpath /opt/vboxshared --automount
5
6 # check if directory has been mounted in the VM
maciek@oel7rac1n1.dba24.pl ~ $ df -k|grep Shared
SharedWithHost 135981224 43987764 91993460 33% /media/sf_SharedWithHost
You can access the directory as root, or grant vboxsf group to the user that should get to the
directory. I will use root sudo to manipulate within it
Unpack software first on the first node oel72rac1n1

1 # grid infrastructure software
2 root@oel7rac1n1.dba24.pl ~ $ cd /u01/app/grid/grid-software/
3 root@oel7rac1n1.dba24.pl ~ $unzip /media/sf_SharedWithHost/Oracle12.1_install/V46
4 096-01_1of2.zip
5 root@oel7rac1n1.dba24.pl ~ $unzip /m edia/sf_SharedWithHost/Oracle12.1_install/V46
6 096-01_2of2.zip
7 root@oel7rac1n1.dba24.pl ~ $chown grid:oinstall -R .
8 root@oel7rac1n1 grid-software ~ ls
9 grid
10
11 # oracle database software
12 root@oel7rac1n1.dba24.pl ~ $cd /u01/app/oracle/oracle-software/
14 095-01_1of2.zip
095-01_2of2.zip
root@oel7rac1n1.dba24.pl ~ $chown oracle:oinstall -R .
root@oel7rac1n1 oracle-software ~ ls
database
Configure SSH equivalency
Generate ssh keys on both nodes for oracle and grid user
1 # grid user
2
3 [grid@oel7rac1n1 ~]$ /usr/bin/ssh-keygen -t dsa
4 Generating public/private dsa key pair.
5 Enter file in which to save the key (/home/grid/.ssh/id_dsa):
6 Created directory '/home/grid/.ssh'.
7 Enter passphrase (empty for no passphrase):
8 Enter same passphrase again:
9 Your identification has been saved in /home/grid/.ssh/id_dsa.
10 Your public key has been saved in /home/grid/.ssh/id_dsa.pub.
11 The key fingerprint is:
12 80:86:19:46:95:f8:b0:0b:2e:7d:9f:b2:dd:75:93:52 grid@oel7rac1n1
13 The key's randomart image is:
14 +--[ DSA 1024]----+
15 | .+o.. |
16 | .o+.. |
17 | o+o . |
18 |. ... . |
19 |.o . S E |
20 |..o . . . |
21 |. . . . o + |
22 | ..o. . o . |
23 | .o. . |
24 +-----------------+
25
26 [grid@oel7rac1n2 ~]$ /usr/bin/ssh-keygen -t dsa
28 Enter file in which to save the key (/home/grid/.ssh/id_dsa):
29 Created directory '/home/grid/.ssh'.
32 Your identification has been saved in /home/grid/.ssh/id_dsa.
33 Your public key has been saved in /home/grid/.ssh/id_dsa.pub.
35 27:3b:bf:81:cb:c8:c0:af:af:19:50:aa:2c:8d:7b:99 grid@oel7rac1n2
35 27:3b:bf:81:cb:c8:c0:af:af:19:50:aa:2c:8d:7b:99 grid@oel7rac1n2
37 +--[ DSA 1024]----+
38 | |
39 | |
40 | . |
41 | o |
42 | o S. |
43 |.+ o = |
44 |+..o+ + . |
45 |..E * o + . |
46 |.. ++= o o. |
47 +-----------------+
48
49 # oracle user
50 [oracle@oel7rac1n1 ~]$ /usr/bin/ssh-keygen -t dsa
52 Enter file in which to save the key (/home/oracle/.ssh/id_dsa):
53 Created directory '/home/oracle/.ssh'.
56 Your identification has been saved in /home/oracle/.ssh/id_dsa.
57 Your public key has been saved in /home/oracle/.ssh/id_dsa.pub.
59 34:08:9b:d3:1c:42:6e:ee:c6:b3:2b:27:0e:76:31:90 oracle@oel7rac1n1
61 +--[ DSA 1024]----+
62 | .+ . |
63 | .. B o |
64 |E *+o |
65 | .o . . . |
66 | o. S |
67 | oo |
68 |.. .= |
69 |..+..o |
70 | ..+o. |
71 +-----------------+
72
73 [oracle@oel7rac1n2 ~]$ /usr/bin/ssh-keygen -t dsa
75 Enter file in which to save the key (/home/oracle/.ssh/id_dsa):
76 Created directory '/home/oracle/.ssh'.
79 Your identification has been saved in /home/oracle/.ssh/id_dsa.
80 Your public key has been saved in /home/oracle/.ssh/id_dsa.pub.
82 be:c2:28:ab:c9:c3:98:fd:60:85:60:92:4c:83:1e:09 oracle@oel7rac1n2
84 +--[ DSA 1024]----+
85 |Eo. |
86 |++. |
87 |=o. |
88 |oo . |
89 | .. S |
90 | . . |
91 |ooo o . |
92 |+=o.. o . |
93 |ooo+. .. |
94 +-----------------+
Distribute public key among nodes
1 # grid user
1 # grid user
2 [grid@oel7rac1n1 ~]$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_key
3 [grid@oel7rac1n2 ~]$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
4
5 # grid - Now exchange public keys - you need to authorize with password this time
6 [grid@oel7rac1n1 ~]$ cat ~/.ssh/id_dsa.pub | ssh 192.168.1.23 'cat >> ~/.ssh/authorize
7 d_keys'
8 [grid@oel7rac1n2 ~]$ cat ~/.ssh/id_dsa.pub | ssh 192.168.1.21 'cat >> ~/.ssh/authorize
9 d_keys'
10
11 # grid - check authorized_keys on both nodes
12 [grid@oel7rac1n1 ~]$ cat ~/.ssh/authorized_keys
13 ssh-dss AAAAB3NzaC1kc3MAAACBAI0dDNJuzAPMJa2tojJYWGY4GbjnYbw8IArzoi7
14 WMjvgqoQMqjVezOJqR/IDmjy01mNROB8/VBGwfDpxeDPfnT/5KvbVEdKiiq1bJLuAKS
15 XX7CPc6aowhRxWlkKt6hJ1LJpc6/VbGCdNi9+LB/Y+blHXom5B0O0VhV4Sdjmb1id7A
16 AAAFQChfDLe3tOEqwB0DbBNTU7ph33ujQAAAIAv/prdXQwxVzFsqMNJEx7Z4veP/D
17 QMWC/zAbVm6nWGFjt2LwclJcIuWCsGNR56PuSkmusHfHrOInYsN49hfUrRutjgOd8C
18 9gCbrxKwZzsfKTvuRgfdLqBoStsLKRzEzrcQgIEICEle//132VUl40ybwM0X/CR+5zKEY9f
19 nMYvblwAAAIAajCd0xKwQbDbDK6bI5yhnevt/ghOP3U3j9yvVHCPTHJZEAbVJ3zUf2Q
20 Tw8C7TjTI6z/YmtcZE+OHC5b2f5WWQYfjjFhluj5d3lZyE0mdnkzqmlZa1/QmXDofgYXV
21 GsHN6GQChX6HiSqDKdeyIMJl/lH6bFeYWmLE3uPBWiHkNUw== grid@oel7rac1n1
22 ssh-dss AAAAB3NzaC1kc3MAAACBAJDBUHAWNv8RDCipyyS+64utARxY8T0tmPVIX
23 J9DasL4/uiceIlmDjZgl75MTmJ/hZIxSdEGsy+d0C03izfWyfswn6uuwOvyehaSt4YJAy9L
24 Vxu2T5kinE1J4HbTGLnViGV5afCAriScf+d7MrqegE9D7p0XoRjUOML53Ug3fy83AAAA
25 FQCYhuD4ul0iA14hfz8d3s1dtzsarQAAAIA43iNxCQtz+TycFM5ATXj75sZ/Z1Xy6CjcOP
26 ed0QzXIjdFSC04/6O3N0eOeKxh74TcLsmhrbKKrhZQMDzJY9BGCLf5zw/8YyXQMSBs
27 0khmscw8UiM0uqtLqREj0gzf3iedNbv99pSbZFuMy1dMvrJE55h6l9QczQfaBvwlx/Pd+Q
28 AAAIBzrC6LC2hrF5rtSaG+pzGvADR7r3BjWdvyRWMSH+EL5LqeiluWYOBxCJb0egLa
29 zdmZ4bybgwTdGGOs0lseK8eg5pU+PYjAmfsqh23kYoX1VhQndnV+WqF+6yrsKtBnhQ
30 nfhxboWXVP2fgOCyZdWU9oza9yZ0rcQTZVtGX1WNuIPw== grid@oel7rac1n2
31
32 [grid@oel7rac1n2 ~]$ cat ~/.ssh/authorized_keys
33 ssh-dss AAAAB3NzaC1kc3MAAACBAJDBUHAWNv8RDCipyyS+64utARxY8T0tmPVIX
34 J9DasL4/uiceIlmDjZgl75MTmJ/hZIxSdEGsy+d0C03izfWyfswn6uuwOvyehaSt4YJAy9L
Vxu2T5kinE1J4HbTGLnViGV5afCAriScf+d7MrqegE9D7p0XoRjUOML53Ug3fy83AAAA
FQCYhuD4ul0iA14hfz8d3s1dtzsarQAAAIA43iNxCQtz+TycFM5ATXj75sZ/Z1Xy6CjcOP
ed0QzXIjdFSC04/6O3N0eOeKxh74TcLsmhrbKKrhZQMDzJY9BGCLf5zw/8YyXQMSBs
0khmscw8UiM0uqtLqREj0gzf3iedNbv99pSbZFuMy1dMvrJE55h6l9QczQfaBvwlx/Pd+Q
AAAIBzrC6LC2hrF5rtSaG+pzGvADR7r3BjWdvyRWMSH+EL5LqeiluWYOBxCJb0egLa
zdmZ4bybgwTdGGOs0lseK8eg5pU+PYjAmfsqh23kYoX1VhQndnV+WqF+6yrsKtBnhQ
nfhxboWXVP2fgOCyZdWU9oza9yZ0rcQTZVtGX1WNuIPw== grid@oel7rac1n2
ssh-dss AAAAB3NzaC1kc3MAAACBAI0dDNJuzAPMJa2tojJYWGY4GbjnYbw8IArzoi7
WMjvgqoQMqjVezOJqR/IDmjy01mNROB8/VBGwfDpxeDPfnT/5KvbVEdKiiq1bJLuAKS
XX7CPc6aowhRxWlkKt6hJ1LJpc6/VbGCdNi9+LB/Y+blHXom5B0O0VhV4Sdjmb1id7A
AAAFQChfDLe3tOEqwB0DbBNTU7ph33ujQAAAIAv/prdXQwxVzFsqMNJEx7Z4veP/D
QMWC/zAbVm6nWGFjt2LwclJcIuWCsGNR56PuSkmusHfHrOInYsN49hfUrRutjgOd8C
9gCbrxKwZzsfKTvuRgfdLqBoStsLKRzEzrcQgIEICEle//132VUl40ybwM0X/CR+5zKEY9f
nMYvblwAAAIAajCd0xKwQbDbDK6bI5yhnevt/ghOP3U3j9yvVHCPTHJZEAbVJ3zUf2Q
Tw8C7TjTI6z/YmtcZE+OHC5b2f5WWQYfjjFhluj5d3lZyE0mdnkzqmlZa1/QmXDofgYXV
GsHN6GQChX6HiSqDKdeyIMJl/lH6bFeYWmLE3uPBWiHkNUw== grid@oel7rac1n1
# oracle user
[oracle@oel7rac1n1 ~]$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
[oracle@oel7rac1n2 ~]$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
# oracle - Now exchange public keys - you need to authorize with password this time
[oracle@oel7rac1n1 ~]$ cat ~/.ssh/id_dsa.pub | ssh 192.168.1.23 'cat >> ~/.ssh/authoriz
ed_keys'
[oracle@oel7rac1n2 ~]$ cat ~/.ssh/id_dsa.pub | ssh 192.168.1.21 'cat >> ~/.ssh/authoriz
ed_keys'
# oracle check authorized_keys on both nodes
[oracle@oel7rac1n1 ~]$ cat ~/.ssh/authorized_keys
ssh-dss AAAAB3NzaC1kc3MAAACBAIKik8bZNzknobVbakJBHVragYJaJPVCKfQ+s1J
+d3178e2F0ob4uL3YPe5uD32xFYEJ+iMY9bEWgTYRAApv9GwrmPK3Aa6OkPrlxb6hz
KZAnZpCIZ3LKInPUX5chRACzEFij3IOQ0jI8zCN6FtYq0+1uvCe5+zrnBHaMiHKALhxA
AAAFQCulsVAW1i2IZKX5UpRMTAcfbsekQAAAIBTANR7Ney6BW56jrOGP970CHblq
G+WM3SREkmnL3pVr2cAemt0FLRNXv4rP42NYndvkVcDJN+BLFwI+JqROCe5EVTW
aiWxmPVKkzlKcpeLtb6PBqMtt+nNeoBobyRbkc2XlYmTrtydLq//nGarCE7NgTlCneCX4v
N4ueNvLkJsoAAAAIBEgTd1l9yBsnC4q0YPkDiKJXBqnIy6LffbCCfHFk2BYgMKFIMKS9
oxFFX5ffS9PkJL5W8FJeaMEr1b7ulunilERevbTsTyNFDYKp5ppIdhNdK6fWfk6zhcy7p5
PBfS6PC04gqI5HQX7OuBSGggtMF2FDhC3dt3IrX1g6MpR1Vjlw== oracle@oel7rac1n
1
ssh-dss AAAAB3NzaC1kc3MAAACBALy8jUD8xVLUrhfYGXJBOn9LSznvFkktXuYtlKjcg
KHmJk9Ix+jrEKbrTs74F1Beu0OXA0WF8BJeCR/GjyCdOKXlOpAcB9mVzZtyV9IEt//xHv
O+vfcctJ/qAuHD//i50oipm1LqtYkbTtKadlo5dF+vjT3d1spCZfFY1vxK9PCBAAAAFQDbd
NJiQViGTXJEFSGL8ctNoRuUnQAAAIEAvIzt2aAqF0qGvwAzFLqyPK9yWAAeoHpv/Of
pHj7LO3OOtMoYY8i1msXaPOWAaOQQPUNTqEILgKEEczLpG81HEikRQ6DC0l1Ilc/q
PIfkJX44ONJAcaH3IigM8cIWdiSKhw/odGWZRNms/67on0hB9mzKil0urBYMHEzicpbK
uBQAAACBAKL+hlA3Vfl0MQb//RwSRum273/u44j33wi0LhZtsSbhvDw9eM4oC4kcubv
KvU0+UFgsvqPShr2OlUa03lKxzZlr6kroDWglk1hy1Z9S7dgpTWybomODYHmffiGqS4dI
87NZvoLGVsEMX68z73vpGH/O/Tz8fwNf+zoOrmEH+Iad oracle@oel7rac1n2
[oracle@oel7rac1n2 ~]$ cat ~/.ssh/authorized_keys
ssh-dss AAAAB3NzaC1kc3MAAACBALy8jUD8xVLUrhfYGXJBOn9LSznvFkktXuYtlKjcg
KHmJk9Ix+jrEKbrTs74F1Beu0OXA0WF8BJeCR/GjyCdOKXlOpAcB9mVzZtyV9IEt//xHv
O+vfcctJ/qAuHD//i50oipm1LqtYkbTtKadlo5dF+vjT3d1spCZfFY1vxK9PCBAAAAFQDbd
NJiQViGTXJEFSGL8ctNoRuUnQAAAIEAvIzt2aAqF0qGvwAzFLqyPK9yWAAeoHpv/Of
pHj7LO3OOtMoYY8i1msXaPOWAaOQQPUNTqEILgKEEczLpG81HEikRQ6DC0l1Ilc/q
PIfkJX44ONJAcaH3IigM8cIWdiSKhw/odGWZRNms/67on0hB9mzKil0urBYMHEzicpbK
uBQAAACBAKL+hlA3Vfl0MQb//RwSRum273/u44j33wi0LhZtsSbhvDw9eM4oC4kcubv
KvU0+UFgsvqPShr2OlUa03lKxzZlr6kroDWglk1hy1Z9S7dgpTWybomODYHmffiGqS4dI
87NZvoLGVsEMX68z73vpGH/O/Tz8fwNf+zoOrmEH+Iad oracle@oel7rac1n2
ssh-dss AAAAB3NzaC1kc3MAAACBAIKik8bZNzknobVbakJBHVragYJaJPVCKfQ+s1J
+d3178e2F0ob4uL3YPe5uD32xFYEJ+iMY9bEWgTYRAApv9GwrmPK3Aa6OkPrlxb6hz
AAAFQCulsVAW1i2IZKX5UpRMTAcfbsekQAAAIBTANR7Ney6BW56jrOGP970CHblq
G+WM3SREkmnL3pVr2cAemt0FLRNXv4rP42NYndvkVcDJN+BLFwI+JqROCe5EVTW
aiWxmPVKkzlKcpeLtb6PBqMtt+nNeoBobyRbkc2XlYmTrtydLq//nGarCE7NgTlCneCX4v
N4ueNvLkJsoAAAAIBEgTd1l9yBsnC4q0YPkDiKJXBqnIy6LffbCCfHFk2BYgMKFIMKS9
oxFFX5ffS9PkJL5W8FJeaMEr1b7ulunilERevbTsTyNFDYKp5ppIdhNdK6fWfk6zhcy7p5
PBfS6PC04gqI5HQX7OuBSGggtMF2FDhC3dt3IrX1g6MpR1Vjlw== oracle@oel7rac1n
1
Remember that you should have set passwords for oracle and grid on all the nodes
Test bidirectional ssh connectivity without passwords both from oracle and grid user.
Establish connections to short and long server names (fqdn) in the public network
(192.168.1.x)
Cluster verification before GI installation – CLUVFY
I strongly recommend using cluvfy tool to check if the cluster is ready before running the
installation process. I does similar checks as OUI, but you won’t be stuck in the middle of the
process if something is wrong. Cluster verification utility provides a lot of options to check
your environment before you run clusterware, database installations or upgrades. The tool is
evolving so expect more to come in later versions
System requirements verification: cluvfy comp sys
Use the cluvfy comp sys component verification command to check that the minimum system
requirements are met for the specified product on all the specified nodes.
1 [grid@oel7rac1n1 bin]$ ./cluvfy comp sys -n oel7rac1n1,oel7rac1n2 -p crs -r 12.1
2
3 Verifying system requirement
4 Total memory check passed
5 Available memory check passed
6 Swap space check passed
7 Free disk space check passed for "oel7rac1n2:/usr,oel7rac1n2:/var,oel7rac1n2:/etc,oel
8 7rac1n2:/sbin,oel7rac1n2:/tmp"
9 Free disk space check passed for "oel7rac1n1:/usr,oel7rac1n1:/var,oel7rac1n1:/etc,oel
10 7rac1n1:/sbin,oel7rac1n1:/tmp"
11 Check for multiple users with UID value 2002 passed
12 User existence check passed for "grid"
13 Group existence check passed for "oinstall"
14 Group existence check passed for "dba"
15 Membership check for user "grid" in group "oinstall" [as Primary] passed
16 Membership check for user "grid" in group "dba" passed
17 Run level check passed
18 Hard limits check passed for "maximum open file descriptors"
19 Soft limits check passed for "maximum open file descriptors"
20 Hard limits check passed for "maximum user processes"
21 Soft limits check passed for "maximum user processes"
22 System architecture check passed
23 Kernel version check passed
24 Kernel parameter check passed for "semmsl"
25 Kernel parameter check passed for "semmns"
26 Kernel parameter check passed for "semopm"
27 Kernel parameter check passed for "semmni"
28 Kernel parameter check passed for "shmmax"
29 Kernel parameter check passed for "shmmni"
30 Kernel parameter check passed for "shmall"
31 Kernel parameter check passed for "file-max"
32 Kernel parameter check passed for "ip_local_port_range"
33 Kernel parameter check passed for "rmem_default"
34 Kernel parameter check passed for "rmem_max"
35 Kernel parameter check passed for "wmem_default"
36 Kernel parameter check passed for "wmem_max"
37 Kernel parameter check passed for "aio-max-nr"
38 Kernel parameter check passed for "panic_on_oops"
39 Package existence check passed for "binutils"
40 Package existence check passed for "compat-libcap1"
41 Package existence check passed for "compat-libstdc++-33(x86_64)"
42 Package existence check passed for "libgcc(x86_64)"
43 Package existence check passed for "libstdc++(x86_64)"
44 Package existence check passed for "libstdc++-devel(x86_64)"
45 Package existence check passed for "sysstat"
46 Package existence check passed for "gcc"
47 Package existence check passed for "gcc-c++"
48 Package existence check passed for "ksh"
49 Package existence check passed for "make"
50 Package existence check passed for "glibc(x86_64)"
51 Package existence check passed for "glibc-devel(x86_64)"
52 Package existence check passed for "libaio(x86_64)"
53 Package existence check passed for "libaio-devel(x86_64)"
54 Package existence check passed for "nfs-utils"
56
57 Starting check for consistency of primary group of root user
58
59 Check for consistency of root user's primary group passed
Time zone consistency check passed
Verification of system requirement was successful.
Verification successful.
Verify storage using: cluvfy comp ssa
Use the cluvfy comp ssa component verification command to discover and check the sharing
of the specified storage locations. CVU checks sharing for nodes in the node list.
1 # Check disks for OCR

2
3 [grid@oel7rac1n1 bin]$ ./cluvfy comp ssa -n oel7rac1n1,oel7rac1n2 -r 12.1 -asm -asm
4 dev /dev/asm/op1-rac1-asm-crs-01,/dev/asm/op2-rac1-asm-crs-02,/dev/asm/op3-rac1-
5 asm-crs-03 -t ocr_vdisk
6 Verifying shared storage accessibility
7
8 Checking shared storage accessibility...
9
10 Disk Sharing Nodes (2 in count)
11 ------------------------------------ ------------------------
12 /dev/asm/op1-rac1-asm-crs-01 oel7rac1n1 oel7rac1n2
13
15 ------------------------------------ ------------------------
17
19 ------------------------------------ ------------------------
21
22
23 Shared storage check was successful on nodes "oel7rac1n1,oel7rac1n2"
24
25 Verification of shared storage accessibility was successful.
26
27
28 ## Check disks for DATA
29
31 dev /dev/asm/op1-rac1-asm-data-01,/dev/asm/op1-rac1-asm-data-02,/dev/asm/op1-ra
32 c1-asm-data-03,/dev/asm/op1-rac1-asm-data-04,/dev/asm/op1-rac1-asm-fra-01,/dev/a
33 sm/op1-rac1-asm-fra-02,/dev/asm/op1-rac1-asm-fra-03,/dev/asm/op1-rac1-asm-fra-04
34 ,/dev/asm/op2-rac1-asm-data-01,/dev/asm/op2-rac1-asm-data-02,/dev/asm/op2-rac1-
35 asm-data-03,/dev/asm/op2-rac1-asm-data-04,/dev/asm/op2-rac1-asm-fra-01,/dev/asm
36 /op2-rac1-asm-fra-02,/dev/asm/op2-rac1-asm-fra-03,/dev/asm/op2-rac1-asm-fra-04 -t
37 data
38
40
42
44 ------------------------------------ ------------------------
45 /dev/asm/op1-rac1-asm-data-01 oel7rac1n1 oel7rac1n2
46
48 ------------------------------------ ------------------------
50
52 ------------------------------------ ------------------------
54
56 ------------------------------------ ------------------------
58
60 ------------------------------------ ------------------------
61 /dev/asm/op1-rac1-asm-fra-01 oel7rac1n1 oel7rac1n2
63 ------------------------------------ ------------------------
65
67 ------------------------------------ ------------------------
69
71 ------------------------------------ ------------------------
73
75 ------------------------------------ ------------------------
77
79 ------------------------------------ ------------------------
81
83 ------------------------------------ ------------------------
85
87 ------------------------------------ ------------------------
89
91 ------------------------------------ ------------------------
93
95 ------------------------------------ ------------------------
97
99 ------------------------------------ ------------------------
Disk Sharing Nodes (2 in count)
------------------------------------ ------------------------
/dev/asm/op2-rac1-asm-fra-04 oel7rac1n1 oel7rac1n2
Shared storage check was successful on nodes "oel7rac1n1,oel7rac1n2"

Verification of shared storage accessibility was successful.
Shared storage verified successfully.
Network verification using: cluvfy stage -post hwos
Use the cluvfy stage -post hwos stage verification command to perform network and storage
verifications on the specified nodes in the cluster before installing Oracle software. This
command also checks for supported storage types and checks each one for sharing.
As we have already checked the disks we are just interested in the network checks, ignore
storage information – storage checks will faile
1 grid@oel7rac1n1 bin]$ ./cluvfy stage -post hwos -n oel7rac1n1,oel7rac1n2

2
3 Performing post-checks for hardware and operating system setup
4
5 Checking node reachability...
6 Node reachability check passed from node "oel7rac1n1"
7
8
9 Checking user equivalence...
10 User equivalence check passed for user "grid"
11
12 Checking node connectivity...
13
14 Checking hosts config file...
15
16 Verification of the hosts config file successful
17
18 Node connectivity passed for subnet "192.168.1.0" with node(s) oel7rac1n1,oel7rac1n2
19 TCP connectivity check passed for subnet "192.168.1.0"
20
21 Node connectivity passed for subnet "192.168.10.0" with node(s) oel7rac1n1,oel7rac1n
22 2
24
26 2
28
30 2
32
34 2
36
38 2
40
41
42 Interfaces found on subnet "192.168.1.0" that are likely candidates for VIP are:
43 oel7rac1n1 enp0s3:192.168.1.21
44 oel7rac1n2 enp0s3:192.168.1.23
45
46 Interfaces found on subnet "192.168.10.0" that are likely candidates for a private interc
47 onnect are:
48 oel7rac1n1 enp0s8:192.168.10.21
49 oel7rac1n2 enp0s8:192.168.10.23
50
52 onnect are:
53 oel7rac1n1 enp0s9:192.168.20.21
54 oel7rac1n2 enp0s9:192.168.20.23
56 onnect are:
57 oel7rac1n1 enp0s10:192.168.21.21
58 oel7rac1n2 enp0s10:192.168.21.23
59
61 onnect are:
62 oel7rac1n1 enp0s16:192.168.30.21
63 oel7rac1n2 enp0s16:192.168.30.23
64
66 onnect are:
67 oel7rac1n1 enp0s17:192.168.31.21
68 oel7rac1n2 enp0s17:192.168.31.23
69 Checking subnet mask consistency...
70 Subnet mask consistency check passed for subnet "192.168.1.0".
76 Subnet mask consistency check passed.
77
78 Node connectivity check passed
79
80 Checking multicast communication...
81
82 Checking subnet "192.168.1.0" for multicast communication with multicast group "224.0
83 .0.251"...
84 Check of subnet "192.168.1.0" for multicast communication with multicast group "224.0.
85 0.251" passed.
86
87 Check of multicast communication passed.
89 Time zone consistency check passed
90
No shared storage found
Shared storage check failed on nodes "oel7rac1n1,oel7rac1n2"

Checking integrity of name service switch configuration file "/etc/nsswitch.conf" ...
All nodes have same "hosts" entry defined in file "/etc/nsswitch.conf"
Check for integrity of name service switch configuration file "/etc/nsswitch.conf" passed
Post-check for hardware and operating system setup was unsuccessful on all the node
s.
Verification successful (ingore storage failed checks)
Verify storage using: cluvfy comp ssa
Use the cluvfy comp ssa component verification command to discover and check the sharing
of the specified storage locations. CVU checks sharing for nodes in the node list.
1 # Check disks for OCR

2
4 dev /dev/asm/op1-rac1-asm-crs-01,/dev/asm/op2-rac1-asm-crs-02,/dev/asm/op3-rac1-
5 asm-crs-03 -t ocr_vdisk
7
9
9
11 ------------------------------------ ------------------------
13
15 ------------------------------------ ------------------------
17
19 ------------------------------------ ------------------------
21
22
23 Shared storage check was successful on nodes "oel7rac1n1,oel7rac1n2"
24
25 Verification of shared storage accessibility was successful.
26
27
28 ## Check disks for DATA
29
31 dev /dev/asm/op1-rac1-asm-data-01,/dev/asm/op1-rac1-asm-data-02,/dev/asm/op1-ra
32 c1-asm-data-03,/dev/asm/op1-rac1-asm-data-04,/dev/asm/op1-rac1-asm-fra-01,/dev/a
33 sm/op1-rac1-asm-fra-02,/dev/asm/op1-rac1-asm-fra-03,/dev/asm/op1-rac1-asm-fra-04
34 ,/dev/asm/op2-rac1-asm-data-01,/dev/asm/op2-rac1-asm-data-02,/dev/asm/op2-rac1-
35 asm-data-03,/dev/asm/op2-rac1-asm-data-04,/dev/asm/op2-rac1-asm-fra-01,/dev/asm
36 /op2-rac1-asm-fra-02,/dev/asm/op2-rac1-asm-fra-03,/dev/asm/op2-rac1-asm-fra-04 -t
37 data
38
40
42
44 ------------------------------------ ------------------------
46
48 ------------------------------------ ------------------------
50
52 ------------------------------------ ------------------------
54
56 ------------------------------------ ------------------------
58
60 ------------------------------------ ------------------------
63 ------------------------------------ ------------------------
65
67 ------------------------------------ ------------------------
69
71 ------------------------------------ ------------------------
73
75 ------------------------------------ ------------------------
75 ------------------------------------ ------------------------
77
79 ------------------------------------ ------------------------
81
83 ------------------------------------ ------------------------
85
87 ------------------------------------ ------------------------
89
91 ------------------------------------ ------------------------
93
95 ------------------------------------ ------------------------
97
99 ------------------------------------ ------------------------
Disk Sharing Nodes (2 in count)
------------------------------------ ------------------------
/dev/asm/op2-rac1-asm-fra-04 oel7rac1n1 oel7rac1n2
Shared storage check was successful on nodes "oel7rac1n1,oel7rac1n2"

Verification of shared storage accessibility was successful.
Shared storage verified successfully.
Nodes connectivity verification: cluvfy comp nodecon
Use the cluvfy comp nodecon component verification command to check the connectivity
among the nodes specified in the node list. If you provide an interface list, then CVU checks
the connectivity using only the specified interfaces.
1 [grid@oel7rac1n1 bin]$ ./cluvfy comp nodecon -n oel7rac1n1,oel7rac1n2

2
3 Verifying node connectivity
4
6
8
10
11 Node connectivity passed for subnet "192.168.1.0" with node(s) oel7rac1n1,oel7rac1n2
13
15 2
17
17
19 2
21
23 2
25
27 2
29
31 2
33
34
36 oel7rac1n1 enp0s3:192.168.1.21
37 oel7rac1n2 enp0s3:192.168.1.23
38
40 onnect are:
41 oel7rac1n1 enp0s8:192.168.10.21
42 oel7rac1n2 enp0s8:192.168.10.23
43
45 onnect are:
46 oel7rac1n1 enp0s9:192.168.20.21
47 oel7rac1n2 enp0s9:192.168.20.23
48
50 onnect are:
51 oel7rac1n1 enp0s10:192.168.21.21
52 oel7rac1n2 enp0s10:192.168.21.23
53
55 onnect are:
56 oel7rac1n1 enp0s16:192.168.30.21
57 oel7rac1n2 enp0s16:192.168.30.23
58
60 onnect are:
61 oel7rac1n1 enp0s17:192.168.31.21
62 oel7rac1n2 enp0s17:192.168.31.23
71
Node connectivity check passed
Checking multicast communication...
Checking subnet "192.168.1.0" for multicast communication with multicast group "224.0
.0.251"...
Check of subnet "192.168.1.0" for multicast communication with multicast group "224.0.
0.251" passed.
Check of multicast communication passed.
Verification of node connectivity was successful.
Node connectivity verification successful
check if servers are ready for GI installation – final check: cluvfy stage [-pre | -post] crsinst
Use the cluvfy stage -pre crsinst command with either the -file, -n, -flex, or -upgrade
parameters to check the specified nodes before installing or upgrading Oracle Clusterware.
Use the cluvfy stage -post crsinst command to check the specified nodes after installing
Oracle Clusterware.
From what you will see, the last check incorporates all the previous checks, but it’s good to
run them all just to make sure we have and understand all the components
1 [grid@oel7rac1n1 ~]$ cd /u01/app/grid/cluvfy/bin/

2 [grid@oel7rac1n1 bin]$ ./cluvfy stage -pre crsinst -n oel7rac1n1,oel7rac1n2
3
4 Performing pre-checks for cluster services setup
5
8
9
12
14
16
18
20 2
22
23 Node connectivity passed for subnet "192.168.10.0" with node(s) oel7rac1n1,oel7rac1
24 n2
26
28 n2
30
32 n2
34
36 n2
38
40 n2
42
43
45 oel7rac1n1 enp0s3:192.168.1.21
45 oel7rac1n1 enp0s3:192.168.1.21
46 oel7rac1n2 enp0s3:192.168.1.23
47
48 Interfaces found on subnet "192.168.10.0" that are likely candidates for a private inter
49 connect are:
50 oel7rac1n1 enp0s8:192.168.10.21
51 oel7rac1n2 enp0s8:192.168.10.23
52
54 connect are:
55 oel7rac1n1 enp0s9:192.168.20.21
56 oel7rac1n2 enp0s9:192.168.20.23
57
59 connect are:
60 oel7rac1n1 enp0s10:192.168.21.21
61 oel7rac1n2 enp0s10:192.168.21.23
62
64 connect are:
65 oel7rac1n1 enp0s16:192.168.30.21
66 oel7rac1n2 enp0s16:192.168.30.23
68 connect are:
69 oel7rac1n1 enp0s17:192.168.31.21
70 oel7rac1n2 enp0s17:192.168.31.23
79
81
83
84 Checking subnet "192.168.1.0" for multicast communication with multicast group "224.
85 0.0.251"...
86 Check of subnet "192.168.1.0" for multicast communication with multicast group "224.
87 0.0.251" passed.
88
90 Total memory check passed
91 Available memory check passed
92 Swap space check passed
93 Free disk space check passed for "oel7rac1n2:/usr,oel7rac1n2:/var,oel7rac1n2:/etc,oe
94 l7rac1n2:/sbin,oel7rac1n2:/tmp"
95 Free disk space check passed for "oel7rac1n1:/usr,oel7rac1n1:/var,oel7rac1n1:/etc,oe
96 l7rac1n1:/sbin,oel7rac1n1:/tmp"
98 User existence check passed for "grid"
99 Group existence check passed for "oinstall"
100 Group existence check passed for "dba"
101 Membership check for user "grid" in group "oinstall" [as Primary] passed
102 Membership check for user "grid" in group "dba" passed
103 Run level check passed
104 Hard limits check passed for "maximum open file descriptors"
105 Soft limits check passed for "maximum open file descriptors"
106 Hard limits check passed for "maximum user processes"
107 Soft limits check passed for "maximum user processes"
108 System architecture check passed
109 Kernel version check passed
110 Kernel parameter check passed for "semmsl"
111
Kernel parameter check passed for "semmsl"
111 Kernel parameter check passed for "semmns"
112 Kernel parameter check passed for "semopm"
113 Kernel parameter check passed for "semmni"
114 Kernel parameter check passed for "shmmax"
115 Kernel parameter check passed for "shmmni"
116 Kernel parameter check passed for "shmall"
117 Kernel parameter check passed for "file-max"
118 Kernel parameter check passed for "ip_local_port_range"
119
120 Kernel parameter check passed for "rmem_default"
121 Kernel parameter check passed for "rmem_max"
122 Kernel parameter check passed for "wmem_default"
123 Kernel parameter check passed for "wmem_max"
124 Kernel parameter check passed for "aio-max-nr"
127 Package existence check passed for "binutils"
128 Package existence check passed for "compat-libcap1"
129 Package existence check passed for "compat-libstdc++-33(x86_64)"
130 Package existence check passed for "libgcc(x86_64)"
131 Package existence check passed for "libstdc++(x86_64)"
132 Package existence check passed for "libstdc++-devel(x86_64)"
133 Package existence check passed for "sysstat"
134 Package existence check passed for "gcc"
135 Package existence check passed for "gcc-c++"
136 Package existence check passed for "ksh"
137 Package existence check passed for "make"
138 Package existence check passed for "glibc(x86_64)"
139 Package existence check passed for "glibc-devel(x86_64)"
140 Package existence check passed for "libaio(x86_64)"
141 Package existence check passed for "libaio-devel(x86_64)"
142 Package existence check passed for "nfs-utils"
143
144 Checking availability of ports "6200,6100" required for component "Oracle Notification
145 Service (ONS)"
146 Port availability check passed for ports "6200,6100"
147
148 Checking availability of ports "42424" required for component "Oracle Cluster Synchro
149 nization Services (CSSD)"
150 Port availability check passed for ports "42424"
152 Current group ID check passed
153
154 Starting check for consistency of primary group of root user
155
156 Check for consistency of root user's primary group passed
157
158 Starting Clock synchronization checks using Network Time Protocol(NTP)...
159 No NTP Daemons or Services were found to be running
160
161 Clock synchronization check using Network Time Protocol(NTP) passed
162
163 Core file name pattern consistency check passed.
164
165 User "grid" is not part of "root" group. Check passed
166 Default user file creation mask check passed
167 Checking integrity of file "/etc/resolv.conf" across nodes
168
169 "domain" and "search" entries do not coexist in any "/etc/resolv.conf" file
170 The DNS response time for an unreachable node is within acceptable limit on all node
171 s
172 checking DNS response from all servers in "/etc/resolv.conf"
173
174 Check for integrity of file "/etc/resolv.conf" passed
175
177
178 Checking integrity of name service switch configuration file "/etc/nsswitch.conf" ...
179 All nodes have same "hosts" entry defined in file "/etc/nsswitch.conf"
180 Check for integrity of name service switch configuration file "/etc/nsswitch.conf" passe
d
Checking daemon "avahi-daemon" is not configured and running

Daemon not configured check passed for process "avahi-daemon"
Daemon not running check passed for process "avahi-daemon"
Starting check for /dev/shm mounted as temporary file system ...
Check for /dev/shm mounted as temporary file system passed
Starting check for /boot mount ...
Check for /boot mount passed
Starting check for zeroconf check ...
Check for zeroconf check passed
Pre-check for cluster services setup was successful.
OK!! we are ready to install the cluster software
OUI Grid Installation
Run OUI installer
1 [grid@oel7rac1n1 ~]$ cd /u01/app/grid/grid-software/grid/

2 [grid@oel7rac1n1 grid]$ ./runInstaller
3 Starting Oracle Universal Installer...
4
5 Checking Temp space: must be greater than 415 MB. Actual 21611 MB Passed
6 Checking swap space: must be greater than 150 MB. Actual 16895 MB Passed
7 Checking monitor: must be configured to display at least 256 colors. Actual 16777216
8 Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2016-12-02_05-18-35
PM. Please wait ...
Choose to install and configure GI for a cluster

Choose configure standard cluster
Choose advance installation
Choose preferred languages to install

Set cluster name, SCAN name and port. Disable GNS
Provide list of nodes to be used as cluster nodes.
Verify ssh user equivalency
Declare cluster networks

Declare cluster networks
Use Flex ASM for storage
Set disk string in order to get our prepared disks to the list
Create crs diskgroup using /dev/asm/*asm*crs* disks
Ooops! As you can see OUI can’t take care of the quorum group, so it complains that the crs-
03 disk presented from openfiler3 is too small and with different size than crs-01 and crs-02.
The same problem also exists in 11g R2.
Some interesting notes:

http://www.oracle.com/technetwork/products/clusterware/overview/grid-infra-
thirdvoteonnfs-131158.pdf
http://www.redstk.com/voting-disk-setup-in-a-12c-grid-infrastructure-extended-rac-
environment/
We need to create temporary disk on openfiler3 with size of 10GB as crs-02 and crs-03. We
need to present it to cluster nodes and prepare on those VMs.
Create new volume (click to enlarge)
Check volumes now (click to enlarge)

Map new lun to the existing target on openfiler3 (click to enlarge)
Check the new lun on openfiler3 system level
1 [root@openfiler3 ~]# cat /proc/net/iet/volume

2 tid:1 name:iqn.2006-01.com.openfiler:op3.rac1.asm
3 lun:0 state:0 iotype:blockio iomode:wt path:/dev/racvg/rac1-asm-crs-03
4 lun:1 state:0 iotype:blockio iomode:wt path:/dev/racvg/rac1-asm-tmp
Discover disk on the cluster nodes
1 [root@oel7rac1n1 ~]# rescan-scsi-bus.sh

2 Scanning SCSI subsystem for new devices
3 Scanning host 0 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs
5 Scanning for device 1 0 0 0 ...
6 OLD: Host: scsi1 Channel: 00 Id: 00 Lun: 00
7 Vendor: VBOX Model: CD-ROM Rev: 1.0
8 Type: CD-ROM ANSI SCSI revision: 05
13 Vendor: ATA Model: VBOX HARDDISK Rev: 1.0
14 Type: Direct-Access ANSI SCSI revision: 05
21 Vendor: OPNFILER Model: VIRTUAL-DISK Rev: 0
98 NEW: Host: scsi11 Channel: 00 Id: 00 Lun: 01
101 1 new or changed device(s) found.
102 [11:0:0:1]
103 0 remapped or resized device(s) found.
104 0 device(s) removed.
105
106 [root@oel7rac1n2 ~]# rescan-scsi-bus.sh
112 Vendor: VBOX Model: CD-ROM Rev: 1.0
113 Type: CD-ROM ANSI SCSI revision: 05
118 Vendor: ATA Model: VBOX HARDDISK Rev: 1.0
133
204 NEW: Host: scsi11 Channel: 00 Id: 00 Lun: 01
207 1 new or changed device(s) found.
208 [11:0:0:1]
209 0 remapped or resized device(s) found.
0 device(s) removed.
New device found [11:0:0:1] on both nodes, this is our temporary LUN.
Find device name assigned to the new disk

1 [root@oel7rac1n1 ~]# lsscsi
2 [1:0:0:0] cd/dvd VBOX CD-ROM 1.0 /dev/sr0
3 [3:0:0:0] disk ATA VBOX HARDDISK 1.0 /dev/sda
4 [7:0:0:0] disk OPNFILER VIRTUAL-DISK 0 /dev/sdb
5 [7:0:0:1] disk OPNFILER VIRTUAL-DISK 0 /dev/sdc
6 [7:0:0:2] disk OPNFILER VIRTUAL-DISK 0 /dev/sdd
7 [7:0:0:3] disk OPNFILER VIRTUAL-DISK 0 /dev/sde
8 [7:0:0:4] disk OPNFILER VIRTUAL-DISK 0 /dev/sdf
9 [7:0:0:5] disk OPNFILER VIRTUAL-DISK 0 /dev/sdg
10 [7:0:0:6] disk OPNFILER VIRTUAL-DISK 0 /dev/sdh
11 [7:0:0:7] disk OPNFILER VIRTUAL-DISK 0 /dev/sdi
12 [7:0:0:8] disk OPNFILER VIRTUAL-DISK 0 /dev/sdj
13 [9:0:0:0] disk OPNFILER VIRTUAL-DISK 0 /dev/sdk
14 [9:0:0:1] disk OPNFILER VIRTUAL-DISK 0 /dev/sdl
15 [9:0:0:2] disk OPNFILER VIRTUAL-DISK 0 /dev/sdm
16 [9:0:0:3] disk OPNFILER VIRTUAL-DISK 0 /dev/sdn
17 [9:0:0:4] disk OPNFILER VIRTUAL-DISK 0 /dev/sdo
18 [9:0:0:5] disk OPNFILER VIRTUAL-DISK 0 /dev/sdp
19 [9:0:0:6] disk OPNFILER VIRTUAL-DISK 0 /dev/sdq
20 [9:0:0:7] disk OPNFILER VIRTUAL-DISK 0 /dev/sdr
21 [9:0:0:8] disk OPNFILER VIRTUAL-DISK 0 /dev/sds
22 [11:0:0:0] disk OPNFILER VIRTUAL-DISK 0 /dev/sdt
23 [11:0:0:1] disk OPNFILER VIRTUAL-DISK 0 /dev/sdu
24
25 [root@oel7rac1n2 ~]# lsscsi
26 [1:0:0:0] cd/dvd VBOX CD-ROM 1.0 /dev/sr0
27 [3:0:0:0] disk ATA VBOX HARDDISK 1.0 /dev/sda
28 [7:0:0:0] disk OPNFILER VIRTUAL-DISK 0 /dev/sdb
29 [7:0:0:1] disk OPNFILER VIRTUAL-DISK 0 /dev/sdc
30 [7:0:0:2] disk OPNFILER VIRTUAL-DISK 0 /dev/sdd
31 [7:0:0:3] disk OPNFILER VIRTUAL-DISK 0 /dev/sde
32 [7:0:0:4] disk OPNFILER VIRTUAL-DISK 0 /dev/sdf
33 [7:0:0:5] disk OPNFILER VIRTUAL-DISK 0 /dev/sdg
34 [7:0:0:6] disk OPNFILER VIRTUAL-DISK 0 /dev/sdh
35 [7:0:0:7] disk OPNFILER VIRTUAL-DISK 0 /dev/sdi
36 [7:0:0:8] disk OPNFILER VIRTUAL-DISK 0 /dev/sdj
37 [9:0:0:0] disk OPNFILER VIRTUAL-DISK 0 /dev/sdk
38 [9:0:0:1] disk OPNFILER VIRTUAL-DISK 0 /dev/sdl
39 [9:0:0:2] disk OPNFILER VIRTUAL-DISK 0 /dev/sdm
40 [9:0:0:3] disk OPNFILER VIRTUAL-DISK 0 /dev/sdo
41 [9:0:0:4] disk OPNFILER VIRTUAL-DISK 0 /dev/sdp
42 [9:0:0:5] disk OPNFILER VIRTUAL-DISK 0 /dev/sdq
43 [9:0:0:6] disk OPNFILER VIRTUAL-DISK 0 /dev/sdr
44 [9:0:0:7] disk OPNFILER VIRTUAL-DISK 0 /dev/sds
45 [9:0:0:8] disk OPNFILER VIRTUAL-DISK 0 /dev/sdt
46 [11:0:0:0] disk OPNFILER VIRTUAL-DISK 0 /dev/sdn
47 [11:0:0:1] disk OPNFILER VIRTUAL-DISK 0 /dev/sdu
/dev/sdu – new disk device name on both nodes
Create partition on /dev/sdu on the first node

1 [root@oel7rac1n1 ~]# parted /dev/sdu mklabel gpt mkpart primary "1 -1"
2 Information: You may need to update /etc/fstab.
3
4 [root@oel7rac1n1 ~]# fdisk -l /dev/sdu
5 WARNING: fdisk GPT support is currently new, and therefore in an experimental phase
6 . Use at your own discretion.
7
8 Disk /dev/sdu: 10.5 GB, 10502537216 bytes, 20512768 sectors
9 Units = sectors of 1 * 512 = 512 bytes
10 Sector size (logical/physical): 512 bytes / 512 bytes
11 I/O size (minimum/optimal): 512 bytes / 512 bytes
12 Disk label type: gpt
13
14
15 # Start End Size Type Name
1 2048 20510719 9,8G Microsoft basic primary
Discover new partition on the second node
1 [root@oel7rac1n2 ~]# kpartx -a /dev/sdu
Find scsi ID
1 [root@oel7rac1n1 ~]# /usr/lib/udev/scsi_id --whitelisted --replace-whitespace /dev/sdu

2 14f504e46494c45524b6878344b792d4869724a2d4c414875
3
4 [root@oel7rac1n2 ~]# /usr/lib/udev/scsi_id --whitelisted --replace-whitespace /dev/sdu
5 14f504e46494c45524b6878344b792d4869724a2d4c414875
Add the following line to /etc/udev/rules.d/90-oracle-asmdevices.rules on both nodes.
1 KERNEL=="sd*1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d $

devnode", RESULT=="14f504e46494c45524b6878344b792d4869724a2d4c414875", SY
MLINK+="asm/op3-rac1-asm-tmp", OWNER="grid", GROUP="asmadmin", MODE="066
0"
Reload udev and check asm disk (owner,group,privs)
1 [root@oel7rac1n1 ~]#udevadm control --reload-rules

2 [root@oel7rac1n1 ~]#udevadm trigger
3
4 [root@oel7rac1n2 ~]#udevadm control --reload-rules
5 [root@oel7rac1n2 ~]#udevadm trigger
6
7 [root@oel7rac1n1 ~]# ls -l /dev/asm/*|grep tmp
8 lrwxrwxrwx 1 root root 7 12-02 18:47 /dev/asm/op3-rac1-asm-tmp ->; ../sdu1
9 [root@oel7rac1n1 ~]# ls -l /dev/sdu1
10 brw-rw---- 1 grid asmadmin 65, 65 12-02 18:47 /dev/sdu1
11
12 [root@oel7rac1n2 ~]# ls -l /dev/asm/*|grep tmp
13 lrwxrwxrwx 1 root root 7 12-02 18:46 /dev/asm/op3-rac1-asm-tmp -> ../sdu1
14 [root@oel7rac1n2 ~]# ls -l /dev/sdu1
Create CRS diskgroup using temporary disk from openfiler3
Set passwords for ASM

Don’t configure IPMI
Do not register with EM Cloud Control unless you have it already
Set asm groups – leave default

Set ORACLE_BASE and ORACLE_HOME directories
Set oraInventory directory
We wan to run root scripts manually

Prerequisits being checked
Prerequisits being checked
It complains about /dev/shm size

As you can see it looks like OUI bug as size is ok
Confirm you want to ignore this warning

Installation plan summary screen
Installation process
Run scripts as root
1 # root scripts
2 [root@oel7rac1n1 ~]# /u01/app/oraInventory/orainstRoot.sh
3 Changing permissions of /u01/app/oraInventory.
4 Adding read,write permissions for group.
5 Removing read,write,execute permissions for world.
6
6
7 Changing groupname of /u01/app/oraInventory to oinstall.
8 The execution of the script is complete.
9
10
11 [root@oel7rac1n2 ~]# /u01/app/oraInventory/orainstRoot.sh
12 Changing permissions of /u01/app/oraInventory.
13 Adding read,write permissions for group.
14 Removing read,write,execute permissions for world.
15
16 Changing groupname of /u01/app/oraInventory to oinstall.
17 The execution of the script is complete.
18
19 [root@oel7rac1n1 ~]# /u01/app/12.1.0/grid/root.sh
20 Performing root user operation.
21
22 The following environment variables are set as:
23 ORACLE_OWNER= grid
24 ORACLE_HOME= /u01/app/12.1.0/grid
25
26 Enter the full pathname of the local bin directory: [/usr/local/bin]:
27 Copying dbhome to /usr/local/bin ...
28 Copying oraenv to /usr/local/bin ...
29 Copying coraenv to /usr/local/bin ...
30
31
32 Creating /etc/oratab file...
33 Entries will be added to the /etc/oratab file as needed by
34 Database Configuration Assistant when a database is created
35 Finished running generic part of root script.
36 Now product-specific root actions will be performed.
37 Using configuration parameter file: /u01/app/12.1.0/grid/crs/install/crsconfig_params
38 2016/12/02 19:27:19 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Colle
39 ctor.
40
41 2016/12/02 19:27:50 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer
42 (TFA) Collector.
43
44 2016/12/02 19:27:52 CLSRSC-363: User ignored prerequisites during installation
45
46 OLR initialization - successful
47 root wallet
48 root wallet cert
49 root cert export
50 peer wallet
51 profile reader wallet
52 pa wallet
53 peer wallet keys
54 pa wallet keys
55 peer cert request
56 pa cert request
57 peer cert
58 pa cert
59 peer root cert TP
60 profile reader root cert TP
61 pa root cert TP
62 peer pa cert TP
63 pa peer cert TP
64 profile reader pa cert TP
65 profile reader peer cert TP
66 peer user cert
67 pa user cert
68 2016/12/02 19:28:32 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.se
69 rvice'
70
71 CRS-4133: Oracle High Availability Services has been stopped.
72 CRS-4123: Oracle High Availability Services has been started.
75 CRS-2672: Attempting to start 'ora.evmd' on 'oel7rac1n1'
76 CRS-2672: Attempting to start 'ora.mdnsd' on 'oel7rac1n1'
77 CRS-2676: Start of 'ora.mdnsd' on 'oel7rac1n1' succeeded
78 CRS-2676: Start of 'ora.evmd' on 'oel7rac1n1' succeeded
79 CRS-2672: Attempting to start 'ora.gpnpd' on 'oel7rac1n1'
80 CRS-2676: Start of 'ora.gpnpd' on 'oel7rac1n1' succeeded
81 CRS-2672: Attempting to start 'ora.cssdmonitor' on 'oel7rac1n1'
82 CRS-2672: Attempting to start 'ora.gipcd' on 'oel7rac1n1'
83 CRS-2676: Start of 'ora.cssdmonitor' on 'oel7rac1n1' succeeded
84 CRS-2676: Start of 'ora.gipcd' on 'oel7rac1n1' succeeded
85 CRS-2672: Attempting to start 'ora.cssd' on 'oel7rac1n1'
86 CRS-2672: Attempting to start 'ora.diskmon' on 'oel7rac1n1'
87 CRS-2676: Start of 'ora.diskmon' on 'oel7rac1n1' succeeded
88 CRS-2676: Start of 'ora.cssd' on 'oel7rac1n1' succeeded
89
90 ASM created and started successfully.
91
92 Disk Group CRS created successfully.
93
94 CRS-2672: Attempting to start 'ora.crf' on 'oel7rac1n1'
95 CRS-2672: Attempting to start 'ora.storage' on 'oel7rac1n1'
96 CRS-2676: Start of 'ora.storage' on 'oel7rac1n1' succeeded
97 CRS-2676: Start of 'ora.crf' on 'oel7rac1n1' succeeded
98 CRS-2672: Attempting to start 'ora.crsd' on 'oel7rac1n1'
99 CRS-2676: Start of 'ora.crsd' on 'oel7rac1n1' succeeded
100 CRS-4256: Updating the profile
101 Successful addition of voting disk 1c193c21750c4fe3bfa28c6b5df261b3.
102 Successful addition of voting disk 54cd207d0d744fd8bf2714d7776f5ad2.
103 Successful addition of voting disk 355efd32461e4f3fbf50305bfac1657e.
104 Successfully replaced voting disk group with +CRS.
105 CRS-4256: Updating the profile
106 CRS-4266: Voting file(s) successfully replaced
107 ## STATE File Universal Id File Name Disk group
108 -- ----- ----------------- --------- ---------
109 1. ONLINE 1c193c21750c4fe3bfa28c6b5df261b3 (/dev/asm/op2-rac1-asm-crs-02) [
110 CRS]
111 2. ONLINE 54cd207d0d744fd8bf2714d7776f5ad2 (/dev/asm/op1-rac1-asm-crs-01) [
112 CRS]
113 3. ONLINE 355efd32461e4f3fbf50305bfac1657e (/dev/asm/op3-rac1-asm-tmp) [CR
114 S]
115 Located 3 voting disk(s).
116 CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources
117 on 'oel7rac1n1'
118 CRS-2673: Attempting to stop 'ora.crsd' on 'oel7rac1n1'
119 CRS-2677: Stop of 'ora.crsd' on 'oel7rac1n1' succeeded
120 CRS-2673: Attempting to stop 'ora.evmd' on 'oel7rac1n1'
121 CRS-2673: Attempting to stop 'ora.ctssd' on 'oel7rac1n1'
122 CRS-2673: Attempting to stop 'ora.mdnsd' on 'oel7rac1n1'
123 CRS-2673: Attempting to stop 'ora.gpnpd' on 'oel7rac1n1'
124 CRS-2677: Stop of 'ora.evmd' on 'oel7rac1n1' succeeded
125 CRS-2677: Stop of 'ora.ctssd' on 'oel7rac1n1' succeeded
126 CRS-2673: Attempting to stop 'ora.storage' on 'oel7rac1n1'
127 CRS-2677: Stop of 'ora.mdnsd' on 'oel7rac1n1' succeeded
128 CRS-2677: Stop of 'ora.gpnpd' on 'oel7rac1n1' succeeded
129 CRS-2677: Stop of 'ora.storage' on 'oel7rac1n1' succeeded
130 CRS-2673: Attempting to stop 'ora.asm' on 'oel7rac1n1'
131 CRS-2677: Stop of 'ora.asm' on 'oel7rac1n1' succeeded
132 CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'oel7rac1n1'
133 CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'oel7rac1n1' succeeded
134 CRS-2673: Attempting to stop 'ora.cssd' on 'oel7rac1n1'
135 CRS-2677: Stop of 'ora.cssd' on 'oel7rac1n1' succeeded
136 CRS-2673: Attempting to stop 'ora.crf' on 'oel7rac1n1'
137 CRS-2677: Stop of 'ora.crf' on 'oel7rac1n1' succeeded
138 CRS-2673: Attempting to stop 'ora.gipcd' on 'oel7rac1n1'
139 CRS-2677: Stop of 'ora.gipcd' on 'oel7rac1n1' succeeded
140 CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'oel
141 7rac1n1' has completed
143 CRS-4123: Starting Oracle High Availability Services-managed resources
158 CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'oel7rac1n1'
159 CRS-2672: Attempting to start 'ora.ctssd' on 'oel7rac1n1'
160 CRS-2676: Start of 'ora.ctssd' on 'oel7rac1n1' succeeded
161 CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'oel7rac1n1' succeeded
162 CRS-2672: Attempting to start 'ora.asm' on 'oel7rac1n1'
163 CRS-2676: Start of 'ora.asm' on 'oel7rac1n1' succeeded
170 CRS-6023: Starting Oracle Cluster Ready Services-managed resources
171 CRS-6017: Processing resource auto-start for servers: oel7rac1n1
172 CRS-6016: Resource auto-start has completed for server oel7rac1n1
173 CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
175 2016/12/02 19:34:26 CLSRSC-343: Successfully started Oracle Clusterware stack
176
177 CRS-2672: Attempting to start 'ora.ASMNET1LSNR_ASM.lsnr' on 'oel7rac1n1'
179 CRS-2676: Start of 'ora.ASMNET1LSNR_ASM.lsnr' on 'oel7rac1n1' succeeded
180 CRS-2676: Start of 'ora.ASMNET2LSNR_ASM.lsnr' on 'oel7rac1n1' succeeded
183 CRS-2672: Attempting to start 'ora.CRS.dg' on 'oel7rac1n1'
184 CRS-2676: Start of 'ora.CRS.dg' on 'oel7rac1n1' succeeded
185 CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources
186 on 'oel7rac1n1'
187 CRS-2673: Attempting to stop 'ora.crsd' on 'oel7rac1n1'
188 CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'oel7r
189 ac1n1'
190 CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'oel7rac1n1'
191 CRS-2673: Attempting to stop 'ora.oc4j' on 'oel7rac1n1'
192 CRS-2673: Attempting to stop 'ora.cvu' on 'oel7rac1n1'
193 CRS-2673: Attempting to stop 'ora.CRS.dg' on 'oel7rac1n1'
196 CRS-2673: Attempting to stop 'ora.oel7rac1n1.vip' on 'oel7rac1n1'
197 CRS-2677: Stop of 'ora.cvu' on 'oel7rac1n1' succeeded
198 CRS-2677: Stop of 'ora.CRS.dg' on 'oel7rac1n1' succeeded
200 CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'oel7rac1n1' succeeded
202 CRS-2673: Attempting to stop 'ora.scan2.vip' on 'oel7rac1n1'
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'oel7rac1n1'
207 CRS-2673: Attempting to stop 'ora.ASMNET2LSNR_ASM.lsnr' on 'oel7rac1n1'
208 CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'oel7rac1n1'
209 CRS-2677: Stop of 'ora.ASMNET2LSNR_ASM.lsnr' on 'oel7rac1n1' succeeded
210 CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'oel7rac1n1' succeeded
211 CRS-2677: Stop of 'ora.oel7rac1n1.vip' on 'oel7rac1n1' succeeded
212 CRS-2677: Stop of 'ora.scan1.vip' on 'oel7rac1n1' succeeded
215 CRS-2677: Stop of 'ora.oc4j' on 'oel7rac1n1' succeeded
216 CRS-2673: Attempting to stop 'ora.ons' on 'oel7rac1n1'
217 CRS-2677: Stop of 'ora.ons' on 'oel7rac1n1' succeeded
218 CRS-2673: Attempting to stop 'ora.net1.network' on 'oel7rac1n1'
219 CRS-2677: Stop of 'ora.net1.network' on 'oel7rac1n1' succeeded
220 CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'oel7rac1n1'
221 has completed
222 CRS-2677: Stop of 'ora.crsd' on 'oel7rac1n1' succeeded
223 CRS-2673: Attempting to stop 'ora.ctssd' on 'oel7rac1n1'
224 CRS-2673: Attempting to stop 'ora.evmd' on 'oel7rac1n1'
225 CRS-2673: Attempting to stop 'ora.storage' on 'oel7rac1n1'
226 CRS-2673: Attempting to stop 'ora.mdnsd' on 'oel7rac1n1'
227 CRS-2673: Attempting to stop 'ora.gpnpd' on 'oel7rac1n1'
228 CRS-2677: Stop of 'ora.storage' on 'oel7rac1n1' succeeded
230 CRS-2677: Stop of 'ora.ctssd' on 'oel7rac1n1' succeeded
231 CRS-2677: Stop of 'ora.evmd' on 'oel7rac1n1' succeeded
232 CRS-2677: Stop of 'ora.mdnsd' on 'oel7rac1n1' succeeded
233 CRS-2677: Stop of 'ora.gpnpd' on 'oel7rac1n1' succeeded
235 CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'oel7rac1n1'
236 CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'oel7rac1n1' succeeded
237 CRS-2673: Attempting to stop 'ora.cssd' on 'oel7rac1n1'
238 CRS-2677: Stop of 'ora.cssd' on 'oel7rac1n1' succeeded
239 CRS-2673: Attempting to stop 'ora.crf' on 'oel7rac1n1'
241 CRS-2673: Attempting to stop 'ora.gipcd' on 'oel7rac1n1'
242 CRS-2677: Stop of 'ora.gipcd' on 'oel7rac1n1' succeeded
243 CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'oel
244 7rac1n1' has completed
273 CRS-6023: Starting Oracle Cluster Ready Services-managed resources
274 CRS-2664: Resource 'ora.CRS.dg' is already running on 'oel7rac1n1'
276 CRS-2672: Attempting to start 'ora.net1.network' on 'oel7rac1n1'
277 CRS-2672: Attempting to start 'ora.oc4j' on 'oel7rac1n1'
278 CRS-2676: Start of 'ora.net1.network' on 'oel7rac1n1' succeeded
279 CRS-2672: Attempting to start 'ora.cvu' on 'oel7rac1n1'
280 CRS-2672: Attempting to start 'ora.oel7rac1n1.vip' on 'oel7rac1n1'
281 CRS-2672: Attempting to start 'ora.ons' on 'oel7rac1n1'
282 CRS-2672: Attempting to start 'ora.scan1.vip' on 'oel7rac1n1'
285 CRS-2676: Start of 'ora.cvu' on 'oel7rac1n1' succeeded
286 CRS-2676: Start of 'ora.oel7rac1n1.vip' on 'oel7rac1n1' succeeded
287 CRS-2676: Start of 'ora.scan1.vip' on 'oel7rac1n1' succeeded
288 CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'oel7rac1n1'
293 CRS-2676: Start of 'ora.ons' on 'oel7rac1n1' succeeded
294 CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'oel7rac1n1' succeeded
297 CRS-2676: Start of 'ora.oc4j' on 'oel7rac1n1' succeeded
298 CRS-6016: Resource auto-start has completed for server oel7rac1n1
299 CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
301 2016/12/02 19:38:37 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster
302 ... succeeded
303
304 # node2 root.sh
305 [root@oel7rac1n2 ~]# /u01/app/12.1.0/grid/root.sh
306 Performing root user operation.
307
308 The following environment variables are set as:
309 ORACLE_OWNER= grid
310 ORACLE_HOME= /u01/app/12.1.0/grid
311
312 Enter the full pathname of the local bin directory: [/usr/local/bin]:
313 Copying dbhome to /usr/local/bin ...
314 Copying oraenv to /usr/local/bin ...
315 Copying coraenv to /usr/local/bin ...
316
317
318 Creating /etc/oratab file...
319 Entries will be added to the /etc/oratab file as needed by
320 Database Configuration Assistant when a database is created
321 Finished running generic part of root script.
322 Now product-specific root actions will be performed.
323 Using configuration parameter file: /u01/app/12.1.0/grid/crs/install/crsconfig_params
324 2016/12/02 20:17:41 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Colle
325 ctor.
326
327 2016/12/02 20:18:12 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer
328 (TFA) Collector.
329
330 2016/12/02 20:18:13 CLSRSC-363: User ignored prerequisites during installation
331
332 OLR initialization - successful
333 2016/12/02 20:19:43 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.se
334 rvice'
334 rvice'
335
370 CRS-2672: Attempting to start 'ora.net1.network' on 'oel7rac1n2'
374 CRS-2676: Start of 'ora.net1.network' on 'oel7rac1n2' succeeded
376 CRS-2672: Attempting to start 'ora.ons' on 'oel7rac1n2'
CRS-2677: Stop of 'ora.scan1.vip' on 'oel7rac1n1' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'oel7rac1n2'
CRS-2676: Start of 'ora.scan1.vip' on 'oel7rac1n2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'oel7rac1n2'
CRS-2676: Start of 'ora.ons' on 'oel7rac1n2' succeeded
CRS-2676: Start of 'ora.ASMNET2LSNR_ASM.lsnr' on 'oel7rac1n2' succeeded
CRS-2676: Start of 'ora.ASMNET1LSNR_ASM.lsnr' on 'oel7rac1n2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'oel7rac1n2'
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'oel7rac1n2' succeeded
CRS-2676: Start of 'ora.asm' on 'oel7rac1n2' succeeded
CRS-6016: Resource auto-start has completed for server oel7rac1n2
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2016/12/02 20:23:53 CLSRSC-343: Successfully started Oracle Clusterware stack
2016/12/02 20:24:22 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster
... succeeded
Installation in progress
Installation finished successfully.
Postinstallation checks
Verify Grid Infrastructure after the installation: cluvfy stage -post crsinst
1 [grid@oel7rac1n1 bin]$ ./cluvfy stage -post crsinst -n oel7rac1n1,oel7rac1n2 -method r

2 oot
3 Enter "ROOT" password:
4
5 Performing post-checks for cluster services setup
6
9
10
13
15
17
19
20 Check: Node connectivity using interfaces on subnet "192.168.1.0"
22 1
24
25
28 n2
30
31
34 n1
36
37
40 n1
42
43
46 n1
48
56
58
60
61 Checking subnet "192.168.20.0" for multicast communication with multicast group "22
62 4.0.0.251"...
63 Check of subnet "192.168.20.0" for multicast communication with multicast group "224
64 .0.0.251" passed.
65
67 4.0.0.251"...
69 .0.0.251" passed.
70
72 4.0.0.251"...
74 .0.0.251" passed.
75
77 4.0.0.251"...
79 .0.0.251" passed.
80
82
83 Checking whether the ASM filter driver is active and consistent on all nodes
84 ASM filter driver library is not installed on any of the cluster nodes.
85 ASM filter driver configuration was found consistent across all the cluster nodes.
ASM filter driver configuration was found consistent across all the cluster nodes.
87
88 Checking Cluster manager integrity...
89
90
91 Checking CSS daemon...
92 Oracle Cluster Synchronization Services appear to be online.
93
94 Cluster manager integrity check passed
95
96
97 UDev attributes check for OCR locations started...
98 UDev attributes check passed for OCR locations
99
100
101 UDev attributes check for Voting Disk locations started...
102 UDev attributes check passed for Voting Disk locations
103
104 Default user file creation mask check passed
105
106 Checking cluster integrity...
107
108
109 Cluster integrity check passed
110 hecking OCR integrity...
111
112 Checking the absence of a non-clustered configuration...
113 All nodes free of non-clustered, local-only configurations
114
115
116 Checking daemon liveness...
117 Liveness check passed for "CRS daemon"
118
119 Checking OCR config file "/etc/oracle/ocr.loc"...
120
121 OCR config file "/etc/oracle/ocr.loc" check successful
122
123
124 Disk group for ocr location "+CRS/rac1/OCRFILE/registry.255.929561455" is available
125 on all the nodes
126
127 Uniqueness check for OCR device passed
128
129 Checking the version of OCR...
130 OCR of correct Version "4" exists
131
132 Checking data integrity of OCR...
133 Data integrity check for OCR passed
134
135 OCR integrity check passed
136
137 Checking CRS integrity...
138
139 Clusterware version consistency passed.
140
141 CRS integrity check passed
142
143 Checking node application existence...
144
145 Checking existence of VIP node application (required)
146 VIP node application check passed
147
148 Checking existence of NETWORK node application (required)
149 NETWORK node application check passed
150
151 Checking existence of ONS node application (optional)
151 Checking existence of ONS node application (optional)
152 ONS node application check passed
153
154
155 Checking Single Client Access Name (SCAN)...
156
157 Checking TCP connectivity to SCAN listeners...
158 TCP connectivity to SCAN listeners exists on all cluster nodes
159
160 Checking name resolution setup for "rac1-scan.dba24.pl"...
161 Checking integrity of name service switch configuration file "/etc/nsswitch.conf" ...
162 All nodes have same "hosts" entry defined in file "/etc/nsswitch.conf"
163 Check for integrity of name service switch configuration file "/etc/nsswitch.conf" passe
164 d
165
166
167 Checking SCAN IP addresses...
168 Check of SCAN IP addresses passed
169
170 Verification of SCAN VIP and listener setup passed
171
172 Checking OLR integrity...
173 Check of existence of OLR configuration file "/etc/oracle/olr.loc" passed
174 Check of attributes of OLR configuration file "/etc/oracle/olr.loc" passed
175
176 WARNING:
177 This check does not verify the integrity of the OLR contents. Execute 'ocrcheck -local'
178 as a privileged user to verify the contents of OLR.
179
180 OLR integrity check passed
181
182 Checking Oracle Cluster Voting Disk configuration...
183
184 Oracle Cluster Voting Disk configuration check passed
185 Task ASM Integrity check started...
186
187 Checking if connectivity exists across cluster nodes on the ASM network
188
190
192
194
197 n2
199
200
203 n2
205
210
212
213 Network connectivity check across cluster nodes on the ASM network passed
214 Starting check to see if ASM is running on all cluster nodes...
215
216 ASM Running check passed. ASM is running on all specified nodes
217 Disk Group Check passed. At least one Disk Group configured
217 Disk Group Check passed. At least one Disk Group configured
218
Task ASM Integrity check passed...
User "grid" is not part of "root" group. Check passed
Oracle Clusterware is installed on all nodes.
CTSS resource check passed
Query of CTSS for time offset passed
CTSS is in Active state. Proceeding with check of clock time offsets on all nodes...
Check of clock time offsets passed
Oracle Cluster Time Synchronization Services check passed

Checking VIP configuration.
Checking VIP Subnet configuration.
Check for VIP Subnet configuration passed.
Checking VIP reachability
Check for VIP reachability passed.
Post-check for cluster services setup was successful.
All checks passed, seems that GI has been installed properly!!
Modify CRS, add DATA, FRA diskgroups
Run asmca as grid
Try to add op3-rac1-asm-crs-03 disk as Quorum failgroup

It complains again. Wow! Bug or a feature?
Replace temporary disk with crs-03 using sqlplus
1 # first let's see votes

2 [grid@oel7rac1n1 ~]$ /u01/app/12.1.0/grid/bin/crsctl query css votedisk
4 -- ----- ----------------- --------- ---------
5 1. ONLINE 1c193c21750c4fe3bfa28c6b5df261b3 (/dev/asm/op2-rac1-asm-crs-02) [CR
6 S]
7 2. ONLINE 54cd207d0d744fd8bf2714d7776f5ad2 (/dev/asm/op1-rac1-asm-crs-01) [CR
S]
3. ONLINE 355efd32461e4f3fbf50305bfac1657e (/dev/asm/op3-rac1-asm-tmp) [CRS]
Add op3-rac1-asm-crs-03 to the CRS diskgroup as quorum disk
1 [grid@oel7rac1n1 bin]$ ./sqlplus / as sysasm

2
3 SQL> alter diskgroup crs add QUORUM FAILGROUP fg3 DISK '/dev/asm/op3-rac1-asm-
4 crs-03';
5
Diskgroup altered.
And what happened? nothing, vote stil on tmp

3 -- ----- ----------------- --------- ---------
5 S]
S]
3. ONLINE 355efd32461e4f3fbf50305bfac1657e (/dev/asm/op3-rac1-asm-tmp) [CRS]
Remove temporary disk
1 SQL> alter diskgroup crs drop disk CRS_0002;

2 Diskgroup altered.
Result?

3 -- ----- ----------------- --------- ---------
5 S]
7 S]
3. ONLINE 0588f4e97cbb4fb8bfb28fd5d18a46d0 (/dev/asm/op3-rac1-asm-crs-03) [CR
S]
Located 3 voting disk(s).
This is what we want!

Create DATA diskgroup. Set failgroup op1 to disks that come from openfiler1 and failgroup
op2 to disks that come from openfiler2. Set AU_SIZE to 4GB
Create FRA diskgroup. Set failgroup op1 to disks that come from openfiler1 and failgroup
op2 to disks that come from openfiler2. Set AU_SIZE to 1GB
Diskgroups successfully created.
Grid Infrastructure installed successfully!
Alternate approach to ASM diskgroups manipulation using ascmca -silent, chdg,

mkdg
I used asmca or sqlplus to create and modify the diskgroup, but there are some other ways.
Please find examples below.
ASMCA – silent
As you can read in the doc https://docs.oracle.com/database/121/OSTMG/GUID-877EB0F8-

E9CA-4C97-965C-AACBB256B12D.htm#OSTMG94309
asmca silent allows only this operations:
-convertToFlexASM
-configureASM
-upgradeASM
-configureParameter
-deleteASM
-createDiskGroup
-addDisk
-createVolume
-createACFS
Let’s add the crs-03 quorum disk to the crs as we did with sqlplus
1 [grid@oel7rac1n1 bin]$ ./asmca -silent -addDisk -diskGroupName crs -diskList '/dev/as
2 m/op3-rac1-asm-crs-03' -failuregroup fg3 -quorum
3
4 Disks added successfully to disk group crs
5
6 [grid@oel7rac1n1 bin]$ ./asmcmd -p
7 ASMCMD [+] > lsdsk -k
8 Total_MB Free_MB OS_MB Name Failgroup Failgroup_Type Library Label UDI
9 D Product Redund Path
10 10014 6965 10014 CRS_0001 CRS_0001 REGULAR System
11 UNKNOWN /dev/asm/op1-rac1-asm-crs-01
14 510 507 510 CRS_0003 FG3 QUORUM System UNK
15 NOWN /dev/asm/op3-rac1-asm-crs-03
17 UNKNOWN /dev/asm/op3-rac1-asm-tmp
18 ASMCMD [+] > exit
19
20 # check how votes look like
[grid@oel7rac1n1 bin]$ ./crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 1c193c21750c4fe3bfa28c6b5df261b3 (/dev/asm/op2-rac1-asm-crs-02) [C
RS]
2. ONLINE 54cd207d0d744fd8bf2714d7776f5ad2 (/dev/asm/op1-rac1-asm-crs-01) [C
RS]
3. ONLINE a25677e1db034f6cbf5b2f88daf9d4c7 (/dev/asm/op3-rac1-asm-tmp) [CRS
]
As we cannot use asmca silent mode to drop disk from diskgroup let’s use chdg command in
ASMCMD. First prepare xml file
1 # so lets use chdg to drop /dev/asm/op3-rac1-asm-tmp

2 # create file /home/grid/change_crs.xml
3 <chdg name="crs" power="3">
4 <drop>
5 <dsk name="CRS_0002"></dsk>
6 </drop>
7 </chdg>
Run chdg
1 ASMCMD [+] > chdg /home/grid/change_crs.xml
2 Diskgroup altered.
3
4 ASMCMD [+] > lsdsk -k
11 510 475 510 CRS_0003 FG3 QUORUM System UNK
12 NOWN /dev/asm/op3-rac1-asm-crs-03
13
14 [grid@oel7rac1n1 bin]$ ./crsctl query css votedisk
16 -- ----- ----------------- --------- ---------
1. ONLINE 1c193c21750c4fe3bfa28c6b5df261b3 (/dev/asm/op2-rac1-asm-crs-02) [C
RS]
2. ONLINE 54cd207d0d744fd8bf2714d7776f5ad2 (/dev/asm/op1-rac1-asm-crs-01) [C
RS]
3. ONLINE 3d51b07f8a6e4f2abf8f7b4a506f67b6 (/dev/asm/op3-rac1-asm-crs-03) [C
RS]
Located 3 voting disk(s).
That’s it!!
Use ASMCA silent mode to create DATA and FRA
1 [grid@oel7rac1n1 bin]$ ./asmca -silent -createDiskGroup -diskGroupName DATA -diskLi

2 st /dev/asm/op1-rac1-asm-data-01,/dev/asm/op1-rac1-asm-data-02,/dev/asm/op1-rac1-a
3 sm-data-03,/dev/asm/op1-rac1-asm-data-04 -failuregroup op1 -diskList /dev/asm/op2-rac
4 1-asm-data-01,/dev/asm/op2-rac1-asm-data-02,/dev/asm/op2-rac1-asm-data-03,/dev/as
5 m/op2-rac1-asm-data-04 -failuregroup op2 -redundancy NORMAL -au_size 1 -compatibl
6 e.asm 12.1 -compatible.rdbms 11.2
7
Disk Group DATA created successfully.
[grid@oel7rac1n1 bin]$ ./asmca -silent -createDiskGroup -diskGroupName FRA -diskList
/dev/asm/op1-rac1-asm-fra-01,/dev/asm/op1-rac1-asm-fra-02,/dev/asm/op1-rac1-asm-fr
a-03,/dev/asm/op1-rac1-asm-fra-04 -failuregroup op1 -diskList /dev/asm/op2-rac1-asm-fr
a-01,/dev/asm/op2-rac1-asm-fra-02,/dev/asm/op2-rac1-asm-fra-03,/dev/asm/op2-rac1-as
m-fra-04 -failuregroup op2 -redundancy NORMAL -au_size 1 -compatible.asm 12.1 -com
patible.rdbms 11.2
Disk Group FRA created successfully.
Check new dgs
1 ASMCMD [+] > lsdsk -k -G DATA

4 2014 1987 2014 DATA_0000 OP1 REGULAR System
5 UNKNOWN /dev/asm/op1-rac1-asm-data-01
20 ASMCMD [+] > lsdsk -k -G FRA
23 1022 997 1022 FRA_0000 OP1 REGULAR System UN
24 KNOWN /dev/asm/op1-rac1-asm-fra-01
39 ASMCMD [+] > lsattr -l -G DATA
40 Name Value
41 access_control.enabled FALSE
42 access_control.umask 066
43 au_size 1048576
44 cell.smart_scan_capable FALSE
45 compatible.asm 12.1.0.0.0
46 compatible.rdbms 11.2.0.0.0
47 content.check FALSE
48 content.type data
49 disk_repair_time 3.6h
50 failgroup_repair_time 24.0h
51 idp.boundary auto
52 idp.type dynamic
53 phys_meta_replicated true
54 sector_size 512
55 thin_provisioned FALSE
ASMCMD [+] > lsattr -l -G FRA
Name Value
access_control.enabled FALSE
access_control.umask 066
au_size 1048576
cell.smart_scan_capable FALSE
compatible.asm 12.1.0.0.0
compatible.rdbms 11.2.0.0.0
content.check FALSE
content.type data
disk_repair_time 3.6h
failgroup_repair_time 24.0h
idp.boundary auto
idp.type dynamic
phys_meta_replicated true
sector_size 512
thin_provisioned FALSE
Use sqlplus to create DATA and FRA

1 SQL> create diskgroup DATA NORMAL REDUNDANCY failgroup op1 disk '/dev/asm/op
2 1-rac1-asm-data-01','/dev/asm/op1-rac1-asm-data-02','/dev/asm/op1-rac1-asm-data-03','
3 /dev/asm/op1-rac1-asm-data-04' failgroup op2 disk '/dev/asm/op2-rac1-asm-data-01','/de
4 v/asm/op2-rac1-asm-data-02','/dev/asm/op2-rac1-asm-data-03','/dev/asm/op2-rac1-asm-
5 data-04' ATTRIBUTE 'compatible.asm'='12.1','compatible.rdbms'='11.2','au_size'='1M';
6
7 Diskgroup created.
SQL> create diskgroup FRA NORMAL REDUNDANCY failgroup op1 disk '/dev/asm/op1-
rac1-asm-fra-01','/dev/asm/op1-rac1-asm-fra-02','/dev/asm/op1-rac1-asm-fra-03','/dev/as
m/op1-rac1-asm-fra-04' failgroup op2 disk '/dev/asm/op2-rac1-asm-fra-01','/dev/asm/op2
-rac1-asm-fra-02','/dev/asm/op2-rac1-asm-fra-03','/dev/asm/op2-rac1-asm-fra-04' ATTRI
BUTE 'compatible.asm'='12.1','compatible.rdbms'='11.2','au_size'='1M';
Diskgroup created.
Check results
1 SMCMD [+] > lsdsk -k -G DATA

20 ASMCMD [+] > lsdsk -k -G FRA
Total_MB Free_MB OS_MB Name Failgroup Failgroup_Type Library Label UDI
D Product Redund Path
1022 997 1022 FRA_0000 OP1 REGULAR System UN
KNOWN /dev/asm/op1-rac1-asm-fra-01
Another way to do it is to build the diskgroups with mkdg command
Create file /home/grid/mkdg_data.xml

1 # /home/grid/mkdg_data.xml
2
3 <dg name="data" redundancy="normal">
4 <fg name="op1">
5 <dsk string="/dev/asm/op1-rac1-asm-data-01"></dsk>
9 </fg>
10 <fg name="op2">
15 </fg>
16 <a name="compatible.asm" value="12.1"></a>
17 <a name="compatible.rdbms" value="11.2"></a>
18 <a name="au_size" value="4M"></a>
19 </dg>
Create file /home/grid/mkdg_fra.xml
1 #/home/grid/mkdg_fra.xml
2 <dg name="fra" redundancy="normal">
3 <fg name="op1">
4 <dsk string="/dev/asm/op1-rac1-asm-fra-01"></dsk>
8 </fg>
9 <fg name="op2">
14 </fg>
15 <a name="compatible.asm" value="12.1"></a>
16 <a name="compatible.rdbms" value="11.2"></a>
17 <a name="au_size" value="1M"></a>
18 </dg>
Create diskgroups using mkdg
1 ASMCMD [+] > mkdg /home/grid/mkdg_data.xml

2 ASMCMD [+] > mkdg /home/grid/mkdg_fra.xml
Check results
6 2012 1968 2014 DATA_0001 DATA_0001 REGULAR System
20
Total_MB Free_MB OS_MB Name Failgroup Failgroup_Type Library Label UDI
D Product Redund Path
2012 1980 2014 DATA_0000 OP1 REGULAR System
UNKNOWN /dev/asm/op1-rac1-asm-data-01
2012 1968 2014 DATA_0001 DATA_0001 REGULAR System
2012 1964 2014 DATA_0004 OP2 REGULAR System
Despite proper configuration in the xml files disks did not get their proper failgroup
assignment. I will try out this approach later again – after patching with latest PSU.
What do you think?? Please post your comments

Source: My experience and WorlWideWeb
About the author

Maciej Tokar
An Oracle technology geek and crazy long distance runner, DBA24 Owner
Senior Oracle DBA / Consultant / [OCP10g, OCP12c, OCE RAC 10g] / [experience: 9y+]
Currently working for Bluegarden (Oslo Norway) by Miratech Group
Past: Mastercard / Trevica by Britenet, Citi International PLC, PZU

RAC Lab Part 10 - Installing Grid Infrastructure

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

RAC Lab Part 10 - Installing Grid Infrastructure

Uploaded by

Copyright:

Available Formats

U a

RAC lab Part 10 – Installing Grid

Please refer to the official 12.1 documentation and CLUVFY documentation:

Grid Infrastructure Installation Guide

Prepare installation files for grid and database software

Prepare directory for shared dir

1 maciek@macieksrv.dba24.pl /home/maciek/Downloads # sudo mkdir -p /opt/vboxshared/

Move files downloaded from Oracle to that directory

1 maciek@macieksrv.dba24.pl /home/maciek/Downloads # sudo mv /home/maciek/Downl

Set this folder as shared with the vms

1 vboxuser1@macieksrv.dba24.pl ~ $ vboxmanage sharedfolder add oel7rac1n1 --name S

Unpack software first on the first node oel72rac1n1

Configure SSH equivalency

Distribute public key among nodes

Cluster verification before GI installation – CLUVFY

1 # Check disks for OCR

Shared storage check was successful on nodes "oel7rac1n1,oel7rac1n2"

Shared storage verified successfully.

Network verification using: cluvfy stage -post hwos

1 grid@oel7rac1n1 bin]$ ./cluvfy stage -post hwos -n oel7rac1n1,oel7rac1n2

Shared storage check failed on nodes "oel7rac1n1,oel7rac1n2"

Verification successful (ingore storage failed checks)

Verify storage using: cluvfy comp ssa

1 # Check disks for OCR

Shared storage check was successful on nodes "oel7rac1n1,oel7rac1n2"

Shared storage verified successfully.

Nodes connectivity verification: cluvfy comp nodecon

1 [grid@oel7rac1n1 bin]$ ./cluvfy comp nodecon -n oel7rac1n1,oel7rac1n2

Node connectivity verification successful

1 [grid@oel7rac1n1 ~]$ cd /u01/app/grid/cluvfy/bin/

Checking daemon "avahi-daemon" is not configured and running

OK!! we are ready to install the cluster software

OUI Grid Installation

Run OUI installer

1 [grid@oel7rac1n1 ~]$ cd /u01/app/grid/grid-software/grid/

Choose to install and configure GI for a cluster

Choose preferred languages to install

Verify ssh user equivalency

Declare cluster networks

Create crs diskgroup using /dev/asm/*asm*crs* disks

Some interesting notes:

Create new volume (click to enlarge)

Check volumes now (click to enlarge)

Check the new lun on openfiler3 system level

1 [root@openfiler3 ~]# cat /proc/net/iet/volume

Discover disk on the cluster nodes

1 [root@oel7rac1n1 ~]# rescan-scsi-bus.sh

Find device name assigned to the new disk

/dev/sdu – new disk device name on both nodes

Create partition on /dev/sdu on the first node

Discover new partition on the second node

1 [root@oel7rac1n2 ~]# kpartx -a /dev/sdu

1 [root@oel7rac1n1 ~]# /usr/lib/udev/scsi_id --whitelisted --replace-whitespace /dev/sdu

Add the following line to /etc/udev/rules.d/90-oracle-asmdevices.rules on both nodes.

1 KERNEL=="sd*1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d $

Reload udev and check asm disk (owner,group,privs)

1 [root@oel7rac1n1 ~]#udevadm control --reload-rules

Create CRS diskgroup using temporary disk from openfiler3

Set passwords for ASM

Set asm groups – leave default

We wan to run root scripts manually

It complains about /dev/shm size

Confirm you want to ignore this warning

Create crs diskgroup using /dev/asm/asmcrs* disks