环境：centos7.9
openfiler
oracle11.2.0.4
-
1IP规划
Hostname
Host-alias
type
ip
interface
rac1
rac1
Public ip
10.8.98.103
Eth0
rac1-vip
rac1-vip
Virtual ip
10.8.98.105
Eth0:1
rac1-priv
rac1-priv
Private ip
192.168.1.103
Eth1
rac2
rac2
Public ip
10.8.98.104
Eth0
rac2-vip
rac2-vip
Virtual ip
10.8.98.106
Eth0:1
rac2-priv
rac2-priv
Private ip
192.168.1.104
Eth1
cluster-scan
cluster-scan
Scan ip
10.8.98.107
Eth0
openfiler
openfiler
10.8.98.100
Eth0
10.8.98.101
Eth1
俩节点hosts文件
[root@rac1 ~]# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 #public ip eth0 10.8.98.103 rac1 10.8.98.104 rac2 #priv ip eth1 192.168.1.103 rac1-priv 192.168.1.104 rac2-priv #vip ip 10.8.98.105 rac1-vip 10.8.98.106 rac2-vip #scan ip 10.8.98.107 cluster-scan
1.2 目录规划
|
Grid infrastructure |
Oracle database |
|
HOME=/home/oracle |
HOME=/home/oracle |
|
ORACLE_BASE=/u01/app/grid |
ORACLE_BASE=/u01/app/oracle |
|
ORACLE_HOME=/u01/app/11.2.0/grid |
ORACLE_HOME=/u01/app/oracle/product/11.2.0/db_1 |
1.3 ORACLE SID
|
DATABASE SID |
prod |
|
Node 1 instance sid |
prod1 |
|
Node 2 instance sid |
prod2 |
1.4 存储规划
存储LUN （分区）规划原则（生产环境）
1. 数据文件和归档文件底层属于不同的raid 组，属于不通的DG( 磁盘组)
2. 在线日志文件同一组中不同成员分别存储在两个DG 中
3. 控制文件不同成员分别存储在两个DG 中
4. Ocr 和votedisk 存放在数据之外单独的ASM 磁盘组
ASM 磁盘划分
|
Rac 节点 |
Lun name |
Asm 磁盘名称/alias |
Size |
ASM 磁盘组名称 |
冗余类型 |
|
Rac1 |
|
asmocrvote1 |
2G |
SYSTEMDG |
High/normal |
|
|
asmocrvote2 |
2G | |||
|
|
asmocrvote3 |
2G | |||
|
|
asmdata01 |
200G |
DATADG1 |
external | |
|
|
asmdata02 |
200G | |||
|
|
Asmarch01 |
200G |
FRADG |
external | |
|
|
|
|
|
|
|
|
Rac2 |
|
asmocrvote1 |
2G |
SYSTEMDG |
High/normal |
|
|
asmocrvote2 |
2G | |||
|
|
asmocrvote3 |
2G | |||
|
|
asmdata01 |
200G |
DATADG1 |
external | |
|
|
asmdata02 |
200G | |||
|
|
Asmarch01 |
200G |
FRADG |
external |
此处我做实验，用的空间为1G 1G 1G 20G 20G 10G
openfile的搭建参考 http://blog.itpub.net/70004783/viewspace-2791487/
本地yum源的搭建参考 http://blog.itpub.net/70004783/viewspace-2789438/
安装需要的软件列表
yum install binutils -y yum install compat-libcap1 -y yum install compat-libstdc -y yum install gcc -y yum install gcc-c++ -y yum install glibc -y yum install glibc-devel -y yum install ksh -y yum install libgcc -y yum install libstdc++ -y yum install libstdc++-devel -y yum install libaio -y yum install libaio-devel -y yum install libXext -y yum install libXtst -y yum install libX11 -y yum install libXau -y yum install libxcb -y yum install libXi -y yum install make -y yum install sysstat -y yum install unixODBC -y yum install unixODBC-devel –y yum install elfutils-libelf-devel* -y yum install compat-libcap* -y yum install libaio-devel* -y yum install compat-libstdc++-* -y
2.1 创建用户组，用户和目录
在两个节点增加用户与组：
/usr/sbin/groupadd -g 501 oinstall /usr/sbin/groupadd -g 502 dba /usr/sbin/groupadd -g 503 oper /usr/sbin/groupadd -g 504 asmadmin /usr/sbin/groupadd -g 505 asmoper /usr/sbin/groupadd -g 506 asmdba /usr/sbin/useradd -g oinstall -G dba,asmdba,oper oracle /usr/sbin/useradd -g oinstall -G asmadmin,asmdba,asmoper,oper,dba grid
修改用户密码：
passwd grid
passwd oracle
登陆测试（必须做），分别用oracle,grid用户通过ssh登录，按提示修改密码
目录
mkdir -p /u01/app/11.2.0/grid mkdir -p /u01/app/grid mkdir -p /u01/app/oraInventory chown -R grid:oinstall /u01 mkdir -p /u01/app/oracle chown oracle:oinstall /u01/app/oracle chmod -R 775 /u01
2.修改操作系统参数2节点
修改/etc/security/limits.conf文件，增加内容如下：
vi /etc/security/limits.conf
#ORACLE SETTING grid soft nproc 2047 grid hard nproc 16384 grid soft nofile 1024 grid hard nofile 65536 grid soft stack 10240 grid hard stack 32768 oracle soft nproc 2047 oracle hard nproc 16384 oracle soft nofile 1024 oracle hard nofile 65536 oracle soft stack 10240 oracle hard stack 32768
3.禁用SELINUX
vi /etc/selinux/config 修改下面
SELINUX=disabled
让SELINUX生效：
setenforce 0
4.关闭防火墙
systemctl stop firewalld systemctl disable firewalld
5、修改/etc/pam.d/login文件
vi /etc/pam.d/login 添加如下
#ORACLE SETTING session required pam_limits.so
6、修改sysctl.conf配置文件
vi /etc/sysctl.conf 添加如下
#ORACLE SETTING fs.aio-max-nr = 1048576 fs.file-max = 6815744 kernel.shmmax = 2147483648 kernel.shmall = 524288 kernel.shmmni = 4096 kernel.sem = 250 32000 100 128 net.ipv4.ip_local_port_range = 9000 65500 net.core.rmem_default = 262144 net.core.rmem_max = 4194304 net.core.wmem_default = 262144 net.core.wmem_max = 1048586
#kernel.shmmax = 2147483648 (大于SGA，小于物理内存)
#kernel.shmall = 524288 （物理内存/4K）
如果以上参数设置不正确，可能导致以下报错：
ORA-27102: out of memory Linux-X86_64 Error: 28: No space left on device .
让配置生效：
sysctl -p
7.禁止NTP
8.配置grid和oracle用户的环境变量
节点1 配置grid
[grid@rac1:/home/grid]$cat .bash_profile # .bash_profile # Get the aliases and functions if [ -f ~/.bashrc ]; then . ~/.bashrc fi # User specific environment and startup programs PATH=$PATH:$HOME/.local/bin:$HOME/bin export PATH PS1="[`whoami`@`hostname`:"'$PWD]$' export PS1 umask 022 alias sqlplus="rlwrap sqlplus" export TMP=/tmp export export TMPDIR=$TMP export ORACLE_HOSTNAME=rac1 ORACLE_SID=+ASM1; export ORACLE_SID ORACLE_TERM=xterm; export ORACLE_TERM ORACLE_BASE=/u01/app/grid; export ORACLE_BASE ORACLE_HOME=/u01/app/11.2.0/grid; export ORACLE_HOME NLS_DATE_FORMAT="yyyy-mm-dd HH24:MI:SS"; export NLS_DATE_FORMAT PATH=.:$PATH:$HOME/bin:$ORACLE_HOME/bin; export PATH THREADS_FLAG=native; export THREADS_FLAG if [ $USER = "oracle" ] || [ $USER = "grid" ]; then if [ $SHELL = "/bin/ksh" ]; then ulimit -p 16384 ulimit -n 65536 else ulimit -u 16384 -n 65536 fi umask 022 fi [grid@rac1:/home/grid]$
节点2 一样，只需要更改以下2行
export ORACLE_HOSTNAME=rac2 ORACLE_SID=+ASM2; export ORACLE_SID
节点1配置oracle
[oracle@rac1:/home/oracle]$cat .bash_profile # .bash_profile # Get the aliases and functions if [ -f ~/.bashrc ]; then . ~/.bashrc fi # User specific environment and startup programs PATH=$PATH:$HOME/.local/bin:$HOME/bin export PATH PS1="[`whoami`@`hostname`:"'$PWD]$' alias sqlplus="rlwrap sqlplus" alias rman="rlwrap rman" export PS1 export TMP=/tmp export export TMPDIR=$TMP export ORACLE_HOSTNAME=rac1 export ORACLE_UNQNAME=prod ORACLE_BASE=/u01/app/oracle; export ORACLE_BASE ORACLE_HOME=$ORACLE_BASE/product/11.2.0/db_1; export ORACLE_HOME ORACLE_SID=prod1; export ORACLE_SID ORACLE_TERM=xterm; export ORACLE_TERM NLS_DATE_FORMAT="yyyy-mm-dd HH24:MI:SS"; export NLS_DATE_FORMAT NLS_LANG=AMERICAN_AMERICA.ZHS16GBK;export NLS_LANG PATH=.:$PATH:$HOME/bin:$ORACLE_BASE/product/11.2.0/db_1/bin:$ORACLE_HOME/bin; export PATH THREADS_FLAG=native; export THREADS_FLAG if [ $USER = "oracle" ] || [ $USER = "grid" ]; then if [ $SHELL = "/bin/ksh" ]; then ulimit -p 16384 ulimit -n 65536 else ulimit -u 16384 -n 65536 fi umask 022 fi
节点2一样，需要更改2行
export ORACLE_HOSTNAME=rac2 ORACLE_SID=prod2; export ORACLE_SID
9. 安装配置iscsi
yum install iscsi-initiator-utils* -y 检查是否开机启动 systemctl list-unit-files | grep iscsi 如果不是,则这样设置 systemctl enable iscsid.service systemctl enable iscsi.service
启动iscsi服务
systemctl restart iscsi systemctl restart iscsid 发现存储上的iscsi目标 iscsiadm -m discovery -t st -p 10.8.98.100:3260 登陆目标 iscsiadm -m node --loginall=all
察看iscsi信息，是不是2条链路
iscsiadm -m session
[root@rac1 ntp]# iscsiadm -m session
tcp: [1] 10.8.98.100:3260,1 iqn.2006-01.com.openfiler:tsn.e7c9f1f8f2df (non-flash)
tcp: [2] 10.8.98.101:3260,1 iqn.2006-01.com.openfiler:tsn.e7c9f1f8f2df (non-flash)
ls -l /dev/disk/by-path/
因为我们是手动发现的iscsi，所以如果重启则会消失，所以将iscsi目标写入iscsi配置文件中
vi /etc/iscsi/initiatorname.iscsi
InitiatorName=iqn.2006-01.com.openfiler:tsn.e7c9f1f8f2df
重启iscsi服务
systemctl restart iscsi systemctl restart iscsid
刷新新增加iscsi 存储目标
iscsiadm -m session -R
察看iscsi通道
cat /proc/scsi/scsi
删除/注销target连接
iscsiadm -m node --logoutall=all
10.安装多路径
yum install device-mapper*
检查device-mapper有没有安装
rpm -qa | grep device-mapper
安装未安装的rpm包
设置maltipath开机启动
[root@rac1 ntp]# systemctl list-unit-files | grep multipath
multipathd.service enabled
把multipath模块加入内核
modprobe dm-multipath
modprobe dm-round-robin
/sbin/mpathconf -help
/sbin/mpathconf --enable
启动服务
systemctl restart multipathd
配置multipath -ll
[root@rac1 ~]# multipath -ll mapthf (14f504e46494c45524153744b37302d556f616c2d4f746342) dm-4 OPNFILER,VIRTUAL-DISK size=10G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active |- 3:0:0:5 sdg 8:96 active ready running `- 4:0:0:5 sdm 8:192 active ready running mapthe (14f504e46494c455279736a6244732d756565502d6755736d) dm-5 OPNFILER,VIRTUAL-DISK size=20G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active |- 3:0:0:4 sdf 8:80 active ready running `- 4:0:0:4 sdl 8:176 active ready running mapthc (14f504e46494c45524e335577524e2d534a57432d32627371) dm-6 OPNFILER,VIRTUAL-DISK size=1.0G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active |- 3:0:0:2 sdd 8:48 active ready running `- 4:0:0:2 sdj 8:144 active ready running mapthd (14f504e46494c45524d6b6f46624a2d634450442d30484c4a) dm-3 OPNFILER,VIRTUAL-DISK size=20G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active |- 3:0:0:3 sde 8:64 active ready running `- 4:0:0:3 sdk 8:160 active ready running mapthb (14f504e46494c45526a4158746b782d584b59772d6c4f4f76) dm-2 OPNFILER,VIRTUAL-DISK size=1.0G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active |- 3:0:0:1 sdc 8:32 active ready running `- 4:0:0:1 sdi 8:128 active ready running maptha (14f504e46494c45526776575844302d74637a682d51784646) dm-7 OPNFILER,VIRTUAL-DISK size=1.0G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active |- 3:0:0:0 sdb 8:16 active ready running `- 4:0:0:0 sdh 8:112 active ready running
根据上面的磁盘大小以及wwid号
统计以及规划如下
/14f504e46494c45526776575844302d74637a682d51784646/ 1G asmvote01 /14f504e46494c45526a4158746b782d584b59772d6c4f4f76/ 1G asmvote02 /14f504e46494c45524e335577524e2d534a57432d32627371/ 1G asmvote03 /14f504e46494c45524d6b6f46624a2d634450442d30484c4a/ 20 asmdata01 /14f504e46494c45524153744b37302d556f616c2d4f746342/ 20 asmarch01 /14f504e46494c455279736a6244732d756565502d6755736d/ 10 asmdata02
还有两个命令可以查找uuid
multipath -v3
more /etc/multipath/wwids
根据以上得到的资料修改多路径的配置文件
备份配置文件模板
[root@rac1 ~]# cp /etc/multipath.conf /etc/multipath.conf.bak2021
根据上面的规划，改后的配置文件如下
cat /etc/multipath.conf
defaults {
user_friendly_names yes
find_multipaths yes
}
blacklist {
devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
devnode "^hd[a-z]"
#我这里sda是本地硬盘，所以加黑名单
devnode "^sda"
}
multipaths {
multipath {
wwid 14f504e46494c45526776575844302d74637a682d51784646
alias asmvote01
path_grouping_policy multibus
path_selector "round-robin 0"
failback immediate
}
multipath {
wwid 14f504e46494c45526a4158746b782d584b59772d6c4f4f76
alias asmvote02
path_grouping_policy multibus
path_selector "round-robin 0"
failback immediate
}
multipath {
wwid 14f504e46494c45524e335577524e2d534a57432d32627371
alias asmvote03
path_grouping_policy multibus
path_selector "round-robin 0"
failback immediate
}
multipath {
wwid 14f504e46494c45524d6b6f46624a2d634450442d30484c4a
alias asmdata01
path_grouping_policy multibus
path_selector "round-robin 0"
failback immediate
}
multipath {
wwid 14f504e46494c455279736a6244732d756565502d6755736d
alias asmdata02
path_grouping_policy multibus
path_selector "round-robin 0"
failback immediate
}
multipath {
wwid 14f504e46494c45524153744b37302d556f616c2d4f746342
alias asmarch01
path_grouping_policy multibus
path_selector "round-robin 0"
failback immediate
}
}
devices {
device {
vendor "openfiler "
product "virtual disk"
path_grouping_policy multibus
path_checker readsector0
path_selector "round-robin 0"
hardware_handler "0"
}
}
节点2同理
删除现有多路径
multipath -F
格式化多路径
multipath -v2
俩节点
multipath -ll
[root@rac1 ~]# multipath -ll asmarch01 (14f504e46494c45524153744b37302d556f616c2d4f746342) dm-4 OPNFILER,VIRTUAL-DISK size=10G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active |- 3:0:0:5 sdg 8:96 active ready running `- 4:0:0:5 sdm 8:192 active ready running asmdata02 (14f504e46494c455279736a6244732d756565502d6755736d) dm-5 OPNFILER,VIRTUAL-DISK size=20G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active |- 3:0:0:4 sdf 8:80 active ready running `- 4:0:0:4 sdl 8:176 active ready running asmvote03 (14f504e46494c45524e335577524e2d534a57432d32627371) dm-6 OPNFILER,VIRTUAL-DISK size=1.0G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active |- 3:0:0:2 sdd 8:48 active ready running `- 4:0:0:2 sdj 8:144 active ready running asmdata01 (14f504e46494c45524d6b6f46624a2d634450442d30484c4a) dm-3 OPNFILER,VIRTUAL-DISK size=20G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active |- 3:0:0:3 sde 8:64 active ready running `- 4:0:0:3 sdk 8:160 active ready running asmvote02 (14f504e46494c45526a4158746b782d584b59772d6c4f4f76) dm-2 OPNFILER,VIRTUAL-DISK size=1.0G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active |- 3:0:0:1 sdc 8:32 active ready running `- 4:0:0:1 sdi 8:128 active ready running asmvote01 (14f504e46494c45526776575844302d74637a682d51784646) dm-7 OPNFILER,VIRTUAL-DISK size=1.0G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active |- 3:0:0:0 sdb 8:16 active ready running `- 4:0:0:0 sdh 8:112 active ready running [root@rac1 ~]#
11.配置udev
cd /usr/share/doc/device-mapper-1.02.170/
cp 12-dm-permissions.rules /etc/udev/rules.d/
[root@rac1 ~]# vi /etc/udev/rules.d/12-dm-permissions.rules
在最后添加如下
# OWNER:="root", GROUP:="root", MODE:="660"
LABEL="dm_end"
ENV{DM_NAME}=="asmvote01", OWNER:="grid", GROUP:="asmadmin", MODE:="660"
ENV{DM_NAME}=="asmvote02", OWNER:="grid", GROUP:="asmadmin", MODE:="660"
ENV{DM_NAME}=="asmvote03", OWNER:="grid", GROUP:="asmadmin", MODE:="660"
ENV{DM_NAME}=="asmdata01", OWNER:="grid", GROUP:="asmadmin", MODE:="660"
ENV{DM_NAME}=="asmdata02", OWNER:="grid", GROUP:="asmadmin", MODE:="660"
ENV{DM_NAME}=="asmarch01", OWNER:="grid", GROUP:="asmadmin", MODE:="660"
[root@rac1 ~]#
启动udev
[root@rac1 rules.d]# /sbin/udevadm trigger --type=devices --action=change
检查是否正确
[root@rac1 ~]# ll /dev/dm* brw-rw---- 1 root disk 253, 0 Sep 15 16:57 /dev/dm-0 brw-rw---- 1 root disk 253, 1 Sep 15 16:57 /dev/dm-1 brw-rw---- 1 grid asmadmin 253, 2 Sep 15 16:58 /dev/dm-2 brw-rw---- 1 grid asmadmin 253, 3 Sep 15 16:58 /dev/dm-3 brw-rw---- 1 grid asmadmin 253, 4 Sep 15 16:58 /dev/dm-4 brw-rw---- 1 grid asmadmin 253, 5 Sep 15 16:58 /dev/dm-5 brw-rw---- 1 grid asmadmin 253, 6 Sep 15 16:58 /dev/dm-6 brw-rw---- 1 grid asmadmin 253, 7 Sep 15 16:58 /dev/dm-7 [root@rac1 ~]#
不重启，重新应用udev规则
/sbin/udevadm control --reload-rules && udevadm trigger
2节点重启系统，检查如上则可以了
[root@rac1 ~]# ll /dev/dm*
12.安装集群软件（只截取重要步骤）
[root@rac1 ~]# yum install tigervnc* [grid@rac1:/home/grid]$vncserver
grid用户vnc图形界面安装（之前遇到子窗口无法拉开的情况）可以加如下参数
./runInstaller -jreLoc /etc/alternatives/jre_1.8.0
俩节点安装
[root@rac1 grid]# yum install elfutils-libelf-devel [root@rac1 rpm]# pwd /home/grid/grid/rpm [root@rac1 rpm]# rpm -ivh cvuqdisk-1.0.9-1.rpm
执行第二个脚本的时候卡住不动了（2节点都需要）
解决方案参考 https://blog.csdn.net/shayuwei/article/details/90483029
此处报错可以忽视，继续
13.创建rac，只安装软件不建库（只截取部分）
同上，加参数，子图形窗口可以调整
[oracle@rac1:/home/oracle/database]$./runInstaller -jreLoc /etc/alternatives/jre_1.8.0
解决方案参照 http://blog.itpub.net/70004783/viewspace-2789569/
14，asmca添加磁盘
15.dbca建库
建库完成
16.查看数据库是否正常。
oracle用户查看
[oracle@rac1:/home/oracle]$sqlplus / as sysdba SQL*Plus: Release 11.2.0.4.0 Production on Thu Sep 16 08:57:30 2021 Copyright (c) 1982, 2013, Oracle. All rights reserved. Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP, Data Mining and Real Application Testing options SQL> select name from v$datafile; NAME -------------------------------------------------------------------------------- +DATADG01/prod/datafile/system.265.1083366415 +DATADG01/prod/datafile/sysaux.266.1083366415 +DATADG01/prod/datafile/undotbs1.267.1083366415 +DATADG01/prod/datafile/users.268.1083366415 +DATADG01/prod/datafile/example.273.1083366513 +DATADG01/prod/datafile/undotbs2.274.1083366691 6 rows selected. SQL>
grid用户查看
[grid@rac1:/home/grid/grid]$crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora....DG01.dg ora....up.type ONLINE ONLINE rac1 ora....DG01.dg ora....up.type ONLINE ONLINE rac1 ora....DG02.dg ora....up.type ONLINE ONLINE rac1 ora....ER.lsnr ora....er.type ONLINE ONLINE rac1 ora....N1.lsnr ora....er.type ONLINE ONLINE rac1 ora....DG01.dg ora....up.type ONLINE ONLINE rac1 ora.asm ora.asm.type ONLINE ONLINE rac1 ora.cvu ora.cvu.type ONLINE ONLINE rac1 ora.gsd ora.gsd.type OFFLINE OFFLINE ora....network ora....rk.type ONLINE ONLINE rac1 ora.oc4j ora.oc4j.type ONLINE ONLINE rac1 ora.ons ora.ons.type ONLINE ONLINE rac1 ora.prod.db ora....se.type ONLINE ONLINE rac1 ora....SM1.asm application ONLINE ONLINE rac1 ora....C1.lsnr application ONLINE ONLINE rac1 ora.rac1.gsd application OFFLINE OFFLINE ora.rac1.ons application ONLINE ONLINE rac1 ora.rac1.vip ora....t1.type ONLINE ONLINE rac1 ora....SM2.asm application ONLINE ONLINE rac2 ora....C2.lsnr application ONLINE ONLINE rac2 ora.rac2.gsd application OFFLINE OFFLINE ora.rac2.ons application ONLINE ONLINE rac2 ora.rac2.vip ora....t1.type ONLINE ONLINE rac2 ora.scan1.vip ora....ip.type ONLINE ONLINE rac1 [grid@rac1:/home/grid/grid]$
root用户环境变量添加下面红色部分
PATH=$PATH:$HOME/bin :/u01/app/11.2.0/grid/bin
测试集群启停，生产环境，建议手动停止数据库
[root@rac1 ~]# crsctl stop crs CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac1' CRS-2673: Attempting to stop 'ora.crsd' on 'rac1' CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac1' CRS-2673: Attempting to stop 'ora.SYSTEMDG01.dg' on 'rac1' CRS-2673: Attempting to stop 'ora.prod.db' on 'rac1' CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'rac1' CRS-2673: Attempting to stop 'ora.oc4j' on 'rac1' CRS-2673: Attempting to stop 'ora.cvu' on 'rac1' CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'rac1' CRS-2677: Stop of 'ora.cvu' on 'rac1' succeeded CRS-2672: Attempting to start 'ora.cvu' on 'rac2' CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'rac1' succeeded CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rac1' CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'rac1' succeeded CRS-2673: Attempting to stop 'ora.rac1.vip' on 'rac1' CRS-2676: Start of 'ora.cvu' on 'rac2' succeeded CRS-2677: Stop of 'ora.prod.db' on 'rac1' succeeded CRS-2673: Attempting to stop 'ora.ARCHDG01.dg' on 'rac1' CRS-2673: Attempting to stop 'ora.DATADG01.dg' on 'rac1' CRS-2673: Attempting to stop 'ora.DATADG02.dg' on 'rac1' CRS-2677: Stop of 'ora.scan1.vip' on 'rac1' succeeded CRS-2672: Attempting to start 'ora.scan1.vip' on 'rac2' CRS-2677: Stop of 'ora.DATADG02.dg' on 'rac1' succeeded CRS-2677: Stop of 'ora.DATADG01.dg' on 'rac1' succeeded CRS-2677: Stop of 'ora.rac1.vip' on 'rac1' succeeded CRS-2672: Attempting to start 'ora.rac1.vip' on 'rac2' CRS-2677: Stop of 'ora.ARCHDG01.dg' on 'rac1' succeeded CRS-2676: Start of 'ora.scan1.vip' on 'rac2' succeeded CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'rac2' CRS-2676: Start of 'ora.rac1.vip' on 'rac2' succeeded CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'rac2' succeeded CRS-2677: Stop of 'ora.oc4j' on 'rac1' succeeded CRS-2672: Attempting to start 'ora.oc4j' on 'rac2' CRS-2677: Stop of 'ora.SYSTEMDG01.dg' on 'rac1' succeeded CRS-2673: Attempting to stop 'ora.asm' on 'rac1' CRS-2677: Stop of 'ora.asm' on 'rac1' succeeded CRS-2676: Start of 'ora.oc4j' on 'rac2' succeeded CRS-2673: Attempting to stop 'ora.ons' on 'rac1' CRS-2677: Stop of 'ora.ons' on 'rac1' succeeded CRS-2673: Attempting to stop 'ora.net1.network' on 'rac1' CRS-2677: Stop of 'ora.net1.network' on 'rac1' succeeded CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rac1' has completed CRS-2677: Stop of 'ora.crsd' on 'rac1' succeeded CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac1' CRS-2673: Attempting to stop 'ora.crf' on 'rac1' CRS-2673: Attempting to stop 'ora.ctssd' on 'rac1' CRS-2673: Attempting to stop 'ora.evmd' on 'rac1' CRS-2673: Attempting to stop 'ora.asm' on 'rac1' CRS-2677: Stop of 'ora.crf' on 'rac1' succeeded CRS-2677: Stop of 'ora.mdnsd' on 'rac1' succeeded CRS-2677: Stop of 'ora.evmd' on 'rac1' succeeded CRS-2677: Stop of 'ora.ctssd' on 'rac1' succeeded CRS-2677: Stop of 'ora.asm' on 'rac1' succeeded CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac1' CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac1' succeeded CRS-2673: Attempting to stop 'ora.cssd' on 'rac1' CRS-2677: Stop of 'ora.cssd' on 'rac1' succeeded CRS-2673: Attempting to stop 'ora.gipcd' on 'rac1' CRS-2677: Stop of 'ora.gipcd' on 'rac1' succeeded CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac1' CRS-2677: Stop of 'ora.gpnpd' on 'rac1' succeeded CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac1' has completed CRS-4133: Oracle High Availability Services has been stopped. [root@rac1 ~]#
所有服务都跑到节点2上了
[grid@rac2:/home/grid]$crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora....DG01.dg ora....up.type ONLINE ONLINE rac2 ora....DG01.dg ora....up.type ONLINE ONLINE rac2 ora....DG02.dg ora....up.type ONLINE ONLINE rac2 ora....ER.lsnr ora....er.type ONLINE ONLINE rac2 ora....N1.lsnr ora....er.type ONLINE ONLINE rac2 ora....DG01.dg ora....up.type ONLINE ONLINE rac2 ora.asm ora.asm.type ONLINE ONLINE rac2 ora.cvu ora.cvu.type ONLINE ONLINE rac2 ora.gsd ora.gsd.type OFFLINE OFFLINE ora....network ora....rk.type ONLINE ONLINE rac2 ora.oc4j ora.oc4j.type ONLINE ONLINE rac2 ora.ons ora.ons.type ONLINE ONLINE rac2 ora.prod.db ora....se.type ONLINE ONLINE rac2 ora.rac1.vip ora....t1.type ONLINE ONLINE rac2 ora....SM2.asm application ONLINE ONLINE rac2 ora....C2.lsnr application ONLINE ONLINE rac2 ora.rac2.gsd application OFFLINE OFFLINE ora.rac2.ons application ONLINE ONLINE rac2 ora.rac2.vip ora....t1.type ONLINE ONLINE rac2 ora.scan1.vip ora....ip.type ONLINE ONLINE rac2 [grid@rac2:/home/grid]$