在RAC安装配置和使用过程的问题解决方法总结二中我们提过因为RAC节点虚拟机重启导致的crs daemon启动失败并进行了简单的解决，
再次测试后发现，每次重启/dev/raw下的raw1/2/3/4/5的权限都会变为root.root，因此把下面两句话加入/etc/rc.d/rc.local：
chown -R oracle.oinstall /dev/raw
chmod -R 777 /dev/raw

然后作如下测试：
[root@10grac1 init.d]# ./init.crs stop
Shutting down Oracle Cluster Ready Services (CRS):
Stopping resources.
Successfully stopped CRS resources 
Stopping CSSD.
Shutting down CSS daemon.
Shutdown request successfully issued.
Shutdown has begun. The daemons should exit soon.


[root@10grac1 init.d]# ./init.crs start
Startup will be queued to init within 90 seconds.


[oracle@10grac1 bin]$ crs_stat -t
名称         类型         目标    状态    主机      
------------------------------------------------------------
ora....SM1.asm application    ONLINE    ONLINE    10grac1     
ora....C1.lsnr application    ONLINE    ONLINE    10grac1     
ora....ac1.gsd application    ONLINE    ONLINE    10grac1     
ora....ac1.ons application    ONLINE    ONLINE    10grac1     
ora....ac1.vip application    ONLINE    ONLINE    10grac1     
ora....SM2.asm application    ONLINE    ONLINE    10grac2     
ora....C2.lsnr application    ONLINE    ONLINE    10grac2     
ora....ac2.gsd application    ONLINE    UNKNOWN   10grac2     
ora....ac2.ons application    ONLINE    UNKNOWN   10grac2     
ora....ac2.vip application    ONLINE    ONLINE    10grac2     
ora.racnan.db  application    ONLINE    ONLINE    10grac2     
ora....vice.cs application    ONLINE    ONLINE    10grac1     
ora....an1.srv application    ONLINE    ONLINE    10grac1     
ora....n1.inst application    ONLINE    ONLINE    10grac1     
ora....n2.inst application    ONLINE    ONLINE    10grac2 

--对UNKNOWN的服务进行手工stop手工start
[oracle@10grac1 bin]$ crs_stop -f ora.10grac2.gsd
Attempting to stop `ora.10grac2.gsd` on member `10grac2`
Stop of `ora.10grac2.gsd` on member `10grac2` succeeded.
[oracle@10grac1 bin]$ crs_start ora.10grac2.gsd
Attempting to start `ora.10grac2.gsd` on member `10grac2`
Start of `ora.10grac2.gsd` on member `10grac2` succeeded.
[oracle@10grac1 bin]$ crs_stat -t
名称         类型         目标    状态    主机      
------------------------------------------------------------
ora....SM1.asm application    ONLINE    ONLINE    10grac1     
ora....C1.lsnr application    ONLINE    ONLINE    10grac1     
ora....ac1.gsd application    ONLINE    ONLINE    10grac1     
ora....ac1.ons application    ONLINE    ONLINE    10grac1     
ora....ac1.vip application    ONLINE    ONLINE    10grac1     
ora....SM2.asm application    ONLINE    ONLINE    10grac2     
ora....C2.lsnr application    ONLINE    ONLINE    10grac2     
ora....ac2.gsd application    ONLINE    ONLINE    10grac2     
ora....ac2.ons application    ONLINE    UNKNOWN   10grac2     
ora....ac2.vip application    ONLINE    ONLINE    10grac2     
ora.racnan.db  application    ONLINE    ONLINE    10grac2     
ora....vice.cs application    ONLINE    ONLINE    10grac1     
ora....an1.srv application    ONLINE    ONLINE    10grac1     
ora....n1.inst application    ONLINE    ONLINE    10grac1     
ora....n2.inst application    ONLINE    ONLINE    10grac2     
[oracle@10grac1 bin]$ crs_stop -f ora.10grac2.ons
Attempting to stop `ora.10grac2.ons` on member `10grac2`
Stop of `ora.10grac2.ons` on member `10grac2` succeeded.
[oracle@10grac1 bin]$ crs_start ora.10grac2.ons
Attempting to start `ora.10grac2.ons` on member `10grac2`
Start of `ora.10grac2.ons` on member `10grac2` succeeded.
[oracle@10grac1 bin]$ crs_stat -t
名称         类型         目标    状态    主机      
------------------------------------------------------------
ora....SM1.asm application    ONLINE    ONLINE    10grac1     
ora....C1.lsnr application    ONLINE    ONLINE    10grac1     
ora....ac1.gsd application    ONLINE    ONLINE    10grac1     
ora....ac1.ons application    ONLINE    ONLINE    10grac1     
ora....ac1.vip application    ONLINE    ONLINE    10grac1     
ora....SM2.asm application    ONLINE    ONLINE    10grac2     
ora....C2.lsnr application    ONLINE    ONLINE    10grac2     
ora....ac2.gsd application    ONLINE    ONLINE    10grac2     
ora....ac2.ons application    ONLINE    ONLINE    10grac2     
ora....ac2.vip application    ONLINE    ONLINE    10grac2     
ora.racnan.db  application    ONLINE    ONLINE    10grac2     
ora....vice.cs application    ONLINE    ONLINE    10grac1     
ora....an1.srv application    ONLINE    ONLINE    10grac1     
ora....n1.inst application    ONLINE    ONLINE    10grac1     
ora....n2.inst application    ONLINE    ONLINE    10grac2 


之后从头再执行一遍：
[root@10grac1 init.d]# ./init.crs stop
Shutting down Oracle Cluster Ready Services (CRS):
Stopping resources.
Successfully stopped CRS resources 
Stopping CSSD.
Shutting down CSS daemon.
Shutdown request successfully issued.
Shutdown has begun. The daemons should exit soon.


[root@10grac1 init.d]# ./init.crs start
Startup will be queued to init within 90 seconds.


[oracle@10grac1 bin]$ crs_stat -t
名称         类型         目标    状态    主机      
------------------------------------------------------------
ora....SM1.asm application    ONLINE    ONLINE    10grac1     
ora....C1.lsnr application    ONLINE    ONLINE    10grac1     
ora....ac1.gsd application    ONLINE    ONLINE    10grac1     
ora....ac1.ons application    ONLINE    ONLINE    10grac1     
ora....ac1.vip application    ONLINE    ONLINE    10grac1     
ora....SM2.asm application    ONLINE    ONLINE    10grac2     
ora....C2.lsnr application    ONLINE    ONLINE    10grac2     
ora....ac2.gsd application    ONLINE    ONLINE    10grac2     
ora....ac2.ons application    ONLINE    ONLINE    10grac2     
ora....ac2.vip application    ONLINE    ONLINE    10grac2     
ora.racnan.db  application    ONLINE    ONLINE    10grac2     
ora....vice.cs application    ONLINE    ONLINE    10grac1     
ora....an1.srv application    ONLINE    ONLINE    10grac1     
ora....n1.inst application    ONLINE    ONLINE    10grac1     
ora....n2.inst application    ONLINE    ONLINE    10grac2 
全部OK!

接下来再次重启两台虚拟机看看是什么状态：
重启完成后，进入其中一个节点：
[oracle@10grac1 ~]$ crs_stat -t
Name           Type           Target    State     Host        
------------------------------------------------------------
ora....SM1.asm application    ONLINE    ONLINE    10grac1     
ora....C1.lsnr application    ONLINE    ONLINE    10grac1     
ora....ac1.gsd application    ONLINE    ONLINE    10grac1     
ora....ac1.ons application    ONLINE    ONLINE    10grac1     
ora....ac1.vip application    ONLINE    ONLINE    10grac1     
ora....SM2.asm application    ONLINE    ONLINE    10grac2     
ora....C2.lsnr application    ONLINE    OFFLINE               
ora....ac2.gsd application    ONLINE    ONLINE    10grac2     
ora....ac2.ons application    ONLINE    ONLINE    10grac2     
ora....ac2.vip application    ONLINE    ONLINE    10grac2     
ora.racnan.db  application    ONLINE    ONLINE    10grac2     
ora....vice.cs application    ONLINE    ONLINE    10grac1     
ora....an1.srv application    ONLINE    ONLINE    10grac1     
ora....n1.inst application    ONLINE    ONLINE    10grac1     
ora....n2.inst application    ONLINE    UNKNOWN   10grac2     
发现第二个节点的listener offline，尝试解决：
NAME=ora.10grac2.LISTENER_10GRAC2.lsnr
TYPE=application
TARGET=ONLINE
STATE=OFFLINE

[oracle@10grac1 ~]$ crsctl start resources ora.10grac2.LISTENER_10GRAC2.lsnr
Starting resources.
Successfully started CRS resources 
[oracle@10grac1 ~]$ crs_stat -t
Name           Type           Target    State     Host        
------------------------------------------------------------
ora....SM1.asm application    ONLINE    ONLINE    10grac1     
ora....C1.lsnr application    ONLINE    ONLINE    10grac1     
ora....ac1.gsd application    ONLINE    ONLINE    10grac1     
ora....ac1.ons application    ONLINE    ONLINE    10grac1     
ora....ac1.vip application    ONLINE    ONLINE    10grac1     
ora....SM2.asm application    ONLINE    ONLINE    10grac2     
ora....C2.lsnr application    ONLINE    UNKNOWN   10grac2     
ora....ac2.gsd application    ONLINE    ONLINE    10grac2     
ora....ac2.ons application    ONLINE    ONLINE    10grac2     
ora....ac2.vip application    ONLINE    ONLINE    10grac2     
ora.racnan.db  application    ONLINE    ONLINE    10grac2     
ora....vice.cs application    ONLINE    ONLINE    10grac1     
ora....an1.srv application    ONLINE    ONLINE    10grac1     
ora....n1.inst application    ONLINE    ONLINE    10grac1     
ora....n2.inst application    ONLINE    UNKNOWN   10grac2     

[oracle@10grac1 ~]$ crs_stop ora.10grac2.LISTENER_10GRAC2.lsnr
Attempting to stop `ora.10grac2.LISTENER_10GRAC2.lsnr` on member `10grac2`
`ora.10grac2.LISTENER_10GRAC2.lsnr` on member `10grac2` has experienced an unrecoverable failure.
Human intervention required to resume its availability.
CRS-0216: Could not stop resource 'ora.10grac2.LISTENER_10GRAC2.lsnr'.


[oracle@10grac1 ~]$ crs_start ora.10grac2.LISTENER_10GRAC2.lsnr
CRS-1028: Dependency analysis failed because of:
'Resource in UNKNOWN state: ora.10grac2.LISTENER_10GRAC2.lsnr'


CRS-0223: Resource 'ora.10grac2.LISTENER_10GRAC2.lsnr' has placement error.


[oracle@10grac1 ~]$ crs_stat -t
Name           Type           Target    State     Host        
------------------------------------------------------------
ora....SM1.asm application    ONLINE    ONLINE    10grac1     
ora....C1.lsnr application    ONLINE    ONLINE    10grac1     
ora....ac1.gsd application    ONLINE    ONLINE    10grac1     
ora....ac1.ons application    ONLINE    ONLINE    10grac1     
ora....ac1.vip application    ONLINE    ONLINE    10grac1     
ora....SM2.asm application    ONLINE    ONLINE    10grac2     
ora....C2.lsnr application    ONLINE    UNKNOWN   10grac2     
ora....ac2.gsd application    ONLINE    ONLINE    10grac2     
ora....ac2.ons application    ONLINE    ONLINE    10grac2     
ora....ac2.vip application    ONLINE    ONLINE    10grac2     
ora.racnan.db  application    ONLINE    ONLINE    10grac2     
ora....vice.cs application    ONLINE    ONLINE    10grac1     
ora....an1.srv application    ONLINE    ONLINE    10grac1     
ora....n1.inst application    ONLINE    ONLINE    10grac1     
ora....n2.inst application    ONLINE    UNKNOWN   10grac2  

等了一会儿，再执行，居然就对了。。。
[oracle@10grac1 ~]$ crs_stop ora.10grac2.LISTENER_10GRAC2.lsnr
Attempting to stop `ora.10grac2.LISTENER_10GRAC2.lsnr` on member `10grac2`
Stop of `ora.10grac2.LISTENER_10GRAC2.lsnr` on member `10grac2` succeeded.
[oracle@10grac1 ~]$ crs_start ora.10grac2.LISTENER_10GRAC2.lsnr
Attempting to start `ora.10grac2.LISTENER_10GRAC2.lsnr` on member `10grac2`
Start of `ora.10grac2.LISTENER_10GRAC2.lsnr` on member `10grac2` succeeded.
[oracle@10grac1 ~]$ crs_stat -t
Name           Type           Target    State     Host        
------------------------------------------------------------
ora....SM1.asm application    ONLINE    ONLINE    10grac1     
ora....C1.lsnr application    ONLINE    ONLINE    10grac1     
ora....ac1.gsd application    ONLINE    ONLINE    10grac1     
ora....ac1.ons application    ONLINE    ONLINE    10grac1     
ora....ac1.vip application    ONLINE    ONLINE    10grac1     
ora....SM2.asm application    ONLINE    ONLINE    10grac2     
ora....C2.lsnr application    ONLINE    ONLINE    10grac2     
ora....ac2.gsd application    ONLINE    ONLINE    10grac2     
ora....ac2.ons application    ONLINE    ONLINE    10grac2     
ora....ac2.vip application    ONLINE    ONLINE    10grac2     
ora.racnan.db  application    ONLINE    ONLINE    10grac2     
ora....vice.cs application    ONLINE    ONLINE    10grac1     
ora....an1.srv application    ONLINE    ONLINE    10grac1     
ora....n1.inst application    ONLINE    ONLINE    10grac1     
ora....n2.inst application    ONLINE    UNKNOWN   10grac2     

神奇啊，crs啊，你究竟在后面捣鼓啥呢！我让你捣鼓个够儿哈！
[oracle@10grac1 ~]$ crs_stop ora.racnan.racnan2.inst
Attempting to stop `ora.racnan.racnan2.inst` on member `10grac2`
Stop of `ora.racnan.racnan2.inst` on member `10grac2` succeeded.
[oracle@10grac1 ~]$ crs_start ora.racnan.racnan2.inst
Attempting to start `ora.racnan.racnan2.inst` on member `10grac2`
Start of `ora.racnan.racnan2.inst` on member `10grac2` succeeded.
[oracle@10grac1 ~]$ crs_stat -t
Name           Type           Target    State     Host        
------------------------------------------------------------
ora....SM1.asm application    ONLINE    ONLINE    10grac1     
ora....C1.lsnr application    ONLINE    ONLINE    10grac1     
ora....ac1.gsd application    ONLINE    ONLINE    10grac1     
ora....ac1.ons application    ONLINE    ONLINE    10grac1     
ora....ac1.vip application    ONLINE    ONLINE    10grac1     
ora....SM2.asm application    ONLINE    ONLINE    10grac2     
ora....C2.lsnr application    ONLINE    ONLINE    10grac2     
ora....ac2.gsd application    ONLINE    ONLINE    10grac2     
ora....ac2.ons application    ONLINE    ONLINE    10grac2     
ora....ac2.vip application    ONLINE    ONLINE    10grac2     
ora.racnan.db  application    ONLINE    ONLINE    10grac2     
ora....vice.cs application    ONLINE    ONLINE    10grac1     
ora....an1.srv application    ONLINE    ONLINE    10grac1     
ora....n1.inst application    ONLINE    ONLINE    10grac1     
ora....n2.inst application    ONLINE    ONLINE    10grac2     

---以后再重启再遇到新情况就再来汇报，今天就到这里啦，俺还要去捣鼓完NOSQL的作业呢