Linux 核心版本号 ：
[root@wmrac01 log]# uname -a
Linux wmrac01 2.6.18-128.el5 #1 SMP Wed Dec 17 11:41:38 EST 2008 x86_64 x86_64 x86_64 GNU/Linux
现象： 节点2 不能正常访问， 开启报错
wmrac01<*+ASM1*/home/grid>$crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.DATA.dg ora....up.type ONLINE ONLINE wmrac01
ora.INDX.dg ora....up.type ONLINE ONLINE wmrac01
ora....ER.lsnr ora....er.type ONLINE ONLINE wmrac01
ora....N1.lsnr ora....er.type ONLINE ONLINE wmrac01
ora.asm ora.asm.type ONLINE ONLINE wmrac01
ora.ccptdb.db ora....se.type ONLINE ONLINE wmrac01
ora.eons ora.eons.type ONLINE ONLINE wmrac01
ora.gsd ora.gsd.type OFFLINE OFFLINE
ora....network ora....rk.type ONLINE ONLINE wmrac01
ora.oc4j ora.oc4j.type OFFLINE OFFLINE
ora.ons ora.ons.type ONLINE ONLINE wmrac01
ora....ry.acfs ora....fs.type ONLINE ONLINE wmrac01
ora.scan1.vip ora....ip.type ONLINE ONLINE wmrac01
ora....SM1.asm application ONLINE ONLINE wmrac01
ora....01.lsnr application ONLINE ONLINE wmrac01
ora....c01.gsd application OFFLINE OFFLINE
ora....c01.ons application ONLINE ONLINE wmrac01
ora....c01.vip ora....t1.type ONLINE ONLINE wmrac01
ora....SM2.asm application ONLINE OFFLINE
ora....02.lsnr application ONLINE OFFLINE
ora....c02.gsd application OFFLINE OFFLINE
ora....c02.ons application ONLINE OFFLINE
ora....c02.vip ora....t1.type ONLINE ONLINE wmrac01
wmrac02<*+ASM2*/home/grid/dba/g_bdump>$crsctl start cluster
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'wmrac02'
CRS-2676: Start of 'ora.cssdmonitor' on 'wmrac02' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'wmrac02'
CRS-2672: Attempting to start 'ora.diskmon' on 'wmrac02'
CRS-2676: Start of 'ora.diskmon' on 'wmrac02' succeeded
CRS-2676: Start of 'ora.cssd' on 'wmrac02' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'wmrac02'
CRS-2676: Start of 'ora.ctssd' on 'wmrac02' succeeded
CRS-2672: Attempting to start 'ora.evmd' on 'wmrac02'
CRS-2672: Attempting to start 'ora.asm' on 'wmrac02'
CRS-2676: Start of 'ora.evmd' on 'wmrac02' succeeded
CRS-5011: Check of resource "+ASM" failed: details at "(:CLSN00006 " in "/u01/product/grid/11.2.0/log/wmrac02/agent/ohasd/oraagent_grid/oraagent_grid.log"
CRS-2674: Start of 'ora.asm' on 'wmrac02' failed
CRS-2679: Attempting to clean 'ora.asm' on 'wmrac02'
CRS-5011: Check of resource "+ASM" failed: details at "(:CLSN00006 " in "/u01/product/grid/11.2.0/log/wmrac02/agent/ohasd/oraagent_grid/oraagent_grid.log"
CRS-2681: Clean of 'ora.asm' on 'wmrac02' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'wmrac02'
CRS-2677: Stop of 'ora.ctssd' on 'wmrac02' succeeded
CRS-2673: Attempting to stop 'ora.diskmon' on 'wmrac02'
CRS-2677: Stop of 'ora.diskmon' on 'wmrac02' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'wmrac02'
CRS-2677: Stop of 'ora.cssd' on 'wmrac02' succeeded
CRS-4000: Command Start failed, or completed with errors.
各种Log 文件 ： RAC01 ORACLE LOG : Thu Nov 10 21:27:12 2011LGWR: Standby redo logfile selected for thread 1 sequence 16654 for destination LOG_ARCHIVE_DEST_2Thread 1 advanced to log sequence 16654 (LGWR switch) Current log# 11 seq# 16654 mem# 0: +DATA/ccptdb/onlinelog/redo11a.log Current log# 11 seq# 16654 mem# 1: +INDX/ccptdb/onlinelog/redo11b.logThu Nov 10 21:27:13 2011 Archived Log entry 58548 added for thread 1 sequence 16653 ID 0xbdc4bf83 dest 1: Thu Nov 10 21:47:45 2011Errors in file /u01/product/oracle/diag/rdbms/ccptdb/ccptdb1/trace/ccptdb1_lms2_15653.trc:ORA-27550: Target ID protocol check failed. tid vers=1, type=3, remote instance number=2, local instance number=1 Thu Nov 10 21:47:47 2011 Trace dumping is performing id=[cdmp_20111110214747]Errors in file /u01/product/oracle/diag/rdbms/ccptdb/ccptdb1/trace/ccptdb1_lms2_15653.trc:ORA-27550: Target ID protocol check failed. tid vers=1, type=3, remote instance number=2, local instance number=1 LMS2 (ospid: 15653): terminating the instance due to error 27550 --节点1 上的实例在 21:47:47 正在被强制关闭 。Thu Nov 10 21:47:48 2011
ORA-1092 : opitsk aborting process
Thu Nov 10 21:47:48 2011
License high water mark = 61
Instance terminated by LMS2, pid = 15653 --节点1 实例在 21:47:48 已被强制关闭。
USER (ospid: 30890): terminating the instance
Instance terminated by USER, pid = 30890
Thu Nov 10 21:48:08 2011
Starting ORACLE instance (normal) -- 节点1实例在 21:48:08 正在重新开启
Thu Nov 10 21:48:24 2011
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Interface type 1 bond1 192.168.1.0 configured from GPnP Profile for use as a cluster interconnect
Interface type 1 bond0 10.118.253.0 configured from GPnP Profile for use as a public interface
Picked latch-free SCN scheme 3
Autotune of undo retention is turned on.
LICENSE_MAX_USERS = 0
…..
Starting background process MARK
Thu Nov 10 21:48:39 2011
MARK started with pid=38, OS id=31282
NOTE: MARK has subscribed
lmon registered with NM - instance number 1 (internal mem no 0)
Reconfiguration started (old inc 0, new inc 8)
List of instances:
1 2 (myinst: 1) --- 21:48:39 可以看到两个实例
Global Resource Directory frozen
* allocate domain 0, invalid = TRUE
Communication channels reestablished
* domain 0 valid according to instance 2
* domain 0 valid = 1 according to instance 2
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
LMS 1: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
LMS 3: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
LMS 2: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
Thu Nov 10 21:52:50 2011 --- 21:52:50 准备踢掉节点2
Remote instance 2 kill is issued with system inc 8
LMON received an instance eviction notification from instance 1
The instance eviction reason is 0x2
The instance eviction map is 2
Reconfiguration started (old inc 8, new inc 10)
List of instances:
1 (myinst: 1) --- 21:52:50 踢掉节点2
Nested reconfiguration detected.
Global Resource Directory frozen
* dead instance detected - domain 0 invalid = TRUE
Communication channels reestablished
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
……..
Thu Nov 10 21:52:54 2011
LMS 2: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
Thu Nov 10 21:52:54 2011
LMS 1: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
Thu Nov 10 21:52:54 2011
Thu Nov 10 21:52:54 2011
LCK0 started with pid=39, OS id=31910
Thu Nov 10 21:52:54 2011
Starting background process RSMN
Thu Nov 10 21:52:55 2011
RSMN started with pid=40, OS id=31912
ORACLE_BASE not set in environment. It is recommended
that ORACLE_BASE be set in the environment
Reusing ORACLE_BASE from an earlier startup = /u01/product/oracle
Thu Nov 10 21:52:55 2011
ALTER DATABASE MOUNT
This instance was first to mount
Thu Nov 10 21:52:55 2011
NOTE: Loaded library: /opt/oracle/extapi/64/asm/orcl/1/libasm.so
NOTE: Loaded library: System
Thu Nov 10 21:52:55 2011
SUCCESS: diskgroup DATA was mounted
Thu Nov 10 21:52:56 2011
NOTE: dependency between database ccptdb and diskgroup resource ora.DATA.dg is established
Thu Nov 10 21:53:00 2011
NSS2 started with pid=43, OS id=31965
Thu Nov 10 21:53:00 2011
Successful mount of redo thread 1, with mount id 3206033079
Database mounted in Shared Mode (CLUSTER_DATABASE=TRUE)
Lost write protection disabled
Completed: ALTER DATABASE MOUNT
ALTER DATABASE OPEN --- 实例1 正常开启 。 后面一直是正常状态 。
This instance was first to open