Oracle 11g RAC 节点实例异常关闭


Linux AS 5.3 64bit,  Oracle 11.2.0.1.0  64bit, 2节点RAC   



现象:   节点2实例自动关闭,但是在alert log中没有看到实例关闭的提示,
使用failover方式连接到数据库的时候, 报错信息为:
ORA-01034: ORACLE not available
ORA-27102: out of memory
Linux-x86_64 Error: 12: Cannot allocate memory
连接多次可以连接到还是正常状态的节点1 中 。  



节点2的报错信息如下:  
Tue Oct 04 22:38:38 2011
LGWR: Standby redo logfile selected for thread 2 sequence 13208 for destination LOG_ARCHIVE_DEST_2
Thread 2 advanced to log sequence 13208 (LGWR switch)
  Current log# 23 seq# 13208 mem# 0: +DATA/ccptdb/onlinelog/redo23a.log
  Current log# 23 seq# 13208 mem# 1: +INDX/ccptdb/onlinelog/redo23b.log
Tue Oct 04 22:38:39 2011
Archived Log entry 48294 added for thread 2 sequence 13207 ID 0xbdc4bf83 dest 1:
Tue Oct 04 22:38:41 2011
Process W000 died, see its trace file
Process W000 died, see its trace file
Process W000 died, see its trace file
Tue Oct 04 22:38:53 2011
Process W000 died, see its trace file
Process W000 died, see its trace file
Process W000 died, see its trace file
Tue Oct 04 22:39:05 2011
Process W000 died, see its trace file
Tue Oct 04 22:39:06 2011
Process m000 died, see its trace file
Process W000 died, see its trace file
Process W000 died, see its trace file
Tue Oct 04 22:39:17 2011
Process W000 died, see its trace file
Process W000 died, see its trace file
Process W000 died, see its trace file

Tue Oct 04 22:54:08 2011
Process W000 died, see its trace file
Process W000 died, see its trace file
Process W000 died, see its trace file
Tue Oct 04 22:54:20 2011
Process W000 died, see its trace file
Tue Oct 04 22:54:22 2011
Process J000 died, see its trace file
kkjcre1p: unable to spawn jobq slave process
Errors in file /u01/product/oracle/diag/rdbms/ccptdb/ccptdb2/trace/ccptdb2_cjq0_14870.trc:
Process J000 died, see its trace file
kkjcre1p: unable to spawn jobq slave process
Errors in file /u01/product/oracle/diag/rdbms/ccptdb/ccptdb2/trace/ccptdb2_cjq0_14870.trc:
Process W000 died, see its trace file
Process W000 died, see its trace file
Tue Oct 04 22:54:32 2011
Process W000 died, see its trace file
Process W000 died, see its trace file
Process W000 died, see its trace file
Tue Oct 04 22:55:09 2011
Process W000 died, see its trace file
Tue Oct 04 22:55:10 2011
Process J000 died, see its trace file
kkjcre1p: unable to spawn jobq slave process
Errors in file /u01/product/oracle/diag/rdbms/ccptdb/ccptdb2/trace/ccptdb2_cjq0_14870.trc:
Process J000 died, see its trace file
kkjcre1p: unable to spawn jobq slave process
Errors in file /u01/product/oracle/diag/rdbms/ccptdb/ccptdb2/trace/ccptdb2_cjq0_14870.trc:
Process W000 died, see its trace file
Process W000 died, see its trace file
Tue Oct 04 22:55:21 2011
Process W000 died, see its trace file
Process W000 died, see its trace file
Process W000 died, see its trace file
Tue Oct 04 22:55:33 2011
Process W000 died, see its trace file
Process W000 died, see its trace file
Process W000 died, see its trace file
Tue Oct 04 23:00:21 2011
Process W000 died, see its trace file
Tue Oct 04 23:00:22 2011
Process J000 died, see its trace file
kkjcre1p: unable to spawn jobq slave process
Errors in file /u01/product/oracle/diag/rdbms/ccptdb/ccptdb2/trace/ccptdb2_cjq0_14870.trc:
Process J000 died, see its trace file
kkjcre1p: unable to spawn jobq slave process
Errors in file /u01/product/oracle/diag/rdbms/ccptdb/ccptdb2/trace/ccptdb2_cjq0_14870.trc:
Process W000 died, see its trace file
Process W000 died, see its trace file
Tue Oct 04 23:00:33 2011
Process W000 died, see its trace file
Tue Oct 04 23:00:35 2011
End automatic SQL Tuning Advisor run for special tuning task  "SYS_AUTO_SQL_TUNING_TASK"
Tue Oct 04 23:08:39 2011
LGWR: Standby redo logfile selected for thread 2 sequence 13209 for destination LOG_ARCHIVE_DEST_2
Thread 2 advanced to log sequence 13209 (LGWR switch)
  Current log# 24 seq# 13209 mem# 0: +DATA/ccptdb/onlinelog/redo24a.log
  Current log# 24 seq# 13209 mem# 1: +INDX/ccptdb/onlinelog/redo24b.log
Wed Oct 05 00:00:42 2011
Archived Log entry 48310 added for thread 2 sequence 13210 ID 0xbdc4bf83 dest 1:
Wed Oct 05 00:00:53 2011
LGWR: Standby redo logfile selected for thread 2 sequence 13212 for destination LOG_ARCHIVE_DEST_2
Thread 2 advanced to log sequence 13212 (LGWR switch)
  Current log# 22 seq# 13212 mem# 0: +DATA/ccptdb/onlinelog/redo22a.log
Wed Oct 05 00:09:52 2011
Archived Log entry 48350 added for thread 2 sequence 13215 ID 0xbdc4bf83 dest 1:
Wed Oct 05 00:10:10 2011
LGWR: Standby redo logfile selected for thread 2 sequence 13217 for destination LOG_ARCHIVE_DEST_2
Thread 2 advanced to log sequence 13217 (LGWR switch)
  Current log# 22 seq# 13217 mem# 0: +DATA/ccptdb/onlinelog/redo22a.log
  Current log# 22 seq# 13217 mem# 1: +INDX/ccptdb/onlinelog/redo22b.log
Wed Oct 05 00:41:26 2011
Archived Log entry 48370 added for thread 2 sequence 13218 ID 0xbdc4bf83 dest 1:
Wed Oct 05 01:11:23 2011
LGWR: Standby redo logfile selected for thread 2 sequence 13220 for destination LOG_ARCHIVE_DEST_2
Thread 2 advanced to log sequence 13220 (LGWR switch)
  Current log# 25 seq# 13220 mem# 0: +DATA/ccptdb/onlinelog/redo25a.log
  Current log# 25 seq# 13220 mem# 1: +INDX/ccptdb/onlinelog/redo25b.log
Wed Oct 05 01:11:24 2011
Archived Log entry 48374 added for thread 2 sequence 13219 ID 0xbdc4bf83 dest 1:
Wed Oct 05 01:41:22 2011
LGWR: Standby redo logfile selected for thread 2 sequence 13221 for destination LOG_ARCHIVE_DEST_2
Thread 2 advanced to log sequence 13221 (LGWR switch)
  Current log# 21 seq# 13221 mem# 0: +DATA/ccptdb/onlinelog/redo21a.log
  Current log# 21 seq# 13221 mem# 1: +INDX/ccptdb/onlinelog/redo21b.log
Wed Oct 05 01:41:23 2011
Archived Log entry 48378 added for thread 2 sequence 13220 ID 0xbdc4bf83 dest 1:
Wed Oct 05 02:00:00 2011
Clearing Resource Manager plan via parameter
Wed Oct 05 02:00:02 2011
Process q005 died, see its trace file
Wed Oct 05 02:00:05 2011
Process m000 died, see its trace file
Wed Oct 05 02:00:30 2011
Process m000 died, see its trace file
Wed Oct 05 02:00:40 2011
Process m000 died, see its trace file
Wed Oct 05 04:11:19 2011
LGWR: Standby redo logfile selected for thread 2 sequence 13226 for destination LOG_ARCHIVE_DEST_2
Thread 2 advanced to log sequence 13226 (LGWR switch)
  Current log# 21 seq# 13226 mem# 0: +DATA/ccptdb/onlinelog/redo21a.log
  Current log# 21 seq# 13226 mem# 1: +INDX/ccptdb/onlinelog/redo21b.log
Wed Oct 05 04:11:22 2011
Archived Log entry 48398 added for thread 2 sequence 13225 ID 0xbdc4bf83 dest 1:
Wed Oct 05 04:41:17 2011
LGWR: Standby redo logfile selected for thread 2 sequence 13227 for destination LOG_ARCHIVE_DEST_2
Thread 2 advanced to log sequence 13227 (LGWR switch)
  Current log# 22 seq# 13227 mem# 0: +DATA/ccptdb/onlinelog/redo22a.log
  Current log# 22 seq# 13227 mem# 1: +INDX/ccptdb/onlinelog/redo22b.log
Wed Oct 05 04:41:18 2011
Archived Log entry 48402 added for thread 2 sequence 13226 ID 0xbdc4bf83 dest 1:
Wed Oct 05 05:00:01 2011
Process m000 died, see its trace file
Wed Oct 05 05:00:11 2011
Process m000 died, see its trace file
Wed Oct 05 05:00:22 2011
Process J000 died, see its trace file
kkjcre1p: unable to spawn jobq slave process
Errors in file /u01/product/oracle/diag/rdbms/ccptdb/ccptdb2/trace/ccptdb2_cjq0_14870.trc:
Process J000 died, see its trace file
kkjcre1p: unable to spawn jobq slave process
Errors in file /u01/product/oracle/diag/rdbms/ccptdb/ccptdb2/trace/ccptdb2_cjq0_14870.trc:
Wed Oct 05 05:00:27 2011
Process m000 died, see its trace file
Wed Oct 05 05:00:49 2011
Process J000 died, see its trace file
kkjcre1p: unable to spawn jobq slave process
Errors in file /u01/product/oracle/diag/rdbms/ccptdb/ccptdb2/trace/ccptdb2_cjq0_14870.trc:
Process J000 died, see its trace file
kkjcre1p: unable to spawn jobq slave process
Errors in file /u01/product/oracle/diag/rdbms/ccptdb/ccptdb2/trace/ccptdb2_cjq0_14870.trc:
Wed Oct 05 05:01:15 2011
Process W000 died, see its trace file
Process W000 died, see its trace file
Process W000 died, see its trace file
Wed Oct 05 08:11:14 2011
Archived Log entry 48430 added for thread 2 sequence 13233 ID 0xbdc4bf83 dest 1:
Wed Oct 05 08:19:04 2011
Process J000 died, see its trace file
kkjcre1p: unable to spawn jobq slave process
Errors in file /u01/product/oracle/diag/rdbms/ccptdb/ccptdb2/trace/ccptdb2_cjq0_14870.trc:
Wed Oct 05 08:30:20 2011
Process J001 died, see its trace file
kkjcre1p: unable to spawn jobq slave process
Errors in file /u01/product/oracle/diag/rdbms/ccptdb/ccptdb2/trace/ccptdb2_cjq0_14870.trc:
Wed Oct 05 08:30:43 2011
Process m000 died, see its trace file
Wed Oct 05 08:41:10 2011
LGWR: Standby redo logfile selected for thread 2 sequence 13235 for destination LOG_ARCHIVE_DEST_2
Thread 2 advanced to log sequence 13235 (LGWR switch)
  Current log# 25 seq# 13235 mem# 0: +DATA/ccptdb/onlinelog/redo25a.log
  Current log# 25 seq# 13235 mem# 1: +INDX/ccptdb/onlinelog/redo25b.log
Wed Oct 05 08:41:11 2011
Archived Log entry 48434 added for thread 2 sequence 13234 ID 0xbdc4bf83 dest 1:
Wed Oct 05 08:45:24 2011
Process q002 died, see its trace file
Thu Oct 06 09:28:05 2011
Process W000 died, see its trace file
Process J000 died, see its trace file
kkjcre1p: unable to spawn jobq slave process
Errors in file /u01/product/oracle/diag/rdbms/ccptdb/ccptdb2/trace/ccptdb2_cjq0_14870.trc:
Thu Oct 06 09:28:07 2011
Process PZ99 died, see its trace file
Process J000 died, see its trace file
kkjcre1p: unable to spawn jobq slave process
Errors in file /u01/product/oracle/diag/rdbms/ccptdb/ccptdb2/trace/ccptdb2_cjq0_14870.trc:
Process W000 died, see its trace file
Process W000 died, see its trace file
Thu Oct 06 09:28:17 2011
Process W000 died, see its trace file
Process W000 died, see its trace file
Process W000 died, see its trace file
Thu Oct 06 09:28:26 2011
Starting ORACLE instance (normal)
WARNING: You are trying to use the MEMORY_TARGET feature. This feature requires the /dev/shm file system to
be mounted for at least 26239565824 bytes. /dev/shm is either not mounted or is mounted with available
space less than this size. Please fix this so that MEMORY_TARGET can work as expected. Current available is
23652847616 and used is 9901584384 bytes. Ensure that the mount point is /dev/shm for this directory.
memory_target needs larger /dev/shm
Thu Oct 06 09:28:29 2011
Process J000 died, see its trace file
kkjcre1p: unable to spawn jobq slave process
Errors in file /u01/product/oracle/diag/rdbms/ccptdb/ccptdb2/trace/ccptdb2_cjq0_14870.trc:
Thu Oct 06 09:28:30 2011
Process W000 died, see its trace file
Process J000 died, see its trace file
kkjcre1p: unable to spawn jobq slave process
Errors in file /u01/product/oracle/diag/rdbms/ccptdb/ccptdb2/trace/ccptdb2_cjq0_14870.trc:
Process W000 died, see its trace file
Process W000 died, see its trace file
Thu Oct 06 09:28:42 2011
Process W000 died, see its trace file




Starting ORACLE instance (normal)
WARNING: You are trying to use the MEMORY_TARGET feature. This feature requires the /dev/shm file system to
be mounted for at least 26239565824 bytes. /dev/shm is either not mounted or is mounted with available
space less than this size. Please fix this so that MEMORY_TARGET can work as expected. Current available is
23652847616 and used is 9901584384 bytes. Ensure that the mount point is /dev/shm for this directory.
memory_target needs larger /dev/shm
Oracle试图开启实例,按照报错提示,好象是说 /dev/shm 中的free不足memory_target中设置的大小 。



查看节点2 内存使用情况(节点1上的使用内存约为18G, 总物理内存为32G):  
top - 09:30:50 up 57 days,  2:08,  7 users,  load average: 0.35, 0.77, 0.73
Tasks: 876 total,   1 running, 875 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.3%us,  0.3%sy,  0.0%ni, 98.8%id,  0.1%wa,  0.0%hi,  0.5%si,  0.0%st
Mem:  32948192k total, 31858492k used,  1089700k free,   534552k buffers
Swap:  8193108k total,        0k used,  8193108k free, 16375616k cached

wmrac02<*+ASM2*/home/grid>$free  
             total       used       free     shared    buffers     cached
Mem:      32948192   31863292    1084900          0     534628   16382552
-/+ buffers/cache:   14946112   18002080
Swap:      8193108          0    8193108



在节点2上看到的状态信息如下(ora.ccptdb.db在节点2上):  

wmrac02<*+ASM2*/home/grid>$crs_stat -t  
Name           Type           Target    State     Host        
------------------------------------------------------------
ora.DATA.dg    ora....up.type ONLINE    ONLINE    wmrac01     
ora.INDX.dg    ora....up.type ONLINE    ONLINE    wmrac01     
ora....ER.lsnr ora....er.type ONLINE    ONLINE    wmrac01     
ora....N1.lsnr ora....er.type ONLINE    ONLINE    wmrac02     
ora.asm        ora.asm.type   ONLINE    ONLINE    wmrac01     
ora.ccptdb.db  ora....se.type ONLINE    ONLINE    wmrac02     
ora.eons       ora.eons.type  ONLINE    ONLINE    wmrac01     
ora.gsd        ora.gsd.type   OFFLINE   OFFLINE               
ora....network ora....rk.type ONLINE    ONLINE    wmrac01     
ora.oc4j       ora.oc4j.type  OFFLINE   OFFLINE               
ora.ons        ora.ons.type   ONLINE    ONLINE    wmrac01     
ora....ry.acfs ora....fs.type ONLINE    ONLINE    wmrac01     
ora.scan1.vip  ora....ip.type ONLINE    ONLINE    wmrac02     
ora....SM1.asm application    ONLINE    ONLINE    wmrac01     
ora....01.lsnr application    ONLINE    ONLINE    wmrac01     
ora....c01.gsd application    OFFLINE   OFFLINE               
ora....c01.ons application    ONLINE    ONLINE    wmrac01     
ora....c01.vip ora....t1.type ONLINE    ONLINE    wmrac01     
ora....SM2.asm application    ONLINE    ONLINE    wmrac02     
ora....02.lsnr application    ONLINE    ONLINE    wmrac02     
ora....c02.gsd application    OFFLINE   OFFLINE               
ora....c02.ons application    ONLINE    ONLINE    wmrac02     
ora....c02.vip ora....t1.type ONLINE    ONLINE    wmrac02     

请使用浏览器的分享功能分享到微信等