Linux AS 5.3 64bit, Oracle 11.2.0.1.0 64bit, 2节点RAC
现象: 节点2实例自动关闭,但是在alert log中没有看到实例关闭的提示,
使用failover方式连接到数据库的时候, 报错信息为:
ORA-01034: ORACLE not available
ORA-27102: out of memory
Linux-x86_64 Error: 12: Cannot allocate memory
连接多次可以连接到还是正常状态的节点1 中 。
节点2的报错信息如下:
Tue Oct 04 22:38:38 2011
LGWR: Standby redo logfile selected for thread 2 sequence 13208 for destination LOG_ARCHIVE_DEST_2
Thread 2 advanced to log sequence 13208 (LGWR switch)
Current log# 23 seq# 13208 mem# 0: +DATA/ccptdb/onlinelog/redo23a.log
Current log# 23 seq# 13208 mem# 1: +INDX/ccptdb/onlinelog/redo23b.log
Tue Oct 04 22:38:39 2011
Archived Log entry 48294 added for thread 2 sequence 13207 ID 0xbdc4bf83 dest 1:
Tue Oct 04 22:38:41 2011
Process W000 died, see its trace file
Process W000 died, see its trace file
Process W000 died, see its trace file
Tue Oct 04 22:38:53 2011
Process W000 died, see its trace file
Process W000 died, see its trace file
Process W000 died, see its trace file
Tue Oct 04 22:39:05 2011
Process W000 died, see its trace file
Tue Oct 04 22:39:06 2011
Process m000 died, see its trace file
Process W000 died, see its trace file
Process W000 died, see its trace file
Tue Oct 04 22:39:17 2011
Process W000 died, see its trace file
Process W000 died, see its trace file
Process W000 died, see its trace file
Tue Oct 04 22:54:08 2011
Process W000 died, see its trace file
Process W000 died, see its trace file
Process W000 died, see its trace file
Tue Oct 04 22:54:20 2011
Process W000 died, see its trace file
Tue Oct 04 22:54:22 2011
Process J000 died, see its trace file
kkjcre1p: unable to spawn jobq slave process
Errors in file /u01/product/oracle/diag/rdbms/ccptdb/ccptdb2/trace/ccptdb2_cjq0_14870.trc:
Process J000 died, see its trace file
kkjcre1p: unable to spawn jobq slave process
Errors in file /u01/product/oracle/diag/rdbms/ccptdb/ccptdb2/trace/ccptdb2_cjq0_14870.trc:
Process W000 died, see its trace file
Process W000 died, see its trace file
Tue Oct 04 22:54:32 2011
Process W000 died, see its trace file
Process W000 died, see its trace file
Process W000 died, see its trace file
Tue Oct 04 22:55:09 2011
Process W000 died, see its trace file
Tue Oct 04 22:55:10 2011
Process J000 died, see its trace file
kkjcre1p: unable to spawn jobq slave process
Errors in file /u01/product/oracle/diag/rdbms/ccptdb/ccptdb2/trace/ccptdb2_cjq0_14870.trc:
Process J000 died, see its trace file
kkjcre1p: unable to spawn jobq slave process
Errors in file /u01/product/oracle/diag/rdbms/ccptdb/ccptdb2/trace/ccptdb2_cjq0_14870.trc:
Process W000 died, see its trace file
Process W000 died, see its trace file
Tue Oct 04 22:55:21 2011
Process W000 died, see its trace file
Process W000 died, see its trace file
Process W000 died, see its trace file
Tue Oct 04 22:55:33 2011
Process W000 died, see its trace file
Process W000 died, see its trace file
Process W000 died, see its trace file
Tue Oct 04 23:00:21 2011
Process W000 died, see its trace file
Tue Oct 04 23:00:22 2011
Process J000 died, see its trace file
kkjcre1p: unable to spawn jobq slave process
Errors in file /u01/product/oracle/diag/rdbms/ccptdb/ccptdb2/trace/ccptdb2_cjq0_14870.trc:
Process J000 died, see its trace file
kkjcre1p: unable to spawn jobq slave process
Errors in file /u01/product/oracle/diag/rdbms/ccptdb/ccptdb2/trace/ccptdb2_cjq0_14870.trc:
Process W000 died, see its trace file
Process W000 died, see its trace file
Tue Oct 04 23:00:33 2011
Process W000 died, see its trace file
Tue Oct 04 23:00:35 2011
End automatic SQL Tuning Advisor run for special tuning task "SYS_AUTO_SQL_TUNING_TASK"
Tue Oct 04 23:08:39 2011
LGWR: Standby redo logfile selected for thread 2 sequence 13209 for destination LOG_ARCHIVE_DEST_2
Thread 2 advanced to log sequence 13209 (LGWR switch)
Current log# 24 seq# 13209 mem# 0: +DATA/ccptdb/onlinelog/redo24a.log
Current log# 24 seq# 13209 mem# 1: +INDX/ccptdb/onlinelog/redo24b.log
Wed Oct 05 00:00:42 2011
Archived Log entry 48310 added for thread 2 sequence 13210 ID 0xbdc4bf83 dest 1:
Wed Oct 05 00:00:53 2011
LGWR: Standby redo logfile selected for thread 2 sequence 13212 for destination LOG_ARCHIVE_DEST_2
Thread 2 advanced to log sequence 13212 (LGWR switch)
Current log# 22 seq# 13212 mem# 0: +DATA/ccptdb/onlinelog/redo22a.log
Wed Oct 05 00:09:52 2011
Archived Log entry 48350 added for thread 2 sequence 13215 ID 0xbdc4bf83 dest 1:
Wed Oct 05 00:10:10 2011
LGWR: Standby redo logfile selected for thread 2 sequence 13217 for destination LOG_ARCHIVE_DEST_2
Thread 2 advanced to log sequence 13217 (LGWR switch)
Current log# 22 seq# 13217 mem# 0: +DATA/ccptdb/onlinelog/redo22a.log
Current log# 22 seq# 13217 mem# 1: +INDX/ccptdb/onlinelog/redo22b.log
Wed Oct 05 00:41:26 2011
Archived Log entry 48370 added for thread 2 sequence 13218 ID 0xbdc4bf83 dest 1:
Wed Oct 05 01:11:23 2011
LGWR: Standby redo logfile selected for thread 2 sequence 13220 for destination LOG_ARCHIVE_DEST_2
Thread 2 advanced to log sequence 13220 (LGWR switch)
Current log# 25 seq# 13220 mem# 0: +DATA/ccptdb/onlinelog/redo25a.log
Current log# 25 seq# 13220 mem# 1: +INDX/ccptdb/onlinelog/redo25b.log
Wed Oct 05 01:11:24 2011
Archived Log entry 48374 added for thread 2 sequence 13219 ID 0xbdc4bf83 dest 1:
Wed Oct 05 01:41:22 2011
LGWR: Standby redo logfile selected for thread 2 sequence 13221 for destination LOG_ARCHIVE_DEST_2
Thread 2 advanced to log sequence 13221 (LGWR switch)
Current log# 21 seq# 13221 mem# 0: +DATA/ccptdb/onlinelog/redo21a.log
Current log# 21 seq# 13221 mem# 1: +INDX/ccptdb/onlinelog/redo21b.log
Wed Oct 05 01:41:23 2011
Archived Log entry 48378 added for thread 2 sequence 13220 ID 0xbdc4bf83 dest 1:
Wed Oct 05 02:00:00 2011
Clearing Resource Manager plan via parameter
Wed Oct 05 02:00:02 2011
Process q005 died, see its trace file
Wed Oct 05 02:00:05 2011
Process m000 died, see its trace file
Wed Oct 05 02:00:30 2011
Process m000 died, see its trace file
Wed Oct 05 02:00:40 2011
Process m000 died, see its trace file
Wed Oct 05 04:11:19 2011
LGWR: Standby redo logfile selected for thread 2 sequence 13226 for destination LOG_ARCHIVE_DEST_2
Thread 2 advanced to log sequence 13226 (LGWR switch)
Current log# 21 seq# 13226 mem# 0: +DATA/ccptdb/onlinelog/redo21a.log
Current log# 21 seq# 13226 mem# 1: +INDX/ccptdb/onlinelog/redo21b.log
Wed Oct 05 04:11:22 2011
Archived Log entry 48398 added for thread 2 sequence 13225 ID 0xbdc4bf83 dest 1:
Wed Oct 05 04:41:17 2011
LGWR: Standby redo logfile selected for thread 2 sequence 13227 for destination LOG_ARCHIVE_DEST_2
Thread 2 advanced to log sequence 13227 (LGWR switch)
Current log# 22 seq# 13227 mem# 0: +DATA/ccptdb/onlinelog/redo22a.log
Current log# 22 seq# 13227 mem# 1: +INDX/ccptdb/onlinelog/redo22b.log
Wed Oct 05 04:41:18 2011
Archived Log entry 48402 added for thread 2 sequence 13226 ID 0xbdc4bf83 dest 1:
Wed Oct 05 05:00:01 2011
Process m000 died, see its trace file
Wed Oct 05 05:00:11 2011
Process m000 died, see its trace file
Wed Oct 05 05:00:22 2011
Process J000 died, see its trace file
kkjcre1p: unable to spawn jobq slave process
Errors in file /u01/product/oracle/diag/rdbms/ccptdb/ccptdb2/trace/ccptdb2_cjq0_14870.trc:
Process J000 died, see its trace file
kkjcre1p: unable to spawn jobq slave process
Errors in file /u01/product/oracle/diag/rdbms/ccptdb/ccptdb2/trace/ccptdb2_cjq0_14870.trc:
Wed Oct 05 05:00:27 2011
Process m000 died, see its trace file
Wed Oct 05 05:00:49 2011
Process J000 died, see its trace file
kkjcre1p: unable to spawn jobq slave process
Errors in file /u01/product/oracle/diag/rdbms/ccptdb/ccptdb2/trace/ccptdb2_cjq0_14870.trc:
Process J000 died, see its trace file
kkjcre1p: unable to spawn jobq slave process
Errors in file /u01/product/oracle/diag/rdbms/ccptdb/ccptdb2/trace/ccptdb2_cjq0_14870.trc:
Wed Oct 05 05:01:15 2011
Process W000 died, see its trace file
Process W000 died, see its trace file
Process W000 died, see its trace file
Wed Oct 05 08:11:14 2011
Archived Log entry 48430 added for thread 2 sequence 13233 ID 0xbdc4bf83 dest 1:
Wed Oct 05 08:19:04 2011
Process J000 died, see its trace file
kkjcre1p: unable to spawn jobq slave process
Errors in file /u01/product/oracle/diag/rdbms/ccptdb/ccptdb2/trace/ccptdb2_cjq0_14870.trc:
Wed Oct 05 08:30:20 2011
Process J001 died, see its trace file
kkjcre1p: unable to spawn jobq slave process
Errors in file /u01/product/oracle/diag/rdbms/ccptdb/ccptdb2/trace/ccptdb2_cjq0_14870.trc:
Wed Oct 05 08:30:43 2011
Process m000 died, see its trace file
Wed Oct 05 08:41:10 2011
LGWR: Standby redo logfile selected for thread 2 sequence 13235 for destination LOG_ARCHIVE_DEST_2
Thread 2 advanced to log sequence 13235 (LGWR switch)
Current log# 25 seq# 13235 mem# 0: +DATA/ccptdb/onlinelog/redo25a.log
Current log# 25 seq# 13235 mem# 1: +INDX/ccptdb/onlinelog/redo25b.log
Wed Oct 05 08:41:11 2011
Archived Log entry 48434 added for thread 2 sequence 13234 ID 0xbdc4bf83 dest 1:
Wed Oct 05 08:45:24 2011
Process q002 died, see its trace file
Thu Oct 06 09:28:05 2011
Process W000 died, see its trace file
Process J000 died, see its trace file
kkjcre1p: unable to spawn jobq slave process
Errors in file /u01/product/oracle/diag/rdbms/ccptdb/ccptdb2/trace/ccptdb2_cjq0_14870.trc:
Thu Oct 06 09:28:07 2011
Process PZ99 died, see its trace file
Process J000 died, see its trace file
kkjcre1p: unable to spawn jobq slave process
Errors in file /u01/product/oracle/diag/rdbms/ccptdb/ccptdb2/trace/ccptdb2_cjq0_14870.trc:
Process W000 died, see its trace file
Process W000 died, see its trace file
Thu Oct 06 09:28:17 2011
Process W000 died, see its trace file
Process W000 died, see its trace file
Process W000 died, see its trace file
Thu Oct 06 09:28:26 2011
Starting ORACLE instance (normal)
WARNING: You are trying to use the MEMORY_TARGET feature. This feature requires the /dev/shm file system to
be mounted for at least 26239565824 bytes. /dev/shm is either not mounted or is mounted with available
space less than this size. Please fix this so that MEMORY_TARGET can work as expected. Current available is
23652847616 and used is 9901584384 bytes. Ensure that the mount point is /dev/shm for this directory.
memory_target needs larger /dev/shm
Thu Oct 06 09:28:29 2011
Process J000 died, see its trace file
kkjcre1p: unable to spawn jobq slave process
Errors in file /u01/product/oracle/diag/rdbms/ccptdb/ccptdb2/trace/ccptdb2_cjq0_14870.trc:
Thu Oct 06 09:28:30 2011
Process W000 died, see its trace file
Process J000 died, see its trace file
kkjcre1p: unable to spawn jobq slave process
Errors in file /u01/product/oracle/diag/rdbms/ccptdb/ccptdb2/trace/ccptdb2_cjq0_14870.trc:
Process W000 died, see its trace file
Process W000 died, see its trace file
Thu Oct 06 09:28:42 2011
Process W000 died, see its trace file
Starting ORACLE instance (normal)
WARNING: You are trying to use the MEMORY_TARGET feature. This feature requires the /dev/shm file system to
be mounted for at least 26239565824 bytes. /dev/shm is either not mounted or is mounted with available
space less than this size. Please fix this so that MEMORY_TARGET can work as expected. Current available is
23652847616 and used is 9901584384 bytes. Ensure that the mount point is /dev/shm for this directory.
memory_target needs larger /dev/shm
Oracle试图开启实例,按照报错提示,好象是说 /dev/shm 中的free不足memory_target中设置的大小 。
查看节点2 内存使用情况(节点1上的使用内存约为18G, 总物理内存为32G):
top - 09:30:50 up 57 days, 2:08, 7 users, load average: 0.35, 0.77, 0.73
Tasks: 876 total, 1 running, 875 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.3%us, 0.3%sy, 0.0%ni, 98.8%id, 0.1%wa, 0.0%hi, 0.5%si, 0.0%st
Mem: 32948192k total, 31858492k used, 1089700k free, 534552k buffers
Swap: 8193108k total, 0k used, 8193108k free, 16375616k cached
wmrac02<*+ASM2*/home/grid>$free
total used free shared buffers cached
Mem: 32948192 31863292 1084900 0 534628 16382552
-/+ buffers/cache: 14946112 18002080
Swap: 8193108 0 8193108
在节点2上看到的状态信息如下(ora.ccptdb.db在节点2上):
wmrac02<*+ASM2*/home/grid>$crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.DATA.dg ora....up.type ONLINE ONLINE wmrac01
ora.INDX.dg ora....up.type ONLINE ONLINE wmrac01
ora....ER.lsnr ora....er.type ONLINE ONLINE wmrac01
ora....N1.lsnr ora....er.type ONLINE ONLINE wmrac02
ora.asm ora.asm.type ONLINE ONLINE wmrac01
ora.ccptdb.db ora....se.type ONLINE ONLINE wmrac02
ora.eons ora.eons.type ONLINE ONLINE wmrac01
ora.gsd ora.gsd.type OFFLINE OFFLINE
ora....network ora....rk.type ONLINE ONLINE wmrac01
ora.oc4j ora.oc4j.type OFFLINE OFFLINE
ora.ons ora.ons.type ONLINE ONLINE wmrac01
ora....ry.acfs ora....fs.type ONLINE ONLINE wmrac01
ora.scan1.vip ora....ip.type ONLINE ONLINE wmrac02
ora....SM1.asm application ONLINE ONLINE wmrac01
ora....01.lsnr application ONLINE ONLINE wmrac01
ora....c01.gsd application OFFLINE OFFLINE
ora....c01.ons application ONLINE ONLINE wmrac01
ora....c01.vip ora....t1.type ONLINE ONLINE wmrac01
ora....SM2.asm application ONLINE ONLINE wmrac02
ora....02.lsnr application ONLINE ONLINE wmrac02
ora....c02.gsd application OFFLINE OFFLINE
ora....c02.ons application ONLINE ONLINE wmrac02
ora....c02.vip ora....t1.type ONLINE ONLINE wmrac02