一、系统环境
服务器:X86两节点,HP存储
操作系统:redhat6.9
数据库版本:11.2.0.4
二、故障介绍
RAC生产库一号节点物理机宕机后,无法启动实例。二号节点服务正常运行,Dataguard同步正常,未中断。
故障节点硬重启后,GI服务正常启动,但数据库实例启动失败
告警如下:
ARC3 started with pid=38, OS id=9473
ARC1: Archival started
ARC2: Archival started
ARC1: Becoming the 'no FAL' ARCH
ARC1: Becoming the 'no SRL' ARCH
ARC2: Becoming the heartbeat ARCH
LGWR: Primary database is in MAXIMUM AVAILABILITY mode
LGWR: Destination LOG_ARCHIVE_DEST_1 is not serviced by LGWR
ARC3: Archival started
ARC0: STARTING ARCH PROCESSES COMPLETE
Destination LOG_ARCHIVE_DEST_2 is UNSYNCHRONIZED
LGWR: Minimum of 1 applicable standby database required
Errors in file /oracle/app/diag/rdbms/orcl/orcl2/trace/orcl2_lgwr_9361.trc:
ORA-16072: a minimum of one standby database destination is required
LGWR (ospid: 9361): terminating the instance due to error 16072
Thu Apr 14 20:58:24 2022
System state dump requested by (instance=2, osid=9361 (LGWR)), summary=[abnormal instance termination].
System State dumped to trace file /oracle/app/diag/rdbms/orcl/orcl2/trace/orcl2_diag_9335_20220414205824.trc
Dumping diagnostic data in directory=[cdmp_20220414205824], requested by (instance=2, osid=9361 (LGWR)), summary=[abnormal instance termination].
Instance terminated by LGWR, pid = 9361
三、处理过程
可从报错
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
LGWR: Primary database is in MAXIMUM AVAILABILITY mode
LGWR: Destination LOG_ARCHIVE_DEST_1 is not serviced by LGWR
Destination LOG_ARCHIVE_DEST_2 is UNSYNCHRONIZED
LGWR: Minimum of 1 applicable standby database required
Errors in file /oracle/app/diag/rdbms/orcl/orcl2/trace/orcl2_lgwr_9361.trc:
ORA-16072: a minimum of one standby database destination is required
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
中看到当前Dataguard运行在MAXIMUM AVAILABILITY模式,LOG_ARCHIVE_DEST_2为非同步状态,要求到少有一个standby database destination是正常的。当前条件不满足导致实例启动时被LGWR中断。
但此问题比较诡异,当前实例2正常在运行,dataguard同步是正常的,未中断,按道理实例1可正常启动并加入集群。
为尽快恢复业务,临时处理方法如下:
1)将dataguard同步模式降级为MAXIMIZE PERFORMANCE
ALTER DATABASE SET STANDBY TO MAXIMIZE PERFORMANCE;
2)启动故障实例
startup
完成上述修改后,实例1正常启动并加入集群
四、 MAXIMUM AVAILABILITY下实例启动故障分析 :
检查生产DG相关参数如下:
log_archive_config string
log_archive_dest_2 string SERVICE=orclstd LGWR SYNC VALI
D_FOR=(ONLINE_LOGFILES,PRIMARY
_ROLE)
dg_broker_config_file1 string /oracle/app/product/11.2.0/db_
1/dbs/dr1orcl.dat
dg_broker_config_file2 string /oracle/app/product/11.2.0/db_
1/dbs/dr2orcl.dat
五、疑问:
log_archive_config 参数未按规范配置是否是导致MAXIMUM AVAILABILITY模式下RAC其中一节点重启后不能再正常启动的原因?
六、测试:
测试一:
1)在测试环境中,去掉log_archive_config参数,并修改DG同步模式为MAXIMUM AVAILABILITY
2)重启其中一个实例遇到相同故障
测试二:
1)配置上log_archive_config DG_CONFIG=(orcl,orclstd),并修改DG同步模式为MAXIMUM AVAILABILITY
2)重启其中一个实例,可正常启动。
七、总结
初始化参数LOG_ARCHIVE_CONFIG用于控制发送归档日志到远程位置、接收远程归档日志,搭建ADG环境时,可以配置也可以不配置,ADG都可正常运行。
但当同步模式为MAXIMUM AVAILABILITY模式时,
1)RAC环境会遇到实例重启后无法正常启动的情况。
2)无法在线从MAXIMIZE PERFORMANCE切换为MAXIMIZE AVAILABILITY,提示需要mount状态进行模式切换。