环境9iRAC，两个节点
我的archive_dest有两个，所以在archive01目录应该可以看到两个线程生产的归档
SQL> show parameter archive
NAME TYPE
------------------------------------ --------------------------------
VALUE
------------------------------
archive_lag_target integer
0
log_archive_dest string
log_archive_dest_1 string
LOCATION=/opt/oracle/oradata/ylerp/archive01
log_archive_dest_10 string
NAME TYPE
------------------------------------ --------------------------------
VALUE
------------------------------
log_archive_dest_2 string
LOCATION=/opt/oracle/oradata/ylerp/archive02
使用了NFS，在节点1 archive02为nfs
oracle@ndb1:~/oradata/ylerp/archive01> df -h
Filesystem Size Used Avail Use% Mounted on
/dev/cciss/c0d0p3 406G 294G 113G 73% /
tmpfs 2.0G 4.0K 2.0G 1% /dev/shm
/dev/cciss/c0d0p1 118M 42M 76M 36% /boot
10.98.132.2:/opt/oracle/oradata/ylerp/archive02
400G 182G 198G 48% /opt/oracle/oradata/ylerp/archive02
但 rman的list archivelog all 查看到线程2的archive 只有一个
821762 1 160778 A 11-DEC-14 /opt/oracle/oradata/ylerp/archive01/1_160778.dbf
821763 1 160778 A 11-DEC-14 /opt/oracle/oradata/ylerp/archive02/1_160778.dbf
821768 1 160779 A 11-DEC-14 /opt/oracle/oradata/ylerp/archive01/1_160779.dbf
821769 1 160779 A 11-DEC-14 /opt/oracle/oradata/ylerp/archive02/1_160779.dbf
821772 1 160780 A 11-DEC-14 /opt/oracle/oradata/ylerp/archive01/1_160780.dbf
821773 1 160780 A 11-DEC-14 /opt/oracle/oradata/ylerp/archive02/1_160780.dbf
821680 2 134757 A 10-DEC-14 /opt/oracle/oradata/ylerp/archive02/2_134757.dbf
821683 2 134758 A 10-DEC-14 /opt/oracle/oradata/ylerp/archive02/2_134758.dbf
821686 2 134759 A 10-DEC-14 /opt/oracle/oradata/ylerp/archive02/2_134759.dbf
821687 2 134760 A 10-DEC-14 /opt/oracle/oradata/ylerp/archive02/2_134760.dbf
821692 2 134761 A 10-DEC-14 /opt/oracle/oradata/ylerp/archive02/2_134761.dbf
821693 2 134762 A 10-DEC-14 /opt/oracle/oradata/ylerp/archive02/2_134762.dbf
821696 2 134763 A 10-DEC-14 /opt/oracle/oradata/ylerp/archive01/2_134763.dbf
821697 2 134763 A 10-DEC-14 /opt/oracle/oradata/ylerp/archive02/2_134763.dbf
821700 2 134764 A 10-DEC-14 /opt/oracle/oradata/ylerp/archive02/2_134764.dbf
但在文件系统可以看到在目录archive01存在线程2的archive
-rw-r----- 1 oracle oinstall 20710400 2014-12-09 04:31 2_134726.dbf
-rw-r----- 1 oracle oinstall 20966912 2014-12-09 05:06 2_134732.dbf
-rw-r----- 1 oracle oinstall 20971008 2014-12-09 09:20 2_134736.dbf
-rw-r----- 1 oracle oinstall 20971008 2014-12-09 10:16 2_134737.dbf
-rw-r----- 1 oracle oinstall 20971008 2014-12-09 11:18 2_134738.dbf
-rw-r----- 1 oracle oinstall 20971008 2014-12-09 12:44 2_134739.dbf
-rw-r----- 1 oracle oinstall 20967424 2014-12-09 14:30 2_134740.dbf
-rw-r----- 1 oracle oinstall 20971008 2014-12-09 15:34 2_134741.dbf
-rw-r----- 1 oracle oinstall 20971008 2014-12-09 17:00 2_134742.dbf
-rw-r----- 1 oracle oinstall 20971008 2014-12-09 19:17 2_134743.dbf
-rw-r----- 1 oracle oinstall 20962816 2014-12-10 00:58 2_134744.dbf
-rw-r----- 1 oracle oinstall 20963328 2014-12-10 00:58 2_134745.dbf
-rw-r----- 1 oracle oinstall 20969984 2014-12-10 02:43 2_134757.dbf
-rw-r----- 1 oracle oinstall 5375488 2014-12-10 04:09 2_134758.dbf
-rw-r----- 1 oracle oinstall 20971008 2014-12-10 05:11 2_134764.dbf
-rw-r----- 1 oracle oinstall 20971008 2014-12-10 09:11 2_134768.dbf
-rw-r----- 1 oracle oinstall 20970496 2014-12-10 10:12 2_134769.dbf
-rw-r----- 1 oracle oinstall 20969984 2014-12-10 11:07 2_134770.dbf
-rw-r----- 1 oracle oinstall 20971008 2014-12-10 12:40 2_134771.dbf
-rw-r----- 1 oracle oinstall 20971008 2014-12-10 14:31 2_134772.dbf
-rw-r----- 1 oracle oinstall 20971008 2014-12-10 15:34 2_134773.dbf
-rw-r----- 1 oracle oinstall 20971008 2014-12-10 16:58 2_134774.dbf
-rw-r----- 1 oracle oinstall 20967936 2014-12-10 19:13 2_134775.dbf
-rw-r----- 1 oracle oinstall 20971008 2014-12-11 00:59 2_134776.dbf
-rw-r----- 1 oracle oinstall 20970496 2014-12-11 01:00 2_134777.dbf
-rw-r----- 1 oracle oinstall 20971008 2014-12-11 01:06 2_134788.dbf
-rw-r----- 1 oracle oinstall 20971008 2014-12-11 01:58 2_134789.dbf
-rw-r----- 1 oracle oinstall 19174400 2014-12-11 04:28 2_134791.dbf
-rw-r----- 1 oracle oinstall 20968960 2014-12-11 05:16 2_134797.dbf
-rw-r----- 1 oracle oinstall 20970496 2014-12-11 09:19 2_134801.dbf
-rw-r----- 1 oracle oinstall 20971008 2014-12-11 10:22 2_134802.dbf
-rw-r----- 1 oracle oinstall 20971008 2014-12-11 11:20 2_134803.dbf
-rw-r----- 1 oracle oinstall 20971008 2014-12-11 12:56 2_134804.dbf
-rw-r----- 1 oracle oinstall 20971008 2014-12-11 14:37 2_134805.dbf
oracle@ndb1:~/oradata/ylerp/archive01> cd ..
oracle@ndb1:~/oradata/ylerp> cd archive02
oracle@ndb1:~/oradata/ylerp/archive02> ll
total 793664
-rw-r----- 1 oracle oinstall 471552 2014-12-10 02:44 1_160757.dbf
-rw-r----- 1 oracle oinstall 286208 2014-12-10 04:11 1_160758.dbf
-rw-r----- 1 oracle oinstall 1024 2014-12-10 04:11 1_160759.dbf
-rw-r----- 1 oracle oinstall 2560 2014-12-10 04:11 1_160760.dbf
-rw-r----- 1 oracle oinstall 1536 2014-12-10 04:11 1_160761.dbf
-rw-r----- 1 oracle oinstall 2048 2014-12-10 04:11 1_160762.dbf
-rw-r----- 1 oracle oinstall 3584 2014-12-10 04:11 1_160763.dbf
-rw-r----- 1 oracle oinstall 323584 2014-12-10 05:14 1_160764.dbf
-rw-r----- 1 oracle oinstall 1245696 2014-12-10 09:13 1_160765.dbf
-rw-r----- 1 oracle oinstall 1090560 2014-12-10 12:42 1_160766.dbf
-rw-r----- 1 oracle oinstall 1341440 2014-12-10 17:00 1_160767.dbf
-rw-r----- 1 oracle oinstall 2439680 2014-12-11 01:02 1_160768.dbf
-rw-r----- 1 oracle oinstall 3584 2014-12-11 01:02 1_160769.dbf
-rw-r----- 1 oracle oinstall 5632 2014-12-11 01:04 1_160770.dbf
-rw-r----- 1 oracle oinstall 6144 2014-12-11 01:04 1_160771.dbf
-rw-r----- 1 oracle oinstall 291840 2014-12-11 02:00 1_160772.dbf
-rw-r----- 1 oracle oinstall 448000 2014-12-11 04:30 1_160773.dbf
-rw-r----- 1 oracle oinstall 1536 2014-12-11 04:30 1_160774.dbf
-rw-r----- 1 oracle oinstall 2560 2014-12-11 04:30 1_160775.dbf
-rw-r----- 1 oracle oinstall 1536 2014-12-11 04:31 1_160776.dbf
-rw-r----- 1 oracle oinstall 1536 2014-12-11 04:31 1_160777.dbf
-rw-r----- 1 oracle oinstall 2560 2014-12-11 04:31 1_160778.dbf
-rw-r----- 1 oracle oinstall 246784 2014-12-11 05:18 1_160779.dbf
-rw-r----- 1 oracle oinstall 1587712 2014-12-11 10:24 1_160780.dbf
-rw-r----- 1 oracle oinstall 1326080 2014-12-11 14:39 1_160781.dbf
-rw-r----- 1 oracle oinstall 20969984 2014-12-10 02:45 2_134757.dbf
-rw-r----- 1 oracle oinstall 5375488 2014-12-10 04:11 2_134758.dbf
-rw-r----- 1 oracle oinstall 1536 2014-12-10 04:11 2_134759.dbf
-rw-r----- 1 oracle oinstall 2048 2014-12-10 04:11 2_134760.dbf
-rw-r----- 1 oracle oinstall 4608 2014-12-10 04:11 2_134761.dbf
这样的话RMAN不会自动清除目录archive01中的文件
于是查看节点2的alert.log
看到如下错误信息
Errors in file /opt/oracle/admin/ylerp/bdump/ylerp2_arc1_29574.trc:
ORA-19504: failed to create file "/opt/oracle/oradata/ylerp/archive01/2_134804.dbf"
ORA-19504: failed to create file "/opt/oracle/oradata/ylerp/archive01/2_134804.dbf"
ORA-27086: skgfglk: unable to lock file - already in use
Linux Error: 5: Input/output error
Additional information: 8
ARC1: Completed archiving log 2 thread 2 sequence 134804
Thu Dec 11 14:39:50 2014
Thread 2 advanced to log sequence 134806
Current log# 6 seq# 134806 mem# 0: /opt/oracle/oradata/ylerp/redo06.log
Thu Dec 11 14:39:50 2014
ARC0: Evaluating archive log 4 thread 2 sequence 134805
ARC0: Beginning to archive log 4 thread 2 sequence 134805
Creating archive destination LOG_ARCHIVE_DEST_2: '/opt/oracle/oradata/ylerp/archive02/2_134805.dbf'
Creating archive destination LOG_ARCHIVE_DEST_1: '/opt/oracle/oradata/ylerp/archive01/2_134805.dbf'
ARC0: Error 19504 Creating archive log file to '/opt/oracle/oradata/ylerp/archive01/2_134805.dbf'
Thu Dec 11 14:39:50 2014
Errors in file /opt/oracle/admin/ylerp/bdump/ylerp2_arc0_29572.trc:
ORA-19504: failed to create file "/opt/oracle/oradata/ylerp/archive01/2_134805.dbf"
ORA-19504: failed to create file "/opt/oracle/oradata/ylerp/archive01/2_134805.dbf"
ORA-27086: skgfglk: unable to lock file - already in use
Linux Error: 5: Input/output error
由于目录archive01是节点1的本地目录，目录archive02是节点2的本地目录，分别使用nfs加载到节点2和节点1
联想到早几天节点1由于报ORA错误数据库自动down掉，应该是nfs出了问题
ndb2:~ # umount /opt/oracle/oradata/ylerp/archive01/
ndb2:~ # mount -t nfs 10.98.32.1:/opt/oracle/oradata/ylerp/archive01 /opt/oracle/oradata/ylerp/archive01/
mount server reported tcp not available, falling back to udp
mount: RPC: Remote system error - Connection refused
ndb1:~ # /usr/sbin/rpcinfo -p localhost | grep nfs
rpcinfo: can't contact portmapper: RPC: Remote system error - Connection refused
ndb1:/etc/rc.d # ./nfsserver start
Starting kernel based NFS servermount: nfsd already mounted or /proc/fs/nfsd busy
mount: according to mtab, nfsd is already mounted on /proc/fs/nfsd
failed
ndb1:/etc/rc.d # ./nfsserver stop
Shutting down kernel based NFS server failed
ndb1:/etc/rc.d # ./nfsserver start
Starting kernel based NFS server
最后没办法，只能重启了节点1的主机，在节点2重新mount,再查看alert.log，发现归档正常
Creating archive destination LOG_ARCHIVE_DEST_2: '/opt/oracle/oradata/ylerp/archive02/2_134805.dbf'
Creating archive destination LOG_ARCHIVE_DEST_1: '/opt/oracle/oradata/ylerp/archive01/2_134805.dbf'
ARC0: Error 19504 Creating archive log file to '/opt/oracle/oradata/ylerp/archive01/2_134805.dbf'
Thu Dec 11 14:39:50 2014
Errors in file /opt/oracle/admin/ylerp/bdump/ylerp2_arc0_29572.trc:
ORA-19504: failed to create file "/opt/oracle/oradata/ylerp/archive01/2_134805.dbf"
ORA-19504: failed to create file "/opt/oracle/oradata/ylerp/archive01/2_134805.dbf"
ORA-27086: skgfglk: unable to lock file - already in use
Linux Error: 5: Input/output error
Additional information: 8
ARC0: Completed archiving log 4 thread 2 sequence 134805
Thu Dec 11 15:44:13 2014
Reconfiguration started (old inc 7, new inc 8)
List of nodes:
1
Global Resource Directory frozen
one node partition
Communication channels reestablished
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
Resources and enqueues cleaned out
Resources remastered 6439
63268 GCS shadows traversed, 0 cancelled, 4842 closed
29990 GCS resources traversed, 0 cancelled
38614 GCS resources on freelist, 68604 on array, 68604 allocated
set master node info
Submitted all remote-enqueue requests
Update rdomain variables
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
63268 GCS shadows traversed, 0 replayed, 4842 unopened
Submitted all GCS remote-cache requests
0 write requests issued in 58426 GCS resources
1 PIs marked suspect, 0 flush PI msgs
Thu Dec 11 15:44:14 2014
Reconfiguration complete
Post SMON to start 1st pass IR
Thu Dec 11 15:44:14 2014
Instance recovery: looking for dead threads
Instance recovery: lock domain invalid but no dead threads
Thu Dec 11 15:49:08 2014
Thread 2 advanced to log sequence 134807
Current log# 2 seq# 134807 mem# 0: /opt/oracle/oradata/ylerp/redo02.log
Thu Dec 11 15:49:08 2014
ARC1: Evaluating archive log 6 thread 2 sequence 134806
ARC1: Beginning to archive log 6 thread 2 sequence 134806
Creating archive destination LOG_ARCHIVE_DEST_2: '/opt/oracle/oradata/ylerp/archive02/2_134806.dbf'
Creating archive destination LOG_ARCHIVE_DEST_1: '/opt/oracle/oradata/ylerp/archive01/2_134806.dbf'
ARC1: Completed archiving log 6 thread 2 sequence 134806
Thu Dec 11 16:56:07 2014
Reconfiguration started (old inc 8, new inc 9)
List of nodes:
0 1
Global Resource Directory frozen
Communication channels reestablished
Master broadcasted resource hash value bitmaps