Oracle  10.2.0.4 ,   4 Nodes RAC,   OCFS2  ， Server : DELL R900 ,  Storage : EMC CX4-480     

昨天发生问题， 系统都运行很慢， 4台节点的负载都基本正常，  节点1 负载稍微高一些， 而且

有缓慢上升的趋势，  i/o 也比平时高一些，  其他节点I/O正常，   怀疑节点1 有问题，  查看所有

节点的 oracle log ,  未发现异常，   这时没有怀疑系统问题，  所以没有去看 Linux log ， 查看了

table lock 方面的信息 ，  发现节点1上的确有很多lock ,  逐一kill 之后，  还是报告说系统运行慢，

节点1 负载有所下降， i/o 也下降一些，  查看ash report ， 发现 gc buffer busy 占top event 的首位，

怀疑心跳线及热点块问题，    其他节点的 ash report 及 awr report 和平时看到的差不多， 都是

cpu time 占第一位，   log file sync 在异常时间段节点1上排第二位 ，  查看log及归档的时间， 发现

有些不正常 ， 日志切换和归档生成的时间相差好几分钟 。   最后才想起来查看linux log ,  看看

是不是硬盘， RAID或网卡的问题 ，  发现下面的一些issue .    最后关闭节点1 的实例，  关闭了好

分钟都关闭不了，  没有办法，  直接reboot 节点1，  结果1分钟后，  其他节点相继都重新启动了 。  

后来系统启动了7, 8 分钟才起来， 等一些服务起来近20分钟过去了，  后来系统恢复正常 。   

1,   google了一下下面的问题，  貌似磁盘有坏块啥的 ， 准备交给DELL或EMC工程师用工具查看一下 。

2， 同事关闭实例没有动静， 直接reboot 节点1 server,   结果其他节点都自动重新启动，  原来也测试

过单独关闭节点1 实例 并reboot linux的情况， 不过都是正常关闭 ， 其他节点都正常运行 。   

Oct 30 04:03:02 mxrac01 syslogd 1.4.1: restart.

Oct 30 16:49:01 mxrac01 auditd[7475]: Audit daemon rotating log files

Oct 31 18:58:00 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b19574 2002.

Oct 31 18:58:01 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:0:1): Abort command issued -- 1 46b19608 2002.

Oct 31 18:59:07 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:0:1): Abort command issued -- 1 46b1a38a 2002.

Oct 31 19:00:11 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b1abe9 2002.

Oct 31 19:00:11 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b1acbe 2002.

Oct 31 19:01:15 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b1add4 2002.

Oct 31 19:01:15 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b1b0ed 2002.

Oct 31 19:02:16 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b1b3b8 2002.

Oct 31 19:02:16 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b1b43f 2002.

Oct 31 19:02:17 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:0:2): Abort command issued -- 1 46b1b498 2002.

Oct 31 19:03:17 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:0:1): Abort command issued -- 1 46b1b711 2002.

Oct 31 19:03:17 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b1b73d 2002.

Oct 31 19:04:19 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:0:1): Abort command issued -- 1 46b1be45 2002.

Oct 31 19:04:19 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:0:2): Abort command issued -- 1 46b1bfc8 2002.

Oct 31 19:04:19 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b1c302 2002.

Oct 31 19:05:20 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b1c513 2002.

Oct 31 19:05:20 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b1c62d 2002.

Oct 31 19:06:22 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:3): Abort command issued -- 1 46b1cc87 2002.

Oct 31 19:06:22 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b1cf83 2002.

Oct 31 19:07:23 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b1d67a 2002.

Oct 31 19:08:27 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b1e4d8 2002.

Oct 31 19:08:27 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:0:1): Abort command issued -- 1 46b1ea92 2002.

Oct 31 19:09:28 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:0:1): Abort command issued -- 1 46b1ebf0 2002.

Oct 31 19:09:29 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b1f496 2002.

Oct 31 19:10:35 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:0:1): Abort command issued -- 1 46b1f544 2002.

Oct 31 19:10:35 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:0:1): Abort command issued -- 1 46b1f9de 2002.

Oct 31 19:10:35 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b1fa59 2002.

Oct 31 19:10:35 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b1faaa 2002.

Oct 31 19:11:41 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b215ab 2002.

Oct 31 19:12:42 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b21bb9 2002.

Oct 31 19:13:50 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b226df 2002.

Oct 31 19:13:50 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:0:1): Abort command issued -- 1 46b227cc 2002.

Oct 31 19:15:00 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b22df4 2002.

Oct 31 19:15:00 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:0:2): Abort command issued -- 1 46b23115 2002.

Oct 31 19:16:01 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b23269 2002.

Oct 31 19:17:01 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b2330c 2002.

Oct 31 19:18:23 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:0:1): Abort command issued -- 1 46b236ea 2002.

Oct 31 19:18:23 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b23b05 2002.

Oct 31 19:19:26 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:0:1): Abort command issued -- 1 46b23f17 2002.

Oct 31 19:20:26 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b26539 2002.

Oct 31 19:22:36 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b27369 2002.

Oct 31 19:23:36 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:3): Abort command issued -- 1 46b27483 2002.

Oct 31 19:25:12 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:0:1): Abort command issued -- 1 46b27df1 2002.

Oct 31 19:25:13 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b27f6f 2002.

Oct 31 19:26:20 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:0:2): Abort command issued -- 1 46b28122 2002.

Oct 31 19:26:20 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b28403 2002.

Oct 31 19:27:50 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b287ab 2002.

Oct 31 19:27:50 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:0:2): Abort command issued -- 1 46b28b83 2002.

Oct 31 19:28:54 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:0:1): Abort command issued -- 1 46b28e17 2002.

Oct 31 19:28:55 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b28eb9 2002.

Oct 31 19:30:08 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:0:1): Abort command issued -- 1 46b2968b 2002.

Oct 31 19:30:08 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b298ca 2002.

Oct 31 19:31:08 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b29bfd 2002.

Oct 31 19:32:08 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:0:2): Abort command issued -- 1 46b29fb1 2002.

Oct 31 19:33:29 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b2a86f 2002.

Oct 31 19:34:29 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b2a98f 2002.

Oct 31 19:35:30 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b2af9b 2002.

Oct 31 19:36:32 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:3): Abort command issued -- 1 46b2b4b3 2002.

Oct 31 19:37:32 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b2b5ec 2002.

Oct 31 19:39:25 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b2be67 2002.

Oct 31 19:39:25 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:0:1): Abort command issued -- 1 46b2bffb 2002.

Oct 31 19:41:42 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b2faf5 2002.

Oct 31 19:43:15 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:0:2): Abort command issued -- 1 46b2fbbe 2002.

Oct 31 19:43:16 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:3): Abort command issued -- 1 46b2fcad 2002.

Oct 31 19:45:34 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:0:1): Abort command issued -- 1 46b3126c 2002.

Oct 31 19:45:34 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b312e1 2002.

Oct 31 19:46:38 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:0:2): Abort command issued -- 1 46b318c7 2002.

Oct 31 19:47:39 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:0:2): Abort command issued -- 1 46b31b7c 2002.

Oct 31 19:54:18 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:3): Abort command issued -- 1 46b37d9a 2002.

Oct 31 19:54:18 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:0:2): Abort command issued -- 1 46b38281 2002.

Oct 31 19:55:18 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:0:2): Abort command issued -- 1 46b382c8 2002.

Oct 31 19:56:52 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:0:1): Abort command issued -- 1 46b38df9 2002.

Oct 31 19:56:52 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b3b42b 2002.

Oct 31 19:57:54 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:0:1): Abort command issued -- 1 46b3bd62 2002.

Oct 31 19:57:54 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b3bd98 2002.

Oct 31 19:59:21 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:0:2): Abort command issued -- 1 46b3bf74 2002.

Oct 31 19:59:21 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b3c2d3 2002.

Oct 31 20:00:21 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:0:1): Abort command issued -- 1 46b3c34e 2002.

Oct 31 20:00:21 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b3c39e 2002.

Oct 31 20:01:25 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b3c41b 2002.

Oct 31 20:01:25 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b3c698 2002.

Oct 31 20:02:25 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b3d207 2002.

Oct 31 20:03:28 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:0:1): Abort command issued -- 1 46b3d7d6 2002.

Oct 31 20:03:29 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b3e3f4 2002.

Oct 31 20:04:30 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:0:1): Abort command issued -- 1 46b3e712 2002.

Oct 31 20:04:30 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b3e8d9 2002.

Oct 31 20:05:35 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:0:2): Abort command issued -- 1 46b3ed87 2002.

Oct 31 20:05:35 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b4012b 2002.

Oct 31 20:06:37 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:0:1): Abort command issued -- 1 46b4103d 2002.

Oct 31 20:06:37 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b411c2 2002.

Oct 31 20:07:40 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:3): Abort command issued -- 1 46b41f5d 2002.

Oct 31 20:07:40 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b4205f 2002.

Oct 31 20:08:41 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:3): Abort command issued -- 1 46b42a74 2002.

Oct 31 20:10:14 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:0:1): Abort command issued -- 1 46b43d8a 2002.

Oct 31 20:10:14 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b43fd8 2002.

Oct 31 20:11:14 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b4401d 2002.

Oct 31 20:12:49 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:0:1): Abort command issued -- 1 46b44dd1 2002.

Oct 31 20:12:49 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b47343 2002.

Oct 31 20:13:49 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:3): Abort command issued -- 1 46b475cd 2002.

Oct 31 20:14:50 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:0:1): Abort command issued -- 1 46b4837c 2002.

Oct 31 20:16:22 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:0:1): Abort command issued -- 1 46b4edc4 2002.

Oct 31 20:16:22 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b525dd 2002.

Oct 31 20:17:29 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:0:1): Abort command issued -- 1 46b52632 2002.

Oct 31 20:17:29 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b53337 2002.

Oct 31 20:17:29 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b53337 2002.

Oct 31 20:18:51 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:0:1): Abort command issued -- 1 46b53675 2002.

Oct 31 20:18:51 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b56ef0 2002.

Oct 31 20:19:51 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:0:1): Abort command issued -- 1 46b570af 2002.

Oct 31 20:20:51 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:0:2): Abort command issued -- 1 46b59aaf 2002.

Oct 31 20:22:06 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b5a342 2002.

Oct 31 20:22:06 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:0:2): Abort command issued -- 1 46b5a57c 2002.

Oct 31 20:23:06 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:0:2): Abort command issued -- 1 46b5a5ad 2002.

Oct 31 20:23:06 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:0:2): Abort command issued -- 1 46b5a5b9 2002.

Oct 31 20:24:06 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b5bebf 2002.

Oct 31 20:25:06 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b5c226 2002.

Oct 31 20:26:35 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b61bda 2002.

Oct 31 20:26:35 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:0:2): Abort command issued -- 1 46b621ca 2002.

Oct 31 20:27:35 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:0:2): Abort command issued -- 1 46b621ed 2002.

Oct 31 20:28:35 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b623f9 2002.

Oct 31 20:29:36 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b62499 2002.

Oct 31 20:29:36 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:0:1): Abort command issued -- 1 46b62618 2002.

Oct 31 20:30:40 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b62ec4 2002.

Oct 31 20:30:40 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:0:1): Abort command issued -- 1 46b63560 2002.

Oct 31 20:31:45 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:0:2): Abort command issued -- 1 46b636f8 2002.

Oct 31 20:31:45 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b64aa7 2002.

Oct 31 20:32:45 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b64af7 2002.

Oct 31 20:34:04 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b65093 2002.

Oct 31 20:34:05 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:0:2): Abort command issued -- 1 46b65854 2002.

Oct 31 20:35:05 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:0:1): Abort command issued -- 1 46b65a0c 2002.

Oct 31 20:36:06 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:0:1): Abort command issued -- 1 46b66a80 2002.

Oct 31 20:36:06 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b66ad7 2002.

Oct 31 20:37:06 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:0:1): Abort command issued -- 1 46b66cd4 2002.

Oct 31 20:38:07 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:0:1): Abort command issued -- 1 46b674ad 2002.

Oct 31 20:39:07 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:0:2): Abort command issued -- 1 46b6787e 2002.

Oct 31 20:39:10 mxrac01 smartd[10401]: smartd received signal 15: Terminated

Oct 31 20:39:10 mxrac01 smartd[10401]: smartd is exiting (exit status 0)

Oct 31 20:39:10 mxrac01 avahi-daemon[9604]: Got SIGTERM, quitting.

Oct 31 20:39:10 mxrac01 avahi-daemon[9604]: Leaving mDNS multicast group on interface eth1.IPv6 with address fe80::215:17ff:fecb:6103.

Oct 31 20:39:10 mxrac01 avahi-daemon[9604]: Leaving mDNS multicast group on interface eth1.IPv4 with address 192.168.1.11.

Oct 31 20:39:10 mxrac01 avahi-daemon[9604]: Leaving mDNS multicast group on interface eth0.IPv6 with address fe80::215:17ff:fecb:6164.

Oct 31 20:39:10 mxrac01 avahi-daemon[9604]: Leaving mDNS multicast group on interface eth0.IPv4 with address 10.13.67.171.

Oct 31 20:39:10 mxrac01 dataeng: dsm_sa_eventmgr32d shutdown succeeded

Oct 31 20:39:17 mxrac01 dataeng: dsm_sa_datamgr32d shutdown succeeded

Oct 31 20:39:45 mxrac01 xinetd[8540]: Exiting...

Oct 31 20:40:08 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:0:1): Abort command issued -- 1 46b67f78 2002.

Oct 31 20:40:08 mxrac01 kernel: qla2xxx 0000:12:00.0: scsi(4:1:0): Abort command issued -- 1 46b67f9a 2002.

Oct 31 20:40:10 mxrac01 hcid[7885]: Got disconnected from the system message bus