一、 说明
1.1 ASMFD 说明
从 12.1.0.2 开始， Oracle 引入了 ASMFD(ASM Filter Driver) ， ASMFD 支持 Linux x86_64 和 Oracle Solaris 平台。安装完 Grid Infrastructure 后，你可以决定是否配置它。如果之前使用了 ASMLIB 或者 UDEV ，迁移到 ASMFD 之后，需要卸载 ASMLIB 或禁用 udev 的规则。通过 Filter driver 可以过滤无效的请求，避免因为非 oracle 的 I/O 请求造成意外的覆写，进而保证了系统的安全和稳定。
1.2 测试环境描述
操作系统版本： Red Hat Enterprise Linux Server release 7.5 (Maipo)
数据库版本： 19.3.0.0.0
数据库架构： RAC
ASM 磁盘绑定方式： UDEV
原 UDEV 绑定磁盘信息：
select name,path,state from v$asm_disk;
NAME PATH STATE
------------------------------------------------------------------------
DATA2_0000 /dev/asmdisk1 NORMAL
DATA_0001 /dev/asmdisk3 NORMAL
DATA_0000 /dev/asmdisk4 NORMAL
DATA2_0001 /dev/asmdisk2 NORMAL
DATA_0002 /dev/asmdisk5 NORMAL
sdb sdc sdd 为测试添加的磁盘：
Disk /dev/sdc: 10.7 GB, 10737418240 bytes, 20971520 sectors
Disk /dev/sdb: 10.7 GB, 10737418240 bytes, 20971520 sectors
Disk /dev/sdf: 85.9 GB, 85899345920 bytes, 167772160 sectors
Disk /dev/sdh: 21.5 GB, 21474836480 bytes, 41943040 sectors
Disk /dev/sdd: 10.7 GB, 10737418240 bytes, 20971520 sectors
Disk /dev/sda: 107.4 GB, 107374182400 bytes, 209715200 sectors
Disk /dev/sdg: 21.5 GB, 21474836480 bytes, 41943040 sectors
Disk /dev/sde: 85.9 GB, 85899345920 bytes, 167772160 sectors
Disk /dev/sdi: 21.5 GB, 21474836480 bytes, 41943040 sectors
二、安装ASMFD
2.1 查看ASMFD是否安装
[root@rac1 ~]# su – grid
[grid@rac1 ~]$ asmcmd afd_state
ASMCMD-9530: The AFD state is 'NOT INSTALLED'
2.2 安装ASMFD
一二节点均需安装
1 使用 root 用户关闭 CRS ： crsctl stop crs
2 使用 root 用户安装 ASMFD ：
[root@rac1 ~]# export ORACLE_BASE=/oracle/gridbase
[root@rac1 ~]# export ORACLE_HOME=/oracle/grid/crs_1
[root@rac1 ~]# /oracle/grid/crs_1/bin/asmcmd afd_configure
AFD-627: AFD distribution files found.
AFD-634: Removing previous AFD installation.
AFD-635: Previous AFD components successfully removed.
AFD-9294: updating file /etc/sysconfig/oracledrivers.conf
AFD-636: Installing requested AFD software.
AFD-637: Loading installed AFD drivers.
AFD-9321: Creating udev for AFD.
AFD-9323: Creating module dependencies - this may take some time.
AFD-9154: Loading 'oracleafd.ko' driver.
AFD-649: Verifying AFD devices.
AFD-9156: Detecting control device '/dev/oracleafd/admin'.
AFD-638: AFD installation correctness verified.
Modifying resource dependencies - this may take some time.
2.3 检查ASMFD的状态
[root@rac1 ~]# /oracle/grid/crs_1/bin/asmcmd afd_state
ASMCMD-9526: The AFD state is 'LOADED' and filtering is 'ENABLED' on host 'rac1'
[grid@rac1 ~]$ crsctl status res -t -init
ora.driver.afd
1 ONLINE ONLINE rac1 STABLE
三、使用afd_label 标识磁盘
3.1 查看afd_diskstring
[root@rac1 dev]# more /etc/oracleafd.conf
afd_diskstring='/dev/asm*'
3.2 使用udev绑定磁盘
vi 99-oracle-asmdevices.rules
KERNEL=="sd*[!0-9]", ENV{DEVTYPE}=="disk", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d $devnode", RESULT=="0QEMU_QEMU_HARDDISK_drive-scsi8", RUN+="/bin/sh -c 'mknod /dev/asmfd1 b $major $minor; chown grid:oinstall /dev/asmfd1; chmod 0660 /dev/asmfd1'"
KERNEL=="sd*[!0-9]", ENV{DEVTYPE}=="disk", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d $devnode", RESULT=="0QEMU_QEMU_HARDDISK_drive-scsi7", RUN+="/bin/sh -c 'mknod /dev/asmfd2 b $major $minor; chown grid:oinstall /dev/asmfd2; chmod 0660 /dev/asmfd2'"
KERNEL=="sd*[!0-9]", ENV{DEVTYPE}=="disk", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d $devnode", RESULT=="0QEMU_QEMU_HARDDISK_drive-scsi6", RUN+="/bin/sh -c 'mknod /dev/asmfd3 b $major $minor; chown grid:oinstall /dev/asmfd3; chmod 0660 /dev/asmfd3'"
重启udev
/sbin/udevadm trigger --type=devices --action=change
3.3 使用afd_label标识磁盘
[grid@rac1 ~]$ asmcmd afd_label ASMFD1 /dev/asmfd1
[grid@rac1 ~]$ asmcmd afd_label ASMFD2 /dev/asmfd2
[grid@rac1 ~]$ asmcmd afd_label ASMFD3 /dev/asmfd3
[grid@rac1 ~]$ asmcmd afd_lsdsk
------------------------------------------------------------------------
Label Filtering Path
=========================================================================
ASMFD1 ENABLED /dev/asmfd1
ASMFD2 ENABLED /dev/asmfd2
ASMFD3 ENABLED /dev/asmfd3
/etc/udev/rules.d/53-afd.rules
[root@rac1 rules.d]# more 53-afd.rules
#
# AFD devices
KERNEL=="oracleafd/.*", OWNER="grid", GROUP="asmadmin", MODE="0770"
KERNEL=="oracleafd/*", OWNER="grid", GROUP="asmadmin", MODE="0770"
KERNEL=="oracleafd/disks/*", OWNER="grid", GROUP="asmadmin", MODE="0660"
若标识错误，使用afd_unlabel 移除标识的磁盘
[grid@rac1 ~]$ asmcmd afd_unlabel ASMFD1
[grid@rac1 ~]$ asmcmd afd_unlabel ASMFD2
[grid@rac1 ~]$ asmcmd afd_unlabel ASMFD3
在另外的节点中，不再需要作 label ，而是直接 scan 即可。
[grid@rac1 ~]$ asmcmd afd_scan
[grid@rac1 ~]$ asmcmd afd_lsdsk
四、创建新的ASM 磁盘组
4.1 图形化界面方式
修改 asm_diskstring
4.2 命令行方式
[grid@rac1 ~]$ sqlplus / as sysasm
show parameter string
NAME TYPE VALUE
------------------------------------ --------------------------
asm_diskstring string /dev/asm*, AFD:*
select name,path,state from v$asm_disk;
NAME PATH STATE
------------ ------------------ --------
/dev/asmfd2 NORMAL
AFD:ASMFD1 NORMAL
AFD:ASMFD2 NORMAL
/dev/asmfd1 NORMAL
/dev/asmfd3 NORMAL
AFD:ASMFD3 NORMAL
DATA2_0000 /dev/asmdisk1 NORMAL
DATA_0001 /dev/asmdisk3 NORMAL
DATA_0000 /dev/asmdisk4 NORMAL
DATA2_0001 /dev/asmdisk2 NORMAL
DATA_0002 /dev/asmdisk5 NORMAL
11 rows selected.
CREATE DISKGROUP DATA3 EXTERNAL REDUNDANCY DISK 'AFD:ASMFD1';
alter diskgroup DATA3 add disk 'AFD:ASMFD2';
alter diskgroup DATA3 add disk 'AFD:ASMFD3';
select name,path,state from v$asm_disk;
NAME PATH STATE
------------- ------------------ --------
/dev/asmfd2 NORMAL
/dev/asmfd1 NORMAL
/dev/asmfd3 NORMAL
DATA2_0000 /dev/asmdisk1 NORMAL
DATA_0001 /dev/asmdisk3 NORMAL
DATA_0000 /dev/asmdisk4 NORMAL
DATA2_0001 /dev/asmdisk2 NORMAL
DATA_0002 /dev/asmdisk5 NORMAL
ASMFD1 AFD:ASMFD1 NORMAL
ASMFD2 AFD:ASMFD2 NORMAL
ASMFD3 AFD:ASMFD3 NORMAL
五、Filter 功能
查看 filter 功能是否启用
[grid@rac1 ~]$ asmcmd afd_lsdsk
--------------------------------------------------------------------
Label Filtering Path
===================================================================
ASMFD1 ENABLED /dev/asmfd1
ASMFD2 ENABLED /dev/asmfd2
ASMFD3 ENABLED /dev/asmfd3
在启用 Filter 功能下， dd 磁盘测试
[root@rac1 dev]# dd if=/dev/zero of=/dev/asmfd1
dd: writing to ‘/dev/asmfd1’: No space left on device
20971521+0 records in
20971520+0 records out
10737418240 bytes (11 GB) copied, 931.568 s, 11.5 MB/s
通过 strings 查看 /dev/sdb ，可以发现，里面的内容并没有被清空
[root@rac1 dev]# strings /dev/asmfd1
ORCLDISKASMFD1
ASMFD1
DATA3
ASMFD1
ORCLDISKASMFD1
ASMFD1
DATA3
ASMFD1
……
卸载、挂载磁盘组测试
SQL> select NAME,STATE from v$asm_diskgroup;
NAME STATE
------------------------------ -----------
DATA2 MOUNTED
DATA MOUNTED
DATA3 MOUNTED
SQL> alter diskgroup DATA3 dismount;
Diskgroup altered.
SQL> alter diskgroup DATA3 mount;
Diskgroup altered.
/var/log/messages 里显示的错误信息
Mar 4 14:31:43 rac1 kernel: buffer_io_error: 17144 callbacks suppressed
Mar 4 14:31:43 rac1 kernel: Buffer I/O error on dev sdb, logical block 2560823, lost async page write
Mar 4 14:31:43 rac1 kernel: Buffer I/O error on dev sdb, logical block 2560824, lost async page write
Mar 4 14:31:43 rac1 kernel: Buffer I/O error on dev sdb, logical block 2560825, lost async page write
六、附录
参考文档：
Grid Infrastructure Installation and Upgrade Guide for Linux
Automatic Storage Management Administrator's Guide
http://docs.oracle.com/database/121/OSTMG/GUID-2F5E344F-AFC2-4768-8C00-6F3C56302123.htm#OSTMG95729
http://docs.oracle.com/database/121/OSTMG/GUID-BB2B3A64-4B83-4A6D-816C-6472FAF9B27A.htm#OSTMG95909