Configure Direct NFS(DNFS) on RHELinux6.4 (11R2)

   In oracle 11g releases,oracle introduce a new feature,this is Direct NFS.Like as oracle 10g DIO,which  implies faster performance and more scalable access to NFS storage located on NAS storage devices (accessible over TCP/IP). Direct NFS is built directly into the database kernel - just like ASM which is mainly used when using DAS or SAN storage as client.and,Direct NFS is provided as part of the database kernel, and is thus available on all supported database platforms - even those that don't support NFS natively, like Windows.
  Direct NFS provides faster performance than what can be provided by the operating system's NFS driver as Oracle bypasses the operating system and generates exactly the requests it needs (no user configuration or tuning required). Data is cached just once in user space, which saves memory (no second copy in kernel space). Performance is further improved by load balancing across multiple network interfaces (if available).
  Direct NFS Client can consume upto four network paths defined for an NFS server. The Direct NFS Client performs load balancing across all specified paths. If a specified path fails, then Direct NFS Client reissues I/O commands over any remaining paths.Direct NFS Client requires an NFS server supporting NFS read/write buffers of at least 16384 bytes.Direct NFS Client issues writes at wtmax granularity to the NFS server. Direct NFS Client does not serve an NFS server with awtmax less than 16384. Oracle recommends that you use the value 32768.

   Oracle Database11g数据库管理艺术描述说明：

Direct NFS Client can use a new configuration file or the mount tab file (/etc/mtab on Linux) to determine the mount

point settings for NFS storage devices.

This file is required only for configuring the Direct NFS for load balancing and specfic to single database.You can

still enable the Direct NFS without configuring oranfstab file.DNFS will take mount point settings for NFS from

/etc/mtab on Linux

In RAC,the oranfstab must be configured on all nodes and keep /etc/oranfstab file synchronized on all nodes.

(When the oranfstab file is placed in $ORACLE_HOME/dbs, the entries in the file are specific to a single database.

In this case, all nodes running an Oracle RAC database use the same ORACLE_HOME/dbs/oranfstab file.

When the oranfstab file is placed in /etc, then it is globally available to all Oracle databases, and can contain

mount points used by all Oracle databases running on nodes in the cluster, including single-instance databases.

However, on Oracle RAC systems, if the oranfstab file is placed in /etc, then you must replicate the file

/etc/oranfstab file on all nodes, and keep each /etc/oranfstab file synchronized on all nodes, just as you must with the

/etc/fstab file.

oracle dnfs client按照如下顺序寻找部署安装点设置的：
1）ORACLE_HOME\dbs\oranfstab（To enable Direct NFS Client, a new Oracle specific file oranfstab can be added toORACLE_HOME\dbs）；
2）/etc/oranfstab;
3) /etc/mtab.

My test env：
sever：135.149.96.152；
client：135.149.96.153；

Database:

Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production

PL/SQL Release 11.2.0.4.0 - Production

CORE 11.2.0.4.0 Production

TNS for Linux: Version 11.2.0.4.0 - Production

NLSRTL Version 11.2.0.4.0 - Production

OS:

[root@ORACLERAC2 ~]# lsb_release -a

LSB Version: :base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch

Distributor ID: RedHatEnterpriseServer

Description: Red Hat Enterprise Linux Server release 6.4 (Santiago)

Release: 6.4

Codename: Santiago
1 创建nfs挂载目录：
client：
[root@ORACLERAC2 ~]# cd /home
[root@ORACLERAC2 home]# ls
grid lost+found oracle zycjk
[root@ORACLERAC2 home]# mkdir -p dnfs
[root@ORACLERAC2 home]# pwd
[root@ORACLERAC2 ~]# chown -R oracle:oinstall /home/dnfs

[root@ORACLERAC2 ~]# service rpcbind status

rpcbind (pid 1846) is running...

#####################################################

由于在6系列里面，portmap已经改名了，6系列需要使用 service rpcbind start启动

#####################################################

[root@ORACLERAC2 ~]# /etc/init.d/nfs start

Starting NFS services: [ OK ]

Starting NFS quotas: [ OK ]

Starting NFS mountd: [ OK ]

Stopping RPC idmapd: [ OK ]

Starting RPC idmapd: [ OK ]

Starting NFS daemon: [ OK ]

[root@ORACLERAC2 ~]#

[root@ORACLERAC2 ~]# /etc/init.d/nfs status

rpc.svcgssd is stopped

rpc.mountd (pid 28460) is running...

nfsd (pid 28525 28524 28523 28522 28521 28520 28519 28518) is running...

rpc.rquotad (pid 28456) is running...

[root@ORACLERAC2 ~]#

server：
[root@ORACLERAC1 home]# pwd
/home
[root@ORACLERAC1 oradata]# cd /home
[root@ORACLERAC1 home]# mkdir -p orashare
[root@ORACLERAC1 home]# vi /etc/exports
/home/orashare 135.149.96.153(rw,sync,no_wdelay,insecure,insecure_locks,no_root_squash)
[root@ORACLERAC1 ~]# exportfs -rv
exporting 135.149.96.153:/home/orashare
[root@ORACLERAC1 ~]#
[root@ORACLERAC1 ~]# exportfs
/home/orashare 135.149.96.153
[root@ORACLERAC1 ~]# service nfs status
rpc.svcgssd is stopped
rpc.mountd (pid 24238) is running...
nfsd (pid 24303 24302 24301 24300 24299 24298 24297 24296) is running...
rpc.rquotad (pid 24234) is running...
[root@ORACLERAC1 ~]# chkconfig --level 345 nfs on

2 enable Direct NFS（client153）:

[oracle@ORACLERAC2 ~]$ cd $ORACLE_HOME/lib;
[oracle@ORACLERAC2 lib]$ pwd
/u01/app/oracle/product/11.2.0/lib
[oracle@ORACLERAC2 lib]$ mv libodm11.so libodm11.so_bak
[oracle@ORACLERAC2 lib]$ ln -s libnfsodm11.so libodm11.so
[oracle@ORACLERAC2 lib]$ pwd
/u01/app/oracle/product/11.2.0/lib

我的环境没有oranfstab，所以我先测试时用 /etc/mtab配置，我的环境里原配置信息如下：
[oracle@ORACLERAC2 etc]$ cat mtab
/dev/mapper/rootvg-lv01 / ext4 rw 0 0
proc /proc proc rw 0 0
sysfs /sys sysfs rw 0 0
devpts /dev/pts devpts rw,gid=5,mode=620 0 0
tmpfs /dev/shm tmpfs rw,rootcontext="system_u:object_r:tmpfs_t:s0" 0 0
/dev/sda1 /boot ext4 rw 0 0
/dev/mapper/rootvg-lv02 /home ext4 rw 0 0
/dev/mapper/rootvg-lv06 /opt ext4 rw 0 0
/dev/mapper/rootvg-lv03 /tmp ext4 rw 0 0
/dev/mapper/rootvg-lv07 /u01 ext4 rw 0 0
/dev/mapper/rootvg-lv04 /usr ext4 rw 0 0
/dev/mapper/rootvg-lv05 /var ext4 rw 0 0
none /proc/sys/fs/binfmt_misc binfmt_misc rw 0 0
sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw 0 0
oracleasmfs /dev/oracleasm oracleasmfs rw 0 0

server:135.149.96.152
path:135.149.96.152
local:135.149.96.153
export:/home/orashare mount:/home/dnfs
[oracle@ORACLERAC2 etc]$
测试没有识别，所以在oracle home继续配置首要ORACLE_HOME\dbs\oranfstab路径：
[oracle@ORACLERAC2 ~]$ cd $ORACLE_HOME/dbs
[oracle@ORACLERAC2 dbs]$ ls
hc_PROD1.dat hc_PROD2.dat init.ora initPROD1.ora lkPROD1 lkPROD2 orapwPROD1 orapwPROD2 spfilePROD1.ora spfilePROD2.ora tbs.sql tmp.sql
[oracle@ORACLERAC2 dbs]$ vi oranfstab

server:135.149.96.152
path:135.149.96.152
local:135.149.96.153
export:/home/orashare mount:/home/dnfs

3 Client mount DNFS：
[root@ORACLERAC2 ~]# vi /etc/fstab
135.149.96.152:/home/orashare /home/dnfs nfs rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,actimeo=0,vers=3,timeo=600 0 0
#
[root@ORACLERAC2 oradata]# mount 135.149.96.152:/home/orashare /home/dnfs
[root@ORACLERAC2 oradata]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/rootvg-lv01
15G 393M 14G 3% /
tmpfs 7.8G 0 7.8G 0% /dev/shm
/dev/sda1 194M 33M 152M 18% /boot
/dev/mapper/rootvg-lv02
9.9G 841M 8.6G 9% /home
/dev/mapper/rootvg-lv06
7.9G 148M 7.4G 2% /opt
/dev/mapper/rootvg-lv03
9.9G 154M 9.2G 2% /tmp
/dev/mapper/rootvg-lv07
50G 27G 21G 57% /u01
/dev/mapper/rootvg-lv04
7.9G 3.3G 4.3G 44% /usr
/dev/mapper/rootvg-lv05
7.9G 255M 7.3G 4% /var
135.149.96.152:/home/orashare
9.9G 151M 9.2G 2% /home/dnfs

4 Client（create tbs test）：

4.1顺带检查一下端口：
[root@ORACLERAC2 ~]# sudo netstat -an |grep 2049
tcp 1 0 135.149.96.153:13289 135.149.96.152:2049 CLOSE_WAIT
tcp 1 0 135.149.96.153:48545 135.149.96.152:2049 CLOSE_WAIT
tcp 0 0 135.149.96.153:835 135.149.96.152:2049 ESTABLISHED
tcp 1 0 135.149.96.153:16083 135.149.96.152:2049 CLOSE_WAIT

restart db：
SQL> shut immediate
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> startup
ORACLE instance started.

Total System Global Area 847630336 bytes
Fixed Size 2257960 bytes
Variable Size 301992920 bytes
Database Buffers 536870912 bytes
Redo Buffers 6508544 bytes
Database mounted.
Database opened.
#################################################################
库启动时对应的dnfs配置生效日志片段：
Oracle instance running with ODM: Oracle Direct NFS ODM Library Version 3.0
Thu Aug 17 10:36:20 2017
#################################################################
SQL> select * from v$dnfs_servers;
no rows selected

SQL> create tablespace dbfs_tbs
2 datafile '/home/dnfs/test_tbs01.dbf' size 10M;
SQL>
Tablespace created.

#################################################################
创建测试dnfs表空间时生效日志片段：
create tablespace dbfs_tbs
datafile '/home/dnfs/test_tbs01.dbf' size 10M
ORA-1119 signalled during: create tablespace dbfs_tbs
datafile '/home/dnfs/test_tbs01.dbf' size 10M...
Thu Aug 17 10:42:58 2017
create tablespace dbfs_tbs
datafile '/home/dnfs/test_tbs01.dbf' size 10M
Thu Aug 17 10:42:58 2017
Direct NFS: channel id [0] path [135.149.96.152] to filer [135.149.96.152] via local [135.149.96.153] is UP
Direct NFS: channel id [1] path [135.149.96.152] to filer [135.149.96.152] via local [135.149.96.153] is UP
Completed: create tablespace dbfs_tbs
datafile '/home/dnfs/test_tbs01.dbf' size 10M

#################################################################

SQL> select * from v$dnfs_servers;

ID SVRNAME DIRNAME MNTPORT NFSPORT WTMAX RTMAX
---------- -------------------------------------------------- -------------------------------------------------- ---------- ---------- ---------- ----------
1 135.149.96.152 /home/orashare 45069 2049 1048576 1048576
SQL> select * from v$dnfs_files;

FILENAME FILESIZE PNUM SVR_ID
-------------------------------------------------- ---------- ---------- ----------
/home/dnfs/test_tbs01.dbf 10493952 10 1

SQL> select * from v$dnfs_channels;

PNUM SVRNAME PATH LOCAL CH_ID SVR_ID SENDS RECVS PINGS RECO
---------- -------------------------------------------------- ------------------------------ -------------------------------------------------- ---------- ---------- ---------- ---------- ---------- ----------
10 135.149.96.152 135.149.96.152 135.149.96.153 0 1 0 0 0 0
10 135.149.96.152 135.149.96.152 135.149.96.153 1 1 0 0 0 0
30 135.149.96.152 135.149.96.152 135.149.96.153 0 1 0 0 0 0

4.2创建的数据文件client和server端都完全一致：

Client：

[root@ORACLERAC2 home]# cd dnfs

[root@ORACLERAC2 dnfs]# ls

test_tbs01.dbf test_tbs02.dbf test_tbs03.dbf test_tbs04.dbf

[root@ORACLERAC2 dnfs]# ls -lart

total 41000

drwxr-xr-x. 7 root root 4096 Aug 17 09:51 ..

-rw-r-----. 1 oracle oinstall 10493952 Aug 17 11:39 test_tbs01.dbf

-rw-r-----. 1 oracle oinstall 10493952 Aug 17 11:41 test_tbs02.dbf

-rw-r-----. 1 oracle oinstall 10493952 Aug 17 11:41 test_tbs03.dbf

drwxr-xr-x. 2 oracle oinstall 4096 Aug 17 15:17 .

-rw-r-----. 1 oracle oinstall 10493952 Aug 17 15:17 test_tbs04.dbf

[root@ORACLERAC2 dnfs]# pwd

/home/dnfs

server：

[root@ORACLERAC1 home]# cd orashare

[root@ORACLERAC1 orashare]# ls -lart

total 41000

drwxr-xr-x. 7 root root 4096 Aug 17 09:34 ..

-rw-r-----. 1 oracle oinstall 10493952 Aug 17 11:39 test_tbs01.dbf

-rw-r-----. 1 oracle oinstall 10493952 Aug 17 11:41 test_tbs02.dbf

-rw-r-----. 1 oracle oinstall 10493952 Aug 17 11:41 test_tbs03.dbf

drwxr-xr-x. 2 oracle oinstall 4096 Aug 17 15:17 .

-rw-r-----. 1 oracle oinstall 10493952 Aug 17 15:17 test_tbs04.dbf

[root@ORACLERAC1 orashare]# pwd

/home/orashare

Reference：
1 http://docs.oracle.com/cd/E11882_01/install.112/e47798/postcfg.htm#NTDBI0451---DNFS；
2 http://www.orafaq.com/wiki/Direct_NFS；
3 Step by Step – Configure Direct NFS Client (DNFS) on Linux (11g) [ID 762374.1]

4 Oracle Database11g数据库管理艺术。