In this Document
|
|
Purpose |
|
|
Details |
| Terms |
| IPOIB |
| RDS |
| Architecture |
| Certification & Platform Specific Information |
| Steps for Implementation |
| Known Issues |
|
|
References |
Applies to:
Oracle Database - Enterprise Edition - Version 10.2.0.1 and laterInformation in this document applies to any platform.
Purpose
This document explains the basics of Infiniband/RDS in Oracle Clusterware and RAC environment.
Details
Terms
IPOIB
IP over Infiniband (IPoIB) defines how Internet Protocol utilizes InfiniBand as a Link Layer protocol such as Ethernet. The uses of IPoIB are transparent to TCP/IP based applications.
RDS
Reliable Datagram Sockets (RDS) is a reliable socket off-load driver and inter-processor communication (IPC) protocol with low-overhead, low-latency, high-bandwidth. RDS over InfiniBand uses approximately 50% less CPU per operation than IPoIB and operates with approximately half the latency of User Datagram Protocol (UDP) over Ethernet. The reliable delivery capabilities inherent in InfiniBand that offload end-to-end error checking to the InfiniBand fabric, freeing CPU cycles for application processing, thereby enabling processor scaling greater than is possible with Ethernet.
Architecture
Figure A. shows the traditional model of using UDP over Ethernet for the cluster interconnect in pre-11gR2.
Note: Oracle Clusterware uses UDP in 11gR2 onward
Figure B. shows the RDS model that utilizes RDS over Infiniband fabric for cluster interconnect in pre-11gR2.
Note: Oracle Clusterware uses UDP in 11gR2 onward
Certification & Platform Specific Information
Linux:
Certify page: http://www.oracle.com/technetwork/database/enterprise-edition/tech-generic-linux-new-086754.html
For example of setting up Pre-11gR2 Oracle Clusterware and RAC with Infiniband/RDS, refer to note 943025.1 or the following white paper:
Refer to the following notes for additional information:
note 761804.1 - Oracle Reliable Datagram Sockets (RDS) and InfiniBand (IB) Support (For Linux x86 and x86-64 Platforms)
note 745616.1 - Oracle Reliable Datagram Sockets (RDS) and InfiniBand (IB) Support for RAC Interconnect and Exadata Storage
UNIX(Solaris, AIX, hp-ux):
Certify page: http://www.oracle.com/technetwork/database/clustering/tech-generic-unix-new-166583.html
Infiniband/RDS is not certified for 11gR2 Oracle Clusterware on HP-UX (bug 13777095)
Steps for Implementation
The following are the high level general steps for implementation of RDS over infiniband:
1. After installing Infiniband HCAs (Host Channel Adaptors) and switches, install the required Infiniband drivers and RDS support libraries on each clustered server. Verify with platform specific command that the infiniband interface appears in the available interface list. For example on Linux, using "ifconfig -a" check infiniband interface like ib0 or ib1 is listed.
2. Configure the appropriate failover and load balancing mode for multiple ports of a HCA and/or between multiple HCAs.
3. Before using RDS for RAC interconnect, it is recommended to use
IPoIB for RAC interconnect to check the hardware installation.
make sure in /etc/infiniband/openib.conf on all nodes you find: RDS_LOAD=yes
If value is set to 'no' change this on all nodes to 'yes' and reboot the nodes.
as
we want to use RDS you have to review your UDP related parameters for
best interconnect performance, the recommended values are:
net.core.rmem_max=4194304
net.core.wmem_default=262144
net.core.wmem_max=2097152
4. Once RAC operation is verified over IPoIB, relink with RDS IPC library. To do this, relink the GI home and all related RDBMS homes by performing the following steps as OS home owner.
2) If relinking Grid Infrastructure (GI) home, as root, unlock GI home:
3) As the ORACLE_HOME/GI_HOME owner, go to ORACLE_HOME/GI_HOME and cd to rdbms/lib
4) As the ORACLE_HOME/GI_HOME owner, issue "make -f ins_rdbms.mk ipc_rds ioracle" (Repeat steps 3&4 for each Oracle Home GI&RDBMS)
5) If relinking Grid Infrastructure (GI) home, as root, lock GI home:
Note: If ASM fails to start with error "ORA-27302: failure occurred at: skgxpvifconf", ensure rds service is enabled, i.e. on Solaris:
# svcs -a | grep -i rds
disabled 9:55:05 svc:/system/rds:default
# svcadm enable rds
# svcs -a | grep -i rds
online 14:23:34 svc:/system/rds:default
Note: For non-shared ORACLE_HOME, the above commands must be executed on each node of the cluster.
If
using multiple Oracle home setup (like ASM running from different
Oracle Home or different databases running from different Oracle Homes,
then this step needs to be performed for all the homes.
5. To revert back RAC to use UDP instead of RDS, relink with UDP IPC library. Again, to do this relink the GI home and all related RDBMS homes by performing the following as OS home owner.
2) If relinking Grid Infrastructure (GI) home, as root, unlock GI home:
3) As the ORACLE_HOME/GI_HOME owner, go to ORACLE_HOME/GI_HOME and cd to rdbms/lib
4) As the ORACLE_HOME/GI_HOME owner, issue "make -f ins_rdbms.mk ipc_g ioracle" (Repeat steps 3&4 for each Oracle Home GI&RDBMS)
5) If relinking Grid Infrastructure (GI) home, as root, lock GI home:
Note: For non-shared ORACLE_HOME, the above commands must be executed on each node of the cluster.
If
using multiple Oracle home setup (like ASM running from different
Oracle Home or different databases running from different Oracle Homes,
then this step needs to be performed for all the homes.
To verify what protocol is used by RAC, look in the alert log of the ASM and Database instances during startup:
172.31.88.109
cluster interconnect IPC version:Oracle RDS/IP (generic)
IPC Vendor 1 proto 3
Version 3.0
In the UDP case it would say "UDP/IP" instead of "RDS/IP".