IB_Switch交换机SB7890配置及Band网卡
1.创建软件存放目录
[root@localhost ~]# mkdir -p /soft
2.上传OS镜像，本案例以RHEL7.7为例
mount -o loop /soft/rhel-server-7.7-x86_64-dvd.iso /mnt/
3.配置yum源
[root@localhost ~]# vi /etc/yum.conf
[rheldvd]
name=rhel7.7
baseurl=file:///mnt
enabled=1
gpgcheck=0
[main]
………………………………
4.IB交接机驱动下载或上传已有驱动包至soft目录
https://www.mellanox.com/
驱动包：
MLNX_OFED_LINUX-5.1-2.3.7.1-rhel7.7-x86_64.tgz
5.解压驱动并安装
cd /soft
tar -xvzf MLNX_OFED_LINUX-5.1-2.3.7.1-rhel7.7-x86_64.tgz
6.安装驱动
yum install pciutils perl gtk2 atk cairo gcc-gfortran tcsh openssl-libs libusbx lsof tcl fuse-libs tk -y
cd /soft/MLNX_OFED_LINUX-5.1-2.3.7.1-rhel7.7-x86_64
./mlnxofedinstall
7.Load HCA驱动
/etc/init.d/openibd restart
8.启动子网管理并设置开机启动
/etc/init.d/opensmd start ——对于当前SB7890交换不自子网管理，因此只能在OS端开启子网管理服务，若多个节点建议开1——2个节点即可，不建议全部开启；
/sbin/chkconfig opensmd on
/sbin/chkconfig opensmd --list
9.重启OS
reboot
10.查看IB是否正常识别
ifconfig -a
正常识别如下：
ib0: flags=4163
Infiniband hardware address can be incorrect! Please read BUGS section in ifconfig(8).
infiniband 00:00:10:87:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 txqueuelen 256 (InfiniBand)
RX packets 2 bytes 232 (232.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
ib1: flags=4099
Infiniband hardware address can be incorrect! Please read BUGS section in ifconfig(8).
infiniband 00:00:18:87:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 txqueuelen 256 (InfiniBand)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
11.Band IB配置网络
01. Create bond0 Interface on the server. Update the following file:
touch /etc/sysconfig/network-scripts/ifcfg-bond0
Note: the only meaningful bonding policy in IPoIB is High-Availability (bonding mode number 1,
or "active-backup").
To learn more about the Linux bonding options click here .
Here is an example:
DEVICE=bond0
IPADDR=11.11.11.1
NETWORK=11.11.11.0
NETMASK=255.255.255.0
BROADCAST=11.11.11.255
USERCTL=no
BOOTPROTO=none
NM_CONTROLLED=yes
BONDING_OPTS="mode=active-backup primary=ib0 miimon=100 updelay=100 downdelay=100"
MTU=2044
02.Update/Create ib0 Interface on the server. Update the following file:
touch /etc/sysconfig/network-scripts/ifcfg-ib0
Here is an example:
DEVICE=ib0
TYPE=InfiniBand
NM_CONTROLLED=yes
USERCTL=no
MASTER=bond0
SLAVE=yes
BOOTPROTO=none
PRIMARY=yes
03.Update/Create ib1 Interface on the server. Update the following file:
touch /etc/sysconfig/network-scripts/ifcfg-ib1
Here is an example:
DEVICE=ib1
TYPE=InfiniBand
NM_CONTROLLED=yes
USERCTL=no
MASTER=bond0
SLAVE=yes
BOOTPROTO=none
PRIMARY=no
04.Create/Update the bond.conf file.
touch /etc/modprobe.d/bond.conf
Configure this file as follows:
alias bond0 bonding # Adapter bonding driver
options bond0 max_bonds=2 miimon=100 mode=1 # 100ms fail-over timer. Mode 1 = Active/Backup
05. Restart the network services and the driver.
# /etc/init.d/network restart
# /etc/init.d/openibd restart
06.Verify the bond configuration:
# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.6.0 (September 26, 2009)
Bonding Mode: fault-tolerance (active-backup) (fail_over_mac active)
Primary Slave: ib0 (primary_reselect always)
Currently Active Slave: ib0
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 100
Down Delay (ms): 100
Slave Interface: ib1
MII Status: up
Speed: 40000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: a0:04:03:00:fe:80
Slave queue ID: 0
Slave Interface: ib0
MII Status: up
Speed: 40000 Mbps
Duplex: full
Link Failure Count: 254
Permanent HW addr: a0:04:02:20:fe:80
Slave queue ID: 0
07.Fail Over MAC
Since rdma_cm does not support MAC enslavement, fail_over_mac=1 (or active) should be added
to the bond interface configuration in case of Ethernet interface.
The value active(1) for fail_over_mac policy indicates that the MAC address of the bond should
always be the MAC address of the currently active slave. The MAC address of the slaves is not
changed; instead, the MAC address of the bond changes during a failover.
Changing to fail_over_mac=1 solves this problem as if this not active, the bond MAC jumps from
one interface to another according to the active interface.
Add the bonding options (BONDING_OPT)? "fail_over_mac=1" to /etc/sysconfig/network- scripts/ifcfg-bond0. Note: the default of this parameter is disabled (0).
Here is an example:
DEVICE=bond0
IPADDR=11.11.11.1
NETWORK=11.11.11.0
NETMASK=255.255.255.0
BROADCAST=11.11.11.255
USERCTL=no
BOOTPROTO=none
NM_CONTROLLED=yes
BONDING_OPTS="mode=active-backup primary=ib0 miimon=100 updelay=100 downdelay=100 fail_over_mac=1"
MTU=2044