Broadcom BCM57412 NetXtreme-E 10Gb SFP+ Driver

# Broadcom NetXtreme-E Linux Installer ## QUICKSTART: Single Host Installation with Default Configuration The install.sh script in this directory can be used to install drivers, libraries, and firmware, as well as configure IP address and RoCE congestion control on a single host. To install all software components, set up RoCE congestion control with default settings, and configure an IP address, issue the following command: bash install.sh -i <interface name or PCIe address> -a <ip address> -n <netmask> **Note:** that for RoCE to function properly in the case of any congestion (many-to-one flows, link speed changes, etc) your network switch must be configured to match the host congestion control settings. Example settings are output at the end of the installation process. ## Introduction This tool automates installation of the Broadcom NetXtreme Ethernet and RoCE drivers, as well as manages installing firmware for Broadcom NetXtreme network controllers. The installer has two modes: - Single Host: The install.sh script can be run on a single system to configure that host only - Cluster Mode: A list of servers can be specified for installation from a remote host The following changes will be applied to your systems: - Install bnxt_en Ethernet driver - Install bnxt_re RoCE driver - Install libbnxt_re RDMA verbs library - Update device firmware - Configure IP address if requested - Configure RoCE congestion control if RoCE is installed Driver modules will be installed in this priority: - RedHat/CentOS: DKMS, Source RPM, Binary RPM, Source - SuSE: Signed RPM if secureboot, DKMS, Source RPM, Binary RPM, Source - Ubuntu/Debian: DKMS, Source ## Requirements ### Supported operating systems See the Supported Operating Systems for Ethernet NIC Controllers page for the full list of supported operating systems and platforms: https://techdocs.broadcom.com/us/en/storage-and-ethernet-connectivity/ethernet-nic-controllers/bcm957xxx/1-0/software-installation/supported-operating-systems.html ### Supported Devices The following Broadcom NetXtreme-E devices are fully supported by this tool: - Whitney+: [BCM9574xx](https://www.broadcom.com/products/ethernet-connectivity/network-adapters/25gb-nic-ocp) - Thor: [BCM9575xx](https://www.broadcom.com/products/ethernet-connectivity/network-adapters/100gb-nic-ocp) All speeds, and cable/transceiver types supported by the above devices are supported by this tool. ### Software Requirements The installer will attempt to automatically install all required packages using the native package management system. This requires that subscription-based systems have package repositories enabled: - RedHat: BaseOS - SuSE: PackageHub If previous versions of Broadcom drivers are installed, potential conflicts between versions can be avoided by running the included uninstall.sh script, or removing all currently installed packages named bnxt* and libbnxt_re. ### SuSE For unregistered SuSE systems: - Set enabled=1 in all files in /etc/zypp/repos.d/ - Install ansible with: pip3 install ansible For SuSE systems with Secure Boot enabled, follow the steps on this page to import the Partner Linux Driver Program key: https://drivers.suse.com/doc/Usage/Secure_Boot_Certificate.html ## Single Host Installation Before starting installation, please enable your distribution's package repositories. The installer will make use of these to install prerequisite packages. Without them, the installation may fail, or may need to fall back to installation from source, rather than source or DKMS packages. The install.sh script is the best tool for installing all software on a single host. The simplest valid command is: bash install.sh -i <interface name or PCIe address> This will install the software found distributed with the installer to the specified network device, upgrade the device firmware, log the installation process details to install.log, and on successful completion, output the configured values and example network switch configuration. The default installation values are: - Software packages: bnxt_en / bnxt_re drivers, libbnxt_re library, bnxtqos and bnxtnvm utilities - Firmware for the selected device - RoCE enabled, with PFC and ECN - RoCE traffic tagged with DSCP value 26, on priority 3 - RoCE CNP packets tagged with DSCP value 48, on priority 7 See `install -h` output for all options. The most common are: - `-a <address> -n <netmask>`: Specify an IP address and netmask for the device being configured - `-v`: Verbose mode. Output installation log while running - `-m <MTU>`: Specify the device MTU. Default is 1500 - `-2`: Install Ethernet driver only. Do not install/configure RoCE - `-q <VID>`: Use Ethernet VLAN tag VID rather than DSCP - `-f`: Force installation, even if this causes a downgrade - `-s <source>`: Specify the software source. Either URL or directory name may be used. Default is `../../../`as the installer is assumed to be packaged with the source files To configure additional interfaces, the install.sh script can be re-run with the additional interface name specified with `-i`. ## Cluster Installation Installation and configuration of multiple servers can be accomplished with the cluster installation mode of this installer. This is accomplished by setting up a management host with [Ansible](https://www.ansible.com/), and creating an inventory file containing the details of the target hosts, and configuration. ### Management Host Setup Most often, the automation scripts, based on Ansible (www.ansible.org) are run from a management host, rather than one of the servers to be managed, due to the fact that connectivity may be interrupted. #### Management Host Requirements - Python 3: Some host package systems (RHEL 8+) no longer support Python2. Use python3 for best compatibility. - pyghmi Python module, if IPMI power control is to be used #### Ansible Installation The recommended way to install Ansible is via the system package manager: Yum systems (CentOS): yum install ansible Apt systems (Debian, Ubuntu): apt install ansible RedHat Enterprise Linux: subscription-manager repos --enable ansible-2.9-for-rhel-8-x86_64-rpms dnf install ansible It is also possible to install Ansible Python "pip" package manager. This can easily be done within a new Python virtual environment with commands like the following: virtualenv --system-site-packages ~/ansible source ~/ansible/bin/activate python3 -m pip install ansible #### pyghmi Installation If "boot_type=cold" will be used to power cycle systems after installing new firmware, the python pyghmi module must be installed. Python virtualenv: pip install pyghmi Apt systems: apt install python3-pyghmi Yum systems: There is no OS-provided package. Use a package from the RHEL OSP (OpenStack Platform) or, for CentOS, the RDO repositories (example below): yum install http://mirror.centos.org/centos/7/cloud/x86_64/openstack-stein/python2-cryptography-2.5-1.el7.x86_64.rpm http://mirror.centos.org/centos/7/cloud/x86_64/openstack-stein/python2-pyghmi-1.2.16-1.el7.noarch.rpm ### Target Host Setup Ansible uses ssh to configure the target hosts, so password-less ssh must be configured on the target hosts. The simplest process for this is: On management host, as the user to be used to run the installer: eval `ssh-agent` ssh-keygen ssh-add For each target host: ssh-copy-id root@<target host> It is required to install python on the target hosts. It is also recommended to update the operating system packages on each host. For example: - For CentOS/RHEL: `yum update -y && yum install -y python` - For Debian/Ubuntu: `apt upgrade -y && apt install -y python` ### Inventory File Configuration The hosts to be managed must be added to the `inventory` file in the ansible directory. All hosts to be configured must be placed in the [nxe_servers] secti