Broadcom netxtreme-bntx-en dirver

README Notes Broadcom bnxt_en Linux Driver Version 1.10.2 Broadcom Inc. 15101 Alton Parkway, Irvine, CA 92618 Copyright (c) 2015 - 2016 Broadcom Corporation Copyright (c) 2016 - 2018 Broadcom Limited Copyright (c) 2018 - 2022 Broadcom Inc. All rights reserved Table of Contents ================= Introduction Limitations Port Speeds BNXT_EN Driver Dependencies BNXT_EN Driver Compilation BNXT_EN Driver Settings Autoneg Energy Efficient Ethernet Enabling Receive Side Scaling (RSS) Enabling Accelerated Receive Flow Steering (RFS) Enabling Busy Poll Sockets Enabling SR-IOV Virtual Ethernet Bridge (VEB) Hardware QoS PTP Hardware Clock Set IRQ Balance manually BNXT_EN Driver Parameters BNXT_EN Driver Defaults Statistics Unloading and Removing Driver Updating Firmware for Broadcom NetXtreme-C and NetXtreme-E devices Updating Firmware for Broadcom Nitro device Devlink Error Recovery Multi-root NUMA Direct Introduction ============ This file describes the bnxt_en Linux driver for the Broadcom NetXtreme-C/ NetXtreme-E BCM573xx, BCM574xx, BCM575xx, NetXtreme-S BCM5880x (up to 200 Gbps) Ethernet Network Controllers and Broadcom Nitro BCM58700 4-port 1/2.5/10 Gbps Ethernet Network Controller. Limitations =========== 1. The current version of the driver should compile on any contemporary Linux distribution and is specifically known to work for the following: - Red Hat: RHEL8.x, RHEL7.x, RHEL6.x; - Oracle Linux: OEL6.x UEK; - SUSE: SLES15, SLES12, SLES11SP1 and newer; - Kernels: 5.x, most 3.x/4.x, and some 2.6 kernels starting from 2.6.32. 2. The driver build depends on installed kernel headers and associated build infrastructure. Present kernels depend on GCC version 4.9, or later, and as a direct consequence, the BNXT_EN driver inherits these constraints (refer to https://www.kernel.org/doc/html/latest/process/changes.html for details). Furthermore, depending on the distribution's packaging choices, the installed headers may have been built using a later version of the toolchain. This may result in the installed headers being configured in ways that are incompatible with older tools. Thus, it is recommended practice to use the latest stable version of the toolchain to build the driver. 3. Laser needs to be brought up for Nitro BCM58700 Ethernet controller using the following command, to bring up the Link. i2cset -f -y 1 0x70 0 7 && i2cset -f -y 1 0x24 0xff 0x0 4. Each device supports hundreds of MSIX vectors. The driver will enable all MSIX vectors when it loads. On some systems running on some kernels, the system may run out of interrupt descriptors, especially when using multiple devices or NPAR devices. 5. In general, configuring VF devices depends on the associated PF being in the ifup state. This is because the PF driver actively participates in allocation and management of shared device resources, while the present software design precludes maintaining the necessary communication with the device firmware to do so when it is down. 6. Disabling devlink health auto recovery for the firmware reporter is not currently supported. Conversely, enabling auto dump has no effect. While crash dumps do not occur during the ordinary devlink health report lifecycle, the device itself does automatically capture this information prior to reset. Thus, the desirable crash context can still be retrieved via devlink health after the fact, but only if recovery has been allowed to proceed. These lazy retrieval semantics may still have user perceptable implications in that no dump is stored in the devlink reporter context until such time as it is first requested. Specifically, an earlier dump will be lost if a subsequent event occurs before capture, whereas devlink has the opposite semantics, retaining the earliest dump while dropping subsequent events. Improved firmware support for more accurate devlink health semantics is planned. Port Speeds =========== On some dual-port devices, the port speed of each port must be compatible with the port speed of the other port. 10Gbps and 25Gbps are not compatible speeds. For example, if one port is set to 10Gbps and link is up, the other port cannot be set to 25Gbps. However, the driver will allow incompatible speeds to be set on the two ports if link is not up yet. Subsequent link up on one port will render the incompatible speed on the other port to become unsupported. A console message like this may appear when this scenario happens: bnxt_en 0000:04:00.0 eth0: Link speed 25000 no longer supported If the link is up on one port, the driver will not allow the other port to be set to an incompatible speed. An attempt to do that will result in an error. For example, eth0 and eth1 are the 2 ports of the dual-port device, eth0 is set to 10Gbps and link is up. ethtool -s eth1 speed 25000 Cannot set new settings: Invalid argument not setting speed This operation will only be allowed when the link goes down on eth0 or if eth0 is brought down using ifconfig/ip. On some NPAR (NIC partioning) devices where one port is shared by multiple PCI functions, the port speed is pre-configured and cannot be changed by the driver. See Autoneg section below for additional information. BNXT_EN Driver Dependencies =========================== The driver has no dependencies on user-space firmware packages as all necessary firmware must be programmed in NVRAM(or QSPI for Nitro BCM58700 devices). Starting with driver version 1.0.0, the goal is that the driver will be compatible with all future versions of production firmware. All future versions of the driver will be backwards compatible with firmware as far back as the first production firmware. The first production firmware is version 20.1.11 using Hardware Resource Manager (HWRM) spec. 1.0.0. ethtool -i displays the firmware versions. For example: ethtool -i eth0 will show among other things: firmware-version: 20.1.11/1.0.0 pkg 20.02.00.03 In this example, the first version number (20.1.11) is the firmware version, the second version number (1.0.0) is the HWRM spec. version. The third version number (20.02.00.03) is the package version of all the different firmware components in NVRAM. The package version may not be available on all devices. Using kernels older than 4.7, if CONFIG_VLAN_MODULE kernel option is set as a module option, the vxlan.ko module must be loaded before the bnxt_en.ko module. Using newer kernels, the hwmon.ko module may need to be loaded first if CONFIG_HWMON_MODULE kernel option is set as a module option. Using kernel versions 4.6 or higher, devlink.ko module may need to be loaded first if CONFIG_NET_DEVLINK kernel option is set as a module option. Some Linux distributions, notably Red Hat Enterprise Linux, backport certain features to earlier kernels. For example, certain 3.10 RHEL kernels also provide a devlink module. Note that in such cases, the feature may not be fully supported. Please consult distribution release notes and documentation for comprehensive details. BNXT_EN Driver Compilation ========================== As noted under "Limitations" above, building the BNXT_EN driver depends on installed kernel headers and build infrastructure. These details are Linux distribution specific, and as such, are beyond the scope of this README. As an example, on Red Hat systems the kernel-devel and kernel-headers packages are required. Standard development packages such as gcc, make, awk, sed, grep, etc. are also needed. Building drivers from source is not a typical user activity. It is inherently specialized subject matter that assumes certain domain knowledge. Thus, the target audience is advanced users and software