################################################################ # Mellanox Technologies Inc # Mellanox Ethernet Microsoft Windows 64-bit Driver Release # July 16, 2018 # # RELEASE Version 05.50.03 ################################################################ Hardware Compatibility ====================== This release of software supports the following hardware: - Mellanox ConnectX-3 Dual Port 10 GbE KR Blade Mezzanine Ethernet Card - Mellanox ConnectX-3 Dual Port 40 GbE QSFP+ Ethernet Adapter - Mellanox ConnectX-3 Dual Port 10 GbE DA/SFP+ Ethernet Adapter - Mellanox ConnectX-3 Pro Dual Port 10 GbE KR Blade Mezzanine Ethernet Card - Mellanox ConnectX-3 Pro Dual Port 40 GbE QSFP+ Ethernet Adapter - Mellanox ConnectX-3 Pro Dual Port 10 GbE DA/SFP+ Ethernet Adapter Software Compatibility ====================== This release of software supports the following operating systems: - Microsoft Windows Server 2012 - Microsoft Windows Server 2012 R2 - Microsoft Windows Server 2016 Release Contents ================== - Mellanox WinOF 5.50.14643 for Microsoft Windows Server 2012 - Mellanox WinOF 5.50.14643 for Microsoft Windows Server 2012 R2 - Mellanox WinOF 5.50.14643 for Microsoft Windows Server 2016 Driver Version Name ------ ------- ---- mlx4eth63.sys 5.50.14643 Mellanox ConnectX Ethernet NDIS6.3 Driver mlx4_bus.sys 5.50.14643 MLX4 Bus Driver * Third-party trademarks and copyrights are the property of their respective owners. Feature Summary =============== N/A Fixes =============== JIT-105879 - WinOF 5.50.01 MUP packaging issue. ################################################################ # Mellanox Technologies Inc # Mellanox Ethernet Microsoft Windows 64-bit Driver Release # June 14, 2018 # # RELEASE Version 05.50.01 ################################################################ Hardware Compatibility ====================== This release of software supports the following hardware: - Mellanox ConnectX-3 Dual Port 10 GbE KR Blade Mezzanine Ethernet Card - Mellanox ConnectX-3 Dual Port 40 GbE QSFP+ Ethernet Adapter - Mellanox ConnectX-3 Dual Port 10 GbE DA/SFP+ Ethernet Adapter - Mellanox ConnectX-3 Pro Dual Port 10 GbE KR Blade Mezzanine Ethernet Card - Mellanox ConnectX-3 Pro Dual Port 40 GbE QSFP+ Ethernet Adapter - Mellanox ConnectX-3 Pro Dual Port 10 GbE DA/SFP+ Ethernet Adapter Software Compatibility ====================== This release of software supports the following operating systems: - Microsoft Windows Server 2012 - Microsoft Windows Server 2012 R2 - Microsoft Windows Server 2016 Release Contents ================== - Mellanox WinOF 5.50.14643 for Microsoft Windows Server 2012 - Mellanox WinOF 5.50.14643 for Microsoft Windows Server 2012 R2 - Mellanox WinOF 5.50.14643 for Microsoft Windows Server 2016 Driver Version Name ------ ------- ---- mlx4eth63.sys 5.50.14643 Mellanox ConnectX Ethernet NDIS6.3 Driver mlx4_bus.sys 5.50.14643 MLX4 Bus Driver * Third-party trademarks and copyrights are the property of their respective owners. Feature Summary =============== - DMN is a bus driver (mlx4_bus.sys) feature that generates dumps and traces from various components, including hardware, firmware and software, upon internally detected issues (by the resiliency sensors), user requests (mlxtool) or ND application requests via the extended Mellanox ND API. DMN is unsupported on VFs. For further information, refer to section Dump Me Now (DMN) in the User Manual - Added supports for systems with up to 252 logical processors when Hyperthreading is enabled and up to 126 logical processors when Hyperthreading is disabled - Added support for RSC solution in TCP/IP traffic to reduce CPU overhead. - Added support for extended NDSPI to control CQ moderation. - Added a new counter for packets with no destination resource. - Added a new registry key that allows users to configure the E2E Congestion Control feature. - Added to the vlan_config tool the ability to create VLANs for the Physical Function (PF) in addition to the Virtual Function (VF) - Added support for collecting firmware MST dumps in cases of system bug check. - Added an event log message (ID 273) that is printed when the number of resources to load the VF is insufficient. - Added a counter for the number of packets discarded due to an invalid QP number. - Added DSCP based counters to support traffic where no VLAN/priority is present. - RoCE TTL default value has been changed from 128 to 16. Fixes =============== - Fixed an issue that caused networks with new Subnet Managers (OpenSM 4.7.0 and up) to drop malformed multicast-join packets issued by the driver. The driver now constructs the multicast join request correctly. - Fixed a BSOD that occurred on servers with more than 64 cores as the Tx traffic did not honor the Tx affinity implied by the TSS, when the number of potential RSS CPUs was greater than 64. - Fixed a BSOD that occurred while installing WinOF v5.35 due to stack usage overrun. - Fixed an issue that could cause BSOD when resizing a number of Queue Pairs (QPs) in Virtual Multiple Machine Queue (VMMQ) mode, while running stress traffic to the VM - Removed the following redundant VM Event Log messages: 122, 104, 144 - Fixed a wrong Link State value in the VSTAT. Important Notes =============== - After the installer completes, the system must be rebooted to complete the device driver installation - Wake on LAN is only supported with Mellanox ConnectX-3 and ConnectX-3 Pro Dual Port 10 GbE KR Blade Mezzanine Ethernet Cards ********************************************************************* Currently known issues and limitations with Mellanox Ethernet drivers and devices ********************************************************************* 1. No current support for Brocade M8428-k IO Module for Dell EMC PowerEdge M1000e chassis 2. No current support for Dell PowerConnect M8024-k IO Module for Dell EMC PowerEdge M1000e chassis 3. DCBX support does not include support for Willing Mode. Mellanox Known General Issues with WinOF 5.50 ============================================= - On Windows 2016 Client: The SR-IOV virtual adapters' counters do not count the data of the Virtual Function. The counters will not rise when the Virtual Function's counters rise neither in the performance monitor nor in the task manager. - The driver will ask for a reboot when a network interface gets disconnected while loading the VMs on hyper-V. - Activating NC-SI in WinOF v4.90.10541 may cause driver’s loading failure when using an older firmware version than 2.30.8000. Workaround: Do not enable NC-SI in machines that WinOF v4.90.10541 is installed in. - Running Ntttcp without the '-a X' flag (X >1) in a NIC configured with 10GbE, may cause low bandwidth in TCP single stream. Workaround: Run Ntttcp with '-a 8' for best performance - The UI displays the network interface’s state incorrectly even after disabling and re-enabling the "disconnected" network interfaces. Workaround: To see the correct state and link speed, perform one of the following: . Run Get-netadapter Powershell cmdlet or . Right click on that interface from 'Network Connections' and click on status - Clearing the Mellanox device counters through perfmon does not always work. Workaround: Restart the driver - When running in SR-IOV mode and Hyper-V, the same driver version should be used on both the Hypervisor and the virtual machine. - Working with user space RDMA resources (i.e. ND) in parallel with hardware reset may cause unexpected behavior. - According to the hardware architecture, Mellanox NIC devices are presented in the registry as virtual devices and not as physical devices. Workaround: Any customer programs or scripts that look into the NIC registry values should not assume it is a physical device. Mellanox Known Ethernet Issues with WinOF 5.50 ============================================= - Disabling the 'Priority & VLAN tag' in the UI which VLANID is configured, may result in sending packets with the configured VLANID. Workaround: Remove the VLANID before disabling the 'Priority & VLAN tag'. - WakeOnMagicPacket registry key is not added to the registry although WoL is supported by the driver and by the NIC. - High multicast drop rate on multicast storming. Workaround: Use 'Multicast traffic' tuning option under the performance tab.For further information, please refer to section 'Tunable Performance Parameters' in the User Manual. - When there is a stress in TCP connection establishments, some of those connections may fail. Workaround: Increase the Ring queue sizes: . ReceiveBuffers - controls the receive ring size . TransmitBuffers - controls the transmit ring size - The DCB component specifies a default traffic classification that is applied to all egress packets that do not match other classification conditions. In this case, the network adapter assigns the IEEE 802.1p priority level that is associated with the default classification to these egress packets. The default traffic classification has the following attributes: . It has a traffic classification condition of type NDIS_QOS_CONDITION_DEFAULT. . It is the first traffic classification defined in the array of NDIS_QOS_CLASSIFICATION_ELEMENT structures. - Virtual Ethernet Interfaces created by vea_man are not tuned by the automatic performance tuning script. Workaround: For optimal performance need to follow the performance tuning guide and apply relevant changes to the VEA interface - Wake on Lan (WoL) cannot be disabled on NICs which supports it. Mellanox Known QoS Issues with WinOF 5.50 ============================================= - Running Quality of Service (QoS) commands without the parameter “-PolicyStore ActiveStore” may cause machines to load without Quality of Service policy. Workaround: Store the QoS policy in the ActiveStore Mellanox Known RoCE NVGRE Issues with WinOF 5.50 ============================================= - RoCE does not support: . Traffic cannot go through the router. It works in the same subnet only. . Multicast traffic . VLAN . Layer 3 feature - In machines with heterogeneous NICs: a NIC which supports RoCE v2, and a NIC which does not support RoCE v2 the following issus might raise: . ConnectX®-3 Pro is loaded with the transport type RoCE v2 . ConnectX®-3 Pro is loaded with the transport type RoCE - When NVGRE off-load is enabled, the GRE traffic cannot be accepted as a regular L2 traffic and requires special L2_TUNNELING steering rules. In such case the GRE packets are dropped or directed to promiscuous queue. - Using different versions of RoCE in your cluster is not supported. Workaround: Use the same RoCE version in all the cluster in the Ethernet ports. - RDMA Activity counters do not count during NetworkDirect RoCE traffic. - GRE traffic steering by inner MAC and by outer MAC simultaneously is currently not supported. Workaround: Configure steering or by inner MAC, or by outer MAC. - If VMQ set filter requests are accepted without a GRE flag (i.e. requested steering by outer MAC), the GRE packets do not reach that VMQ. Workaround: N/A - Set the bus driver registry keyAcceptGREbyOuterMAC_P1/2 per port to accept GRE traffic by outer MAC and to duplicate L2 steering rule to L2_TUNNELING rule for each VMQ set filter request without GRE flag. Note: For regular NVGRE Hyper-V scenarios the value of the registry key below must be set to 0:AcceptGREbyOuterMAC_P1/2 Workaround: N/A Mellanox Known Performance Issues with WinOF 5.50 ============================================= - When using WinOF 4.40 or above, low throughput will be seen on 40GbE adapters when QoS is enabled. Workaround: Disable QoS when it is not in use: . Open a PowerShell prompt. Run: . Disable-NetAdapterQos -name where is e.g. "Ethernet 1" - perf_tuning is supported only when one of the two NUMA nodes are in use. - Running performance benchmarks for a short period of time (< 1 sec) may provide bad latency. Workaround: Set 'Rx Interrupt Moderation Profile' and 'Tx Interrupt Moderation Profile', to 'Low Latency' to avoid bad latency. Note: This may increase CPU utilization. - The driver uses optimal interrupt moderation values for 10 GbE SR-IOV VF scenario. For other scenarios, the optimal values yet to be found. - While running in a Virtual Machine (working with VF) or in a Native Machine, performance counters, if read directly or by using any tool (as Windows Task Manager), may show that no packet has been sent/received. This happens because the driver periodically examines the actual performance counters and caches the results. If the sample rate is too high, the counter values will remain the same. Workaround: Reduce the counters sample rate. Mellanox Known Hyper-V Issues with WinOF 5.50 ============================================= - When the vSwitch is detached from the ETH device while the driver is disabled, the device does not reacquire the static IP it had before the attachment of the vSwitch. When the vSwitch is attached to the ETH device while there is no link, it will not receive the device IP when the link is back up. - After attaching the vSwitch to the ETH device, changing the 'Jumbo Packet' registry key on the ETH device does not affect the vSwitch configuration and vice versa. For example, if the user sets the 'Jumbo Packet' on the ETH device to X, and the 'Jumbo Packet' on the vSwitch to X+Y, X+Y sized packets will be passed from NDIS down to the driver and they will be dropped by it. Workaround: Reattach the vSwitch to sync with the value set in the ETH device. - Unexpected behavior might occur when running in a virtualized environment and creating two virtual switches bound to each of the ports of a dual port NIC and then using both of them with two vNICs from the same VM. Workaround: N/A - In Ethernet to achieve better iperf TCP performance between a Linux VM and a Windows VM on different hosts, when using MS MUX over the Ethernet driver, use the non VMQ mode for the VMs. Workaround: N/A - After disabling and enabling a port on a guest, a ping to it may be renewed after a minute. The ARP requests sent by Windows are less frequent as the time passes. If the guest port was down for a while, it could take time until Windows decides to send another ARP request to it. Workaround: N/A - When VMQ is enabled after reset, the driver loads all the VMQs that existed before the reset. However, it is not guaranteed that each VMQ will receive the same QP number it had before the reset. This can cause some delay as a result of resetting before connectivity is reestablished. The delay is caused by the time it takes for the ARP table to update after initiating the Gratuitous ARP. Workaround: N/A Mellanox Known SR-IOV Issues with WinOF 5.50 ============================================= - Working with Windows guest OS over non-Windows SR-IOV hypervisor may result with higher latency compared to Windows Hypervisor. - When working with OEM adapter cards in SR-IOV mode, the VF is identified as a Mellanox adapter. There is no other effect on the behavior of the VF. Workaround: Verify that the FW INI contains the parameter vf_subsystem_id with the OEM encoding in - In SR-IOV mode over Hyper-V, all ports are in pure Ethernet mode and RDMA is not supported on either port. - In SR-IOV mode, enabling SR-IOV in the BIOS may change the interfaces names. If any VSwitch is bounded to an interface whose name was changed, there will not be any way to perform any operation on them. Workaround: Unbond all Vswitches from the NIC's interfaces before enabling SR-IOV in the BIOS. - In SR-IOV, 40Gbps interfaces of vSwitch interface on HyperV, and the VMNIC are identified as 10Gbps. Despite the incorrect speed identification, they still achieve 40Gbps performance. - Device name in the Device Manager does not show the correct OEM branding for SR-IOV Virtual Function devices without the updated firmware. Mellanox Known Install and Upgrade Issues with WinOF 5.50 ============================================= - Upgrading the driver while the UI is opened with the 'ConnectX NIC device' may cause the installation process to never end. Work-around: Close the UI before driver upgrade. - Canceling the installation process may leave the bus driver in a disable state. The driver appears in a yellow bang containing the following error message: 'Windows cannot start this hardware device because its configuration information (in the registry) is incomplete or damaged. (Code 19)'. Work-around: Scan for new hardware and reboot the machine. - Downgrade is not supported. Work-around: Uninstall the current version and install the older one. - Configuration is not restored when replacing a ConnectX®-3 NIC with a ConnectX®-3 Pro NIC located on the same PCI slot Work-around: Clean the old network adapter configuration prior to upgrade. - Configuration can be restored only in Windows Server 2012 and above - IPv6 configuration restore is not supported - Upon upgrade, the following Registry Key values will be overwritten with the following: *ReceiveBuffers = 512 *MaxRssProcessors = 8 *RssBaseProcNumber = 0 *NumRSSQueues = 8 *RssMaxProcNumber = 63 *RssProfile = 1 DefaultRecvRingProcessor = -1 TxInterruptProcessor = -1 TxForwardingProcessor = -1 RxIntModerationProfile = 1 TxIntModerationProfile = 1 RecvCompletionMethod = 1 SingleStream = 0 TxRingNum = 8 - Upon upgrade the following Ethernet Registry Keys will be deleted: . SendCompletionMethod . UseRSSForRawIP . UseRSSForUDP - Uninstalling the driver after upgrade won't remove the directory %ProgramFiles%\Mellanox - Uninstalling the driver when multiple of VLANs are configured never ends. Work-around: Remove the VLANs before uninstallation. - The installation process does not close any applications running in the background, and may cause a BSOD as a result of a stuck cmd. Work-around: It is recommended to close all running applications prior to upgrading the driver. - Installation/upgrade fails due to PNP failure to copy the driver files to the driver store, and the following text is printed in the event logs: Fault bucket, type 0 Event Name: PnPDriverImportError Response: Not available Attached files: C:\Users\\AppData\Local\Temp\DMI151A.tmp.log.xml C:\Program Files\Mellanox\MLNX_VPI\ETH\mlx4eth63.inf Work-around: Reboot the machine and reinstall - Installation/upgrade fails due to failure to stop the WMI service, and the following text is printed in the installation log: 'CustomAction StopWMIService returned actual error code 1603' Work-around: Kill the WMIPrvSE.exe tasks in the task manager and reinstall. - Following the upgrade of Mellanox driver to WinOF-4.60 and above or on servers with no internet access, the first PowerShell command might be stuck for ~2-3 minutes before its completed. According to the following content, this issue is related to .Net framework version or an issue with the internet access: http://www.minasi.com/forum/topic.asp?TOPIC_ID=39253 Work-around: Run the following script on the server to optimizes loading PowerShell DLLs: $Env:PATH = [Runtime.InteropServices.RuntimeEnvironment]::GetRuntimeDirectory() [AppDomain]::CurrentDomain.GetAssemblies() | % {$pt = $_.Location if (! $pt) {continue} if ($cn++) {''} $na = Split-Path -Leaf $pt Write-Host -ForegroundColor Yellow "NGENing $na" ngen install $pt } - Uninstall from the Device Manager is currently not supported - If there are disabled network interfaces or a disabled Mellanox bus driver, they will be enabled after the WinOF upgrade. - Uninstalling mlx4 bus device from the Device Manager results in the removal of performance counters for all WinOF devices. - The ucrtbse.dll file is missing in operating systems older than Windows Server 2016. This leads to failure of the installation. - The UI and installation log will show messages for all missing updates such as "missing windows updates: KB2999226" Work-Around: Install Windows update kb2999226. The update files can be downloaded from https://support.microsoft.com/en-us/kb/2999226. Mellanox Known Utilities Issues with WinOF 5.50 ============================================= - Pcap file issues for RoCE IP Based: . The packets 'capture-time' field is wrong (may be 0 or negative numbers). . For every captured packet, an additional 0 size flame is added. This appears in Wireshark as a 'malformed Ethernet packet'. - When the tracer tool traces many events, it may consume a large amount of memory (up to several GB RAM). Use the button to set maximum number of the displayed lines at a maximum buffer size. Workaround: Verify Ethernet flow control is enable to ensure a lossless link - Pcap file issues for RoCE IP Based: . The packets 'capture-time' field is wrong (may be 0 or negative numbers). . For every captured packet, an additional 0 size flame is added. This appears in Wireshark as a 'malformed Ethernet packet'. - When the tracer tool traces many events, it may consume a large amount of memory (up to several GB RAM). Mellanox Known CIM/WMI Issues with WinOF 5.50 ============================================= - Running Microsoft CIM cmdlets operations and their derived classes on classes MLNX_NetAdapterSettingData and MLNX_NetAdapterRoceSettingData is not supported. Calling those commands may cause the debugger, if connected to the machine, to assert. Work-around: Use DriverCoreSettings instead. - For PCI Gen3, PcieLinkSpeed is reported as 'Unknown' when running Get-NetAdapterHardwareInfo Powershell cmdlet - WMI does not work due to lack of permissions. Work-around: Change the execution policy. Run: Set-ExecutionPolicy AllSigned - The information that is printed in the cmdlets get-netadaptersriov and Get-MlnxPCIDeviceSetting is inconsistent. Work-around: Use only the cmdlet Get-MlnxPCIDeviceSetting. Mellanox Known ND Issues with WinOF 5.50 ============================================= - There is an interoperability problem between NDK and ND in RDMA operations during the write and read operations. However, the send operation resumes working. This happens since ND uses remote token in a network order (big endian) while NDK uses remote token in CPU order (little endian). Therefore, an inconsistency is caused between ND and NDK in RDMA operations. Work-around: An ND application that works with NDK using RDMA operations must handle this issue by changing the remote token to the appropriate byte order before sending it to NDK. - When working with the default NDv1 and NDv2 providers, the following error message might be displayed: 0xC0000120 NT_STATUS_CANCELLED This error does not affect any functionality and can be safely ignored. - Changing the default ND providers may cause random errors, such as: 0xC0000238 (NT_STATUS_ADDRESS_ALREADY_ASSOCIATED) Mellanox Known Misc Issues with WinOF 5.50 ============================================= - Windows 2016 mistakenly recognizes ConnectX-3 Pro card as a USB interface. This may disconnect the network and cause failure of critical network services. - RoCE fallback mode is not working when SR-IOV is enabled. - Errors that may occur during initial driver start in the DriverEntry function do not appear in the event log. These errors will be indicated by a yellow exclamation mark on the device, with code 37. - Several error messages from the mlx4_bus may appear in the VM Event Log. These messages (104, 122, 144) do not indicate any error, and can be safely ignored. - On hypervisor, when one Ethernet port is bind to vmswitch in which SR-IOV is enabled, Network Direct applications do not work on the second port. - Vlan_config tool allows to modify the vlan name to a name which is invalid by the vlan_config addvlan command. - A VF can initialize only one Comm Channel: either MS Comm Channel or MLNX Comm Channel. - Changing RoCE mode in a virtual machine is not supported. ################################################################ # Mellanox Technologies Inc # Mellanox Ethernet Microsoft Windows 64-bit Driver Release # June 14, 2018 # # RELEASE Version 05.50.01 ################################################################ Hardware Compatibility ====================== This release of software supports the following hardware: - Mellanox ConnectX-3 Dual Port 10 GbE KR Blade Mezzanine Ethernet Card - Mellanox ConnectX-3 Dual Port 40 GbE QSFP+ Ethernet Adapter - Mellanox ConnectX-3 Dual Port 10 GbE DA/SFP+ Ethernet Adapter - Mellanox ConnectX-3 Pro Dual Port 10 GbE KR Blade Mezzanine Ethernet Card - Mellanox ConnectX-3 Pro Dual Port 40 GbE QSFP+ Ethernet Adapter - Mellanox ConnectX-3 Pro Dual Port 10 GbE DA/SFP+ Ethernet Adapter Software Compatibility ====================== This release of software supports the following operating systems: - Microsoft Windows Server 2012 - Microsoft Windows Server 2012 R2 - Microsoft Windows Server 2016 Release Contents ================== - Mellanox WinOF 5.50.14643 for Microsoft Windows Server 2012 - Mellanox WinOF 5.50.14643 for Microsoft Windows Server 2012 R2 - Mellanox WinOF 5.50.14643 for Microsoft Windows Server 2016 Driver Version Name ------ ------- ---- mlx4eth63.sys 5.50.14643 Mellanox ConnectX Ethernet NDIS6.3 Driver mlx4_bus.sys 5.50.14643 MLX4 Bus Driver * Third-party trademarks and copyrights are the property of their respective owners. Feature Summary =============== - DMN is a bus driver (mlx4_bus.sys) feature that generates dumps and traces from various components, including hardware, firmware and software, upon internally detected issues (by the resiliency sensors), user requests (mlxtool) or ND application requests via the extended Mellanox ND API. DMN is unsupported on VFs. For further information, refer to section Dump Me Now (DMN) in the User Manual - Added supports for systems with up to 252 logical processors when Hyperthreading is enabled and up to 126 logical processors when Hyperthreading is disabled - Added support for RSC solution in TCP/IP traffic to reduce CPU overhead. - Added support for extended NDSPI to control CQ moderation. - Added a new counter for packets with no destination resource. - Added a new registry key that allows users to configure the E2E Congestion Control feature. - Added to the vlan_config tool the ability to create VLANs for the Physical Function (PF) in addition to the Virtual Function (VF) - Added support for collecting firmware MST dumps in cases of system bug check. - Added an event log message (ID 273) that is printed when the number of resources to load the VF is insufficient. - Added a counter for the number of packets discarded due to an invalid QP number. - Added DSCP based counters to support traffic where no VLAN/priority is present. - RoCE TTL default value has been changed from 128 to 16. Fixes =============== - Fixed an issue that caused networks with new Subnet Managers (OpenSM 4.7.0 and up) to drop malformed multicast-join packets issued by the driver. The driver now constructs the multicast join request correctly. - Fixed a BSOD that occurred on servers with more than 64 cores as the Tx traffic did not honor the Tx affinity implied by the TSS, when the number of potential RSS CPUs was greater than 64. - Fixed a BSOD that occurred while installing WinOF v5.35 due to stack usage overrun. - Fixed an issue that could cause BSOD when resizing a number of Queue Pairs (QPs) in Virtual Multiple Machine Queue (VMMQ) mode, while running stress traffic to the VM - Removed the following redundant VM Event Log messages: 122, 104, 144 - Fixed a wrong Link State value in the VSTAT. Important Notes =============== - After the installer completes, the system must be rebooted to complete the device driver installation - Wake on LAN is only supported with Mellanox ConnectX-3 and ConnectX-3 Pro Dual Port 10 GbE KR Blade Mezzanine Ethernet Cards ********************************************************************* Currently known issues and limitations with Mellanox Ethernet drivers and devices ********************************************************************* 1. No current support for Brocade M8428-k IO Module for Dell EMC PowerEdge M1000e chassis 2. No current support for Dell PowerConnect M8024-k IO Module for Dell EMC PowerEdge M1000e chassis 3. DCBX support does not include support for Willing Mode. Mellanox Known General Issues with WinOF 5.50 ============================================= - On Windows 2016 Client: The SR-IOV virtual adapters' counters do not count the data of the Virtual Function. The counters will not rise when the Virtual Function's counters rise neither in the performance monitor nor in the task manager. - The driver will ask for a reboot when a network interface gets disconnected while loading the VMs on hyper-V. - Activating NC-SI in WinOF v4.90.10541 may cause driver’s loading failure when using an older firmware version than 2.30.8000. Workaround: Do not enable NC-SI in machines that WinOF v4.90.10541 is installed in. - Running Ntttcp without the '-a X' flag (X >1) in a NIC configured with 10GbE, may cause low bandwidth in TCP single stream. Workaround: Run Ntttcp with '-a 8' for best performance - The UI displays the network interface’s state incorrectly even after disabling and re-enabling the "disconnected" network interfaces. Workaround: To see the correct state and link speed, perform one of the following: . Run Get-netadapter Powershell cmdlet or . Right click on that interface from 'Network Connections' and click on status - Clearing the Mellanox device counters through perfmon does not always work. Workaround: Restart the driver - When running in SR-IOV mode and Hyper-V, the same driver version should be used on both the Hypervisor and the virtual machine. - Working with user space RDMA resources (i.e. ND) in parallel with hardware reset may cause unexpected behavior. - According to the hardware architecture, Mellanox NIC devices are presented in the registry as virtual devices and not as physical devices. Workaround: Any customer programs or scripts that look into the NIC registry values should not assume it is a physical device. Mellanox Known Ethernet Issues with WinOF 5.50 ============================================= - Disabling the 'Priority & VLAN tag' in the UI which VLANID is configured, may result in sending packets with the configured VLANID. Workaround: Remove the VLANID before disabling the 'Priority & VLAN tag'. - WakeOnMagicPacket registry key is not added to the registry although WoL is supported by the driver and by the NIC. - High multicast drop rate on multicast storming. Workaround: Use 'Multicast traffic' tuning option under the performance tab.For further information, please refer to section 'Tunable Performance Parameters' in the User Manual. - When there is a stress in TCP connection establishments, some of those connections may fail. Workaround: Increase the Ring queue sizes: . ReceiveBuffers - controls the receive ring size . TransmitBuffers - controls the transmit ring size - The DCB component specifies a default traffic classification that is applied to all egress packets that do not match other classification conditions. In this case, the network adapter assigns the IEEE 802.1p priority level that is associated with the default classification to these egress packets. The default traffic classification has the following attributes: . It has a traffic classification condition of type NDIS_QOS_CONDITION_DEFAULT. . It is the first traffic classification defined in the array of NDIS_QOS_CLASSIFICATION_ELEMENT structures. - Virtual Ethernet Interfaces created by vea_man are not tuned by the automatic performance tuning script. Workaround: For optimal performance need to follow the performance tuning guide and apply relevant changes to the VEA interface - Wake on Lan (WoL) cannot be disabled on NICs which supports it. Mellanox Known QoS Issues with WinOF 5.50 ============================================= - Running Quality of Service (QoS) commands without the parameter “-PolicyStore ActiveStore” may cause machines to load without Quality of Service policy. Workaround: Store the QoS policy in the ActiveStore Mellanox Known RoCE NVGRE Issues with WinOF 5.50 ============================================= - RoCE does not support: . Traffic cannot go through the router. It works in the same subnet only. . Multicast traffic . VLAN . Layer 3 feature - In machines with heterogeneous NICs: a NIC which supports RoCE v2, and a NIC which does not support RoCE v2 the following issus might raise: . ConnectX®-3 Pro is loaded with the transport type RoCE v2 . ConnectX®-3 Pro is loaded with the transport type RoCE - When NVGRE off-load is enabled, the GRE traffic cannot be accepted as a regular L2 traffic and requires special L2_TUNNELING steering rules. In such case the GRE packets are dropped or directed to promiscuous queue. - Using different versions of RoCE in your cluster is not supported. Workaround: Use the same RoCE version in all the cluster in the Ethernet ports. - RDMA Activity counters do not count during NetworkDirect RoCE traffic. - GRE traffic steering by inner MAC and by outer MAC simultaneously is currently not supported. Workaround: Configure steering or by inner MAC, or by outer MAC. - If VMQ set filter requests are accepted without a GRE flag (i.e. requested steering by outer MAC), the GRE packets do not reach that VMQ. Workaround: N/A - Set the bus driver registry keyAcceptGREbyOuterMAC_P1/2 per port to accept GRE traffic by outer MAC and to duplicate L2 steering rule to L2_TUNNELING rule for each VMQ set filter request without GRE flag. Note: For regular NVGRE Hyper-V scenarios the value of the registry key below must be set to 0:AcceptGREbyOuterMAC_P1/2 Workaround: N/A Mellanox Known Performance Issues with WinOF 5.50 ============================================= - When using WinOF 4.40 or above, low throughput will be seen on 40GbE adapters when QoS is enabled. Workaround: Disable QoS when it is not in use: . Open a PowerShell prompt. Run: . Disable-NetAdapterQos -name where is e.g. "Ethernet 1" - perf_tuning is supported only when one of the two NUMA nodes are in use. - Running performance benchmarks for a short period of time (< 1 sec) may provide bad latency. Workaround: Set 'Rx Interrupt Moderation Profile' and 'Tx Interrupt Moderation Profile', to 'Low Latency' to avoid bad latency. Note: This may increase CPU utilization. - The driver uses optimal interrupt moderation values for 10 GbE SR-IOV VF scenario. For other scenarios, the optimal values yet to be found. - While running in a Virtual Machine (working with VF) or in a Native Machine, performance counters, if read directly or by using any tool (as Windows Task Manager), may show that no packet has been sent/received. This happens because the driver periodically examines the actual performance counters and caches the results. If the sample rate is too high, the counter values will remain the same. Workaround: Reduce the counters sample rate. Mellanox Known Hyper-V Issues with WinOF 5.50 ============================================= - When the vSwitch is detached from the ETH device while the driver is disabled, the device does not reacquire the static IP it had before the attachment of the vSwitch. When the vSwitch is attached to the ETH device while there is no link, it will not receive the device IP when the link is back up. - After attaching the vSwitch to the ETH device, changing the 'Jumbo Packet' registry key on the ETH device does not affect the vSwitch configuration and vice versa. For example, if the user sets the 'Jumbo Packet' on the ETH device to X, and the 'Jumbo Packet' on the vSwitch to X+Y, X+Y sized packets will be passed from NDIS down to the driver and they will be dropped by it. Workaround: Reattach the vSwitch to sync with the value set in the ETH device. - Unexpected behavior might occur when running in a virtualized environment and creating two virtual switches bound to each of the ports of a dual port NIC and then using both of them with two vNICs from the same VM. Workaround: N/A - In Ethernet to achieve better iperf TCP performance between a Linux VM and a Windows VM on different hosts, when using MS MUX over the Ethernet driver, use the non VMQ mode for the VMs. Workaround: N/A - After disabling and enabling a port on a guest, a ping to it may be renewed after a minute. The ARP requests sent by Windows are less frequent as the time passes. If the guest port was down for a while, it could take time until Windows decides to send another ARP request to it. Workaround: N/A - When VMQ is enabled after reset, the driver loads all the VMQs that existed before the reset. However, it is not guaranteed that each VMQ will receive the same QP number it had before the reset. This can cause some delay as a result of resetting before connectivity is reestablished. The delay is caused by the time it takes for the ARP table to update after initiating the Gratuitous ARP. Workaround: N/A Mellanox Known SR-IOV Issues with WinOF 5.50 ============================================= - Working with Windows guest OS over non-Windows SR-IOV hypervisor may result with higher latency compared to Windows Hypervisor. - When working with OEM adapter cards in SR-IOV mode, the VF is identified as a Mellanox adapter. There is no other effect on the behavior of the VF. Workaround: Verify that the FW INI contains the parameter vf_subsystem_id with the OEM encoding in - In SR-IOV mode over Hyper-V, all ports are in pure Ethernet mode and RDMA is not supported on either port. - In SR-IOV mode, enabling SR-IOV in the BIOS may change the interfaces names. If any VSwitch is bounded to an interface whose name was changed, there will not be any way to perform any operation on them. Workaround: Unbond all Vswitches from the NIC's interfaces before enabling SR-IOV in the BIOS. - In SR-IOV, 40Gbps interfaces of vSwitch interface on HyperV, and the VMNIC are identified as 10Gbps. Despite the incorrect speed identification, they still achieve 40Gbps performance. - Device name in the Device Manager does not show the correct OEM branding for SR-IOV Virtual Function devices without the updated firmware. Mellanox Known Install and Upgrade Issues with WinOF 5.50 ============================================= - Upgrading the driver while the UI is opened with the 'ConnectX NIC device' may cause the installation process to never end. Work-around: Close the UI before driver upgrade. - Canceling the installation process may leave the bus driver in a disable state. The driver appears in a yellow bang containing the following error message: 'Windows cannot start this hardware device because its configuration information (in the registry) is incomplete or damaged. (Code 19)'. Work-around: Scan for new hardware and reboot the machine. - Downgrade is not supported. Work-around: Uninstall the current version and install the older one. - Configuration is not restored when replacing a ConnectX®-3 NIC with a ConnectX®-3 Pro NIC located on the same PCI slot Work-around: Clean the old network adapter configuration prior to upgrade. - Configuration can be restored only in Windows Server 2012 and above - IPv6 configuration restore is not supported - Upon upgrade, the following Registry Key values will be overwritten with the following: *ReceiveBuffers = 512 *MaxRssProcessors = 8 *RssBaseProcNumber = 0 *NumRSSQueues = 8 *RssMaxProcNumber = 63 *RssProfile = 1 DefaultRecvRingProcessor = -1 TxInterruptProcessor = -1 TxForwardingProcessor = -1 RxIntModerationProfile = 1 TxIntModerationProfile = 1 RecvCompletionMethod = 1 SingleStream = 0 TxRingNum = 8 - Upon upgrade the following Ethernet Registry Keys will be deleted: . SendCompletionMethod . UseRSSForRawIP . UseRSSForUDP - Uninstalling the driver after upgrade won't remove the directory %ProgramFiles%\Mellanox - Uninstalling the driver when multiple of VLANs are configured never ends. Work-around: Remove the VLANs before uninstallation. - The installation process does not close any applications running in the background, and may cause a BSOD as a result of a stuck cmd. Work-around: It is recommended to close all running applications prior to upgrading the driver. - Installation/upgrade fails due to PNP failure to copy the driver files to the driver store, and the following text is printed in the event logs: Fault bucket, type 0 Event Name: PnPDriverImportError Response: Not available Attached files: C:\Users\\AppData\Local\Temp\DMI151A.tmp.log.xml C:\Program Files\Mellanox\MLNX_VPI\ETH\mlx4eth63.inf Work-around: Reboot the machine and reinstall - Installation/upgrade fails due to failure to stop the WMI service, and the following text is printed in the installation log: 'CustomAction StopWMIService returned actual error code 1603' Work-around: Kill the WMIPrvSE.exe tasks in the task manager and reinstall. - Following the upgrade of Mellanox driver to WinOF-4.60 and above or on servers with no internet access, the first PowerShell command might be stuck for ~2-3 minutes before its completed. According to the following content, this issue is related to .Net framework version or an issue with the internet access: http://www.minasi.com/forum/topic.asp?TOPIC_ID=39253 Work-around: Run the following script on the server to optimizes loading PowerShell DLLs: $Env:PATH = [Runtime.InteropServices.RuntimeEnvironment]::GetRuntimeDirectory() [AppDomain]::CurrentDomain.GetAssemblies() | % {$pt = $_.Location if (! $pt) {continue} if ($cn++) {''} $na = Split-Path -Leaf $pt Write-Host -ForegroundColor Yellow "NGENing $na" ngen install $pt } - Uninstall from the Device Manager is currently not supported - If there are disabled network interfaces or a disabled Mellanox bus driver, they will be enabled after the WinOF upgrade. - Uninstalling mlx4 bus device from the Device Manager results in the removal of performance counters for all WinOF devices. - The ucrtbse.dll file is missing in operating systems older than Windows Server 2016. This leads to failure of the installation. - The UI and installation log will show messages for all missing updates such as "missing windows updates: KB2999226" Work-Around: Install Windows update kb2999226. The update files can be downloaded from https://support.microsoft.com/en-us/kb/2999226. Mellanox Known Utilities Issues with WinOF 5.50 ============================================= - Pcap file issues for RoCE IP Based: . The packets 'capture-time' field is wrong (may be 0 or negative numbers). . For every captured packet, an additional 0 size flame is added. This appears in Wireshark as a 'malformed Ethernet packet'. - When the tracer tool traces many events, it may consume a large amount of memory (up to several GB RAM). Use the button to set maximum number of the displayed lines at a maximum buffer size. Workaround: Verify Ethernet flow control is enable to ensure a lossless link - Pcap file issues for RoCE IP Based: . The packets 'capture-time' field is wrong (may be 0 or negative numbers). . For every captured packet, an additional 0 size flame is added. This appears in Wireshark as a 'malformed Ethernet packet'. - When the tracer tool traces many events, it may consume a large amount of memory (up to several GB RAM). Mellanox Known CIM/WMI Issues with WinOF 5.50 ============================================= - Running Microsoft CIM cmdlets operations and their derived classes on classes MLNX_NetAdapterSettingData and MLNX_NetAdapterRoceSettingData is not supported. Calling those commands may cause the debugger, if connected to the machine, to assert. Work-around: Use DriverCoreSettings instead. - For PCI Gen3, PcieLinkSpeed is reported as 'Unknown' when running Get-NetAdapterHardwareInfo Powershell cmdlet - WMI does not work due to lack of permissions. Work-around: Change the execution policy. Run: Set-ExecutionPolicy AllSigned - The information that is printed in the cmdlets get-netadaptersriov and Get-MlnxPCIDeviceSetting is inconsistent. Work-around: Use only the cmdlet Get-MlnxPCIDeviceSetting. Mellanox Known ND Issues with WinOF 5.50 ============================================= - There is an interoperability problem between NDK and ND in RDMA operations during the write and read operations. However, the send operation resumes working. This happens since ND uses remote token in a network order (big endian) while NDK uses remote token in CPU order (little endian). Therefore, an inconsistency is caused between ND and NDK in RDMA operations. Work-around: An ND application that works with NDK using RDMA operations must handle this issue by changing the remote token to the appropriate byte order before sending it to NDK. - When working with the default NDv1 and NDv2 providers, the following error message might be displayed: 0xC0000120 NT_STATUS_CANCELLED This error does not affect any functionality and can be safely ignored. - Changing the default ND providers may cause random errors, such as: 0xC0000238 (NT_STATUS_ADDRESS_ALREADY_ASSOCIATED) Mellanox Known Misc Issues with WinOF 5.50 ============================================= - Windows 2016 mistakenly recognizes ConnectX-3 Pro card as a USB interface. This may disconnect the network and cause failure of critical network services. - RoCE fallback mode is not working when SR-IOV is enabled. - Errors that may occur during initial driver start in the DriverEntry function do not appear in the event log. These errors will be indicated by a yellow exclamation mark on the device, with code 37. - Several error messages from the mlx4_bus may appear in the VM Event Log. These messages (104, 122, 144) do not indicate any error, and can be safely ignored. - On hypervisor, when one Ethernet port is bind to vmswitch in which SR-IOV is enabled, Network Direct applications do not work on the second port. - Vlan_config tool allows to modify the vlan name to a name which is invalid by the vlan_config addvlan command. - A VF can initialize only one Comm Channel: either MS Comm Channel or MLNX Comm Channel. - Changing RoCE mode in a virtual machine is not supported. ################################################################ # Mellanox Technologies Inc # Mellanox Ethernet Microsoft Windows 64-bit Driver Release # April 5, 2018 # # RELEASE Version 05.35.06 ################################################################ Hardware Compatibility ====================== This release of software supports the following hardware: - Mellanox ConnectX-3 Dual Port 10 GbE KR Blade Mezzanine Ethernet Card - Mellanox ConnectX-3 Dual Port 40 GbE QSFP+ Ethernet Adapter - Mellanox ConnectX-3 Dual Port 10 GbE DA/SFP+ Ethernet Adapter - Mellanox ConnectX-3 Pro Dual Port 10 GbE KR Blade Mezzanine Ethernet Card - Mellanox ConnectX-3 Pro Dual Port 40 GbE QSFP+ Ethernet Adapter - Mellanox ConnectX-3 Pro Dual Port 10 GbE DA/SFP+ Ethernet Adapter Software Compatibility ====================== This release of software supports the following operating systems: - Microsoft Windows Server 2012 - Microsoft Windows Server 2012 R2 - Microsoft Windows Server 2016 Release Contents ================== - Mellanox WinOF 05.35.12978 for Microsoft Windows Server 2012 - Mellanox WinOF 05.35.12978 for Microsoft Windows Server 2012 R2 - Mellanox WinOF 05.35.12978 for Microsoft Windows Server 2016 Driver Version Name ------ ------- ---- mlx4eth63.sys 5.35.12978.0 Mellanox ConnectX Ethernet NDIS6.3 Driver mlx4_bus.sys 5.35.12978.0 MLX4 Bus Driver * Third-party trademarks and copyrights are the property of their respective owners. Feature Summary =============== - Added mechanism to automatically install required DLLs from KB2999226 for Windows Server 2012 and Windows Server 2012 R2 Fixes =============== N/A Important Notes =============== - After the installer completes, the system must be rebooted to complete the device driver installation - Wake on LAN is only supported with Mellanox ConnectX-3 and ConnectX-3 Pro Dual Port 10 GbE KR Blade Mezzanine Ethernet Cards ********************************************************************* Currently known issues and limitations with Mellanox Ethernet drivers and devices ********************************************************************* 1. No current support for Brocade M8428-k IO Module for Dell EMC PowerEdge M1000e chassis 2. No current support for Dell PowerConnect M8024-k IO Module for Dell EMC PowerEdge M1000e chassis 3. DCBX support does not include support for Willing Mode. Mellanox Known General Issues with WinOF 5.35 ============================================= - Activating NC-SI in WinOF v4.90.10541 may cause driver’s loading failure when using an older firmware version than 2.30.8000. Workaround: Do not enable NC-SI in machines that WinOF v4.90.10541 is installed in. - Running Ntttcp without the '-a X' flag (X >1) in a NIC configured with 10GbE, may cause low bandwidth in TCP single stream. Workaround: Run Ntttcp with '-a 8' for best performance - The UI displays the network interface’s state incorrectly even after disabling and re-enabling the "disconnected" network interfaces. Workaround: To see the correct state and link speed, perform one of the following: . Run Get-netadapter Powershell cmdlet or . Right click on that interface from 'Network Connections' and click on status - Clearing the Mellanox device counters through perfmon does not always work. Workaround: Restart the driver - When running in SR-IOV mode and Hyper-V, the same driver version should be used on both the Hypervisor and the virtual machine. - Working with user space RDMA resources (i.e. ND) in parallel with hardware reset may cause unexpected behavior. - According to the hardware architecture, Mellanox NIC devices are presented in the registry as virtual devices and not as physical devices. Workaround: Any customer programs or scripts that look into the NIC registry values should not assume it is a physical device. Mellanox Known Ethernet Issues with WinOF 5.35 ============================================= - Disabling the 'Priority & VLAN tag' in the UI which VLANID is configured, may result in sending packets with the configured VLANID. Workaround: Remove the VLANID before disabling the 'Priority & VLAN tag'. - WakeOnMagicPacket registry key is not added to the registry although WoL is supported by the driver and by the NIC. - High multicast drop rate on multicast storming. Workaround: Use 'Multicast traffic' tuning option under the performance tab.For further information, please refer to section 'Tunable Performance Parameters' in the User Manual. - When there is a stress in TCP connection establishments, some of those connections may fail. Workaround: Increase the Ring queue sizes: . ReceiveBuffers - controls the receive ring size . TransmitBuffers - controls the transmit ring size - The DCB component specifies a default traffic classification that is applied to all egress packets that do not match other classification conditions. In this case, the network adapter assigns the IEEE 802.1p priority level that is associated with the default classification to these egress packets. The default traffic classification has the following attributes: . It has a traffic classification condition of type NDIS_QOS_CONDITION_DEFAULT. . It is the first traffic classification defined in the array of NDIS_QOS_CLASSIFICATION_ELEMENT structures. - Virtual Ethernet Interfaces created by vea_man are not tuned by the automatic performance tuning script. Workaround: For optimal performance need to follow the performance tuning guide and apply relevant changes to the VEA interface - Wake on Lan (WoL) cannot be disabled on NICs which supports it. Mellanox Known QoS Issues with WinOF 5.35 ============================================= - Running Quality of Service (QoS) commands without the parameter “-PolicyStore ActiveStore” may cause machines to load without Quality of Service policy. Workaround: Store the QoS policy in the ActiveStore Mellanox Known RoCE NVGRE Issues with WinOF 5.35 ============================================= - RoCE does not support: . Traffic cannot go through the router. It works in the same subnet only. . Multicast traffic . VLAN . Layer 3 feature - In machines with heterogeneous NICs: a NIC which supports RoCE v2, and a NIC which does not support RoCE v2 the following issus might raise: . ConnectX®-3 Pro is loaded with the transport type RoCE v2 . ConnectX®-3 Pro is loaded with the transport type RoCE - When NVGRE off-load is enabled, the GRE traffic cannot be accepted as a regular L2 traffic and requires special L2_TUNNELING steering rules. In such case the GRE packets are dropped or directed to promiscuous queue. - Using different versions of RoCE in your cluster is not supported. Workaround: Use the same RoCE version in all the cluster in the Ethernet ports. - RDMA Activity counters do not count during NetworkDirect RoCE traffic. - GRE traffic steering by inner MAC and by outer MAC simultaneously is currently not supported. Workaround: Configure steering or by inner MAC, or by outer MAC. - If VMQ set filter requests are accepted without a GRE flag (i.e. requested steering by outer MAC), the GRE packets do not reach that VMQ. Workaround: N/A - Set the bus driver registry keyAcceptGREbyOuterMAC_P1/2 per port to accept GRE traffic by outer MAC and to duplicate L2 steering rule to L2_TUNNELING rule for each VMQ set filter request without GRE flag. Note: For regular NVGRE Hyper-V scenarios the value of the registry key below must be set to 0:AcceptGREbyOuterMAC_P1/2 Workaround: N/A Mellanox Known Performance Issues with WinOF 5.35 ============================================= - When using WinOF 4.40 or above, low throughput will be seen on 40GbE adapters when QoS is enabled. Workaround: Disable QoS when it is not in use: . Open a PowerShell prompt. Run: . Disable-NetAdapterQos -name where is e.g. "Ethernet 1" - perf_tuning is supported only when one of the two NUMA nodes are in use. - Running performance benchmarks for a short period of time (< 1 sec) may provide bad latency. Workaround: Set 'Rx Interrupt Moderation Profile' and 'Tx Interrupt Moderation Profile', to 'Low Latency' to avoid bad latency. Note: This may increase CPU utilization. - The driver uses optimal interrupt moderation values for 10 GbE SR-IOV VF scenario. For other scenarios, the optimal values yet to be found. - While running in a Virtual Machine (working with VF) or in a Native Machine, performance counters, if read directly or by using any tool (as Windows Task Manager), may show that no packet has been sent/received. This happens because the driver periodically examines the actual performance counters and caches the results. If the sample rate is too high, the counter values will remain the same. Workaround: Reduce the counters sample rate. Mellanox Known Hyper-V Issues with WinOF 5.35 ============================================= - When the vSwitch is detached from the ETH device while the driver is disabled, the device does not reacquire the static IP it had before the attachment of the vSwitch. When the vSwitch is attached to the ETH device while there is no link, it will not receive the device IP when the link is back up. - After attaching the vSwitch to the ETH device, changing the 'Jumbo Packet' registry key on the ETH device does not affect the vSwitch configuration and vice versa. For example, if the user sets the 'Jumbo Packet' on the ETH device to X, and the 'Jumbo Packet' on the vSwitch to X+Y, X+Y sized packets will be passed from NDIS down to the driver and they will be dropped by it. Workaround: Reattach the vSwitch to sync with the value set in the ETH device. - Unexpected behavior might occur when running in a virtualized environment and creating two virtual switches bound to each of the ports of a dual port NIC and then using both of them with two vNICs from the same VM. Workaround: N/A - In Ethernet to achieve better iperf TCP performance between a Linux VM and a Windows VM on different hosts, when using MS MUX over the Ethernet driver, use the non VMQ mode for the VMs. Workaround: N/A - After disabling and enabling a port on a guest, a ping to it may be renewed after a minute. The ARP requests sent by Windows are less frequent as the time passes. If the guest port was down for a while, it could take time until Windows decides to send another ARP request to it. Workaround: N/A - When VMQ is enabled after reset, the driver loads all the VMQs that existed before the reset. However, it is not guaranteed that each VMQ will receive the same QP number it had before the reset. This can cause some delay as a result of resetting before connectivity is reestablished. The delay is caused by the time it takes for the ARP table to update after initiating the Gratuitous ARP. Workaround: N/A Mellanox Known SR-IOV Issues with WinOF 5.35 ============================================= - Working with Windows guest OS over non-Windows SR-IOV hypervisor may result with higher latency compared to Windows Hypervisor. - When working with OEM adapter cards in SR-IOV mode, the VF is identified as a Mellanox adapter. There is no other effect on the behavior of the VF. Workaround: Verify that the FW INI contains the parameter vf_subsystem_id with the OEM encoding in - In SR-IOV mode over Hyper-V, all ports are in pure Ethernet mode and RDMA is not supported on either port. - In SR-IOV mode, enabling SR-IOV in the BIOS may change the interfaces names. If any VSwitch is bounded to an interface whose name was changed, there will not be any way to perform any operation on them. Workaround: Unbond all Vswitches from the NIC's interfaces before enabling SR-IOV in the BIOS. - In SR-IOV, 40Gbps interfaces of vSwitch interface on HyperV, and the VMNIC are identified as 10Gbps. Despite the incorrect speed identification, they still achieve 40Gbps performance. - Device name in the Device Manager does not show the correct OEM branding for SR-IOV Virtual Function devices without the updated firmware. Mellanox Known Install and Upgrade Issues with WinOF 5.35 ============================================= - Upgrading the driver while the UI is opened with the 'ConnectX NIC device' may cause the installation process to never end. Work-around: Close the UI before driver upgrade. - Canceling the installation process may leave the bus driver in a disable state. The driver appears in a yellow bang containing the following error message: 'Windows cannot start this hardware device because its configuration information (in the registry) is incomplete or damaged. (Code 19)'. Work-around: Scan for new hardware and reboot the machine. - Downgrade is not supported. Work-around: Uninstall the current version and install the older one. - Configuration is not restored when replacing a ConnectX®-3 NIC with a ConnectX®-3 Pro NIC located on the same PCI slot Work-around: Clean the old network adapter configuration prior to upgrade. - Configuration can be restored only in Windows Server 2012 and above - IPv6 configuration restore is not supported - Upon upgrade, the following Registry Key values will be overwritten with the following: *ReceiveBuffers = 512 *MaxRssProcessors = 8 *RssBaseProcNumber = 0 *NumRSSQueues = 8 *RssMaxProcNumber = 63 *RssProfile = 1 DefaultRecvRingProcessor = -1 TxInterruptProcessor = -1 TxForwardingProcessor = -1 RxIntModerationProfile = 1 TxIntModerationProfile = 1 RecvCompletionMethod = 1 SingleStream = 0 TxRingNum = 8 - Upon upgrade the following Ethernet Registry Keys will be deleted: . SendCompletionMethod . UseRSSForRawIP . UseRSSForUDP - Uninstalling the driver after upgrade won't remove the directory %ProgramFiles%\Mellanox - Uninstalling the driver when multiple of VLANs are configured never ends. Work-around: Remove the VLANs before uninstallation. - The installation process does not close any applications running in the background, and may cause a BSOD as a result of a stuck cmd. Work-around: It is recommended to close all running applications prior to upgrading the driver. - Installation/upgrade fails due to PNP failure to copy the driver files to the driver store, and the following text is printed in the event logs: Fault bucket, type 0 Event Name: PnPDriverImportError Response: Not available Attached files: C:\Users\\AppData\Local\Temp\DMI151A.tmp.log.xml C:\Program Files\Mellanox\MLNX_VPI\ETH\mlx4eth63.inf Work-around: Reboot the machine and reinstall - Installation/upgrade fails due to failure to stop the WMI service, and the following text is printed in the installation log: 'CustomAction StopWMIService returned actual error code 1603' Work-around: Kill the WMIPrvSE.exe tasks in the task manager and reinstall. - Following the upgrade of Mellanox driver to WinOF-4.60 and above or on servers with no internet access, the first PowerShell command might be stuck for ~2-3 minutes before its completed. According to the following content, this issue is related to .Net framework version or an issue with the internet access: http://www.minasi.com/forum/topic.asp?TOPIC_ID=39253 Work-around: Run the following script on the server to optimizes loading PowerShell DLLs: $Env:PATH = [Runtime.InteropServices.RuntimeEnvironment]::GetRuntimeDirectory() [AppDomain]::CurrentDomain.GetAssemblies() | % {$pt = $_.Location if (! $pt) {continue} if ($cn++) {''} $na = Split-Path -Leaf $pt Write-Host -ForegroundColor Yellow "NGENing $na" ngen install $pt } - Uninstall from the Device Manager is currently not supported - If there are disabled network interfaces or a disabled Mellanox bus driver, they will be enabled after the WinOF upgrade. - Uninstalling mlx4 bus device from the Device Manager results in the removal of performance counters for all WinOF devices. - The ucrtbse.dll file is missing in operating systems older than Windows Server 2016. This leads to failure of the installation. - The UI and installation log will show messages for all missing updates such as "missing windows updates: KB2999226" Work-Around: Install Windows update kb2999226. The update files can be downloaded from https://support.microsoft.com/en-us/kb/2999226. Mellanox Known Utilities Issues with WinOF 5.35 ============================================= - Pcap file issues for RoCE IP Based: . The packets 'capture-time' field is wrong (may be 0 or negative numbers). . For every captured packet, an additional 0 size flame is added. This appears in Wireshark as a 'malformed Ethernet packet'. - When the tracer tool traces many events, it may consume a large amount of memory (up to several GB RAM). Use the button to set maximum number of the displayed lines at a maximum buffer size. Workaround: Verify Ethernet flow control is enable to ensure a lossless link - Pcap file issues for RoCE IP Based: . The packets 'capture-time' field is wrong (may be 0 or negative numbers). . For every captured packet, an additional 0 size flame is added. This appears in Wireshark as a 'malformed Ethernet packet'. - When the tracer tool traces many events, it may consume a large amount of memory (up to several GB RAM). Mellanox Known CIM/WMI Issues with WinOF 5.35 ============================================= - Running Microsoft CIM cmdlets operations and their derived classes on classes MLNX_NetAdapterSettingData and MLNX_NetAdapterRoceSettingData is not supported. Calling those commands may cause the debugger, if connected to the machine, to assert. Work-around: Use DriverCoreSettings instead. - For PCI Gen3, PcieLinkSpeed is reported as 'Unknown' when running Get-NetAdapterHardwareInfo Powershell cmdlet - WMI does not work due to lack of permissions. Work-around: Change the execution policy. Run: Set-ExecutionPolicy AllSigned - The information that is printed in the cmdlets get-netadaptersriov and Get-MlnxPCIDeviceSetting is inconsistent. Work-around: Use only the cmdlet Get-MlnxPCIDeviceSetting. Mellanox Known ND Issues with WinOF 5.35 ============================================= - There is an interoperability problem between NDK and ND in RDMA operations during the write and read operations. However, the send operation resumes working. This happens since ND uses remote token in a network order (big endian) while NDK uses remote token in CPU order (little endian). Therefore, an inconsistency is caused between ND and NDK in RDMA operations. Work-around: An ND application that works with NDK using RDMA operations must handle this issue by changing the remote token to the appropriate byte order before sending it to NDK. - When working with the default NDv1 and NDv2 providers, the following error message might be displayed: 0xC0000120 NT_STATUS_CANCELLED This error does not affect any functionality and can be safely ignored. - Changing the default ND providers may cause random errors, such as: 0xC0000238 (NT_STATUS_ADDRESS_ALREADY_ASSOCIATED) Mellanox Known Misc Issues with WinOF 5.35 ============================================= - Windows 2016 mistakenly recognizes ConnectX-3 Pro card as a USB interface. This may disconnect the network and cause failure of critical network services. - RoCE fallback mode is not working when SR-IOV is enabled. - Errors that may occur during initial driver start in the DriverEntry function do not appear in the event log. These errors will be indicated by a yellow exclamation mark on the device, with code 37. - Several error messages from the mlx4_bus may appear in the VM Event Log. These messages (104, 122, 144) do not indicate any error, and can be safely ignored. - On hypervisor, when one Ethernet port is bind to vmswitch in which SR-IOV is enabled, Network Direct applications do not work on the second port. - Vlan_config tool allows to modify the vlan name to a name which is invalid by the vlan_config addvlan command. - A VF can initialize only one Comm Channel: either MS Comm Channel or MLNX Comm Channel. - Changing RoCE mode in a virtual machine is not supported. ################################################################ # Mellanox Technologies Inc # Mellanox Ethernet Microsoft Windows 64-bit Driver Release # November 15, 2017 # # RELEASE Version 05.50.01 ################################################################ Hardware Compatibility ====================== This release of software supports the following hardware: - Mellanox ConnectX-3 Dual Port 10 GbE KR Blade Mezzanine Ethernet Card - Mellanox ConnectX-3 Dual Port 40 GbE QSFP+ Ethernet Adapter - Mellanox ConnectX-3 Dual Port 10 GbE DA/SFP+ Ethernet Adapter - Mellanox ConnectX-3 Pro Dual Port 10 GbE KR Blade Mezzanine Ethernet Card - Mellanox ConnectX-3 Pro Dual Port 40 GbE QSFP+ Ethernet Adapter - Mellanox ConnectX-3 Pro Dual Port 10 GbE DA/SFP+ Ethernet Adapter Software Compatibility ====================== This release of software supports the following operating systems: - Microsoft Windows Server 2012 - Microsoft Windows Server 2012 R2 - Microsoft Windows Server 2016 Release Contents ================== - Mellanox WinOF 5.35 for Microsoft Windows Server 2012 - Mellanox WinOF 5.35 for Microsoft Windows Server 2012 R2 - Mellanox WinOF 5.35 for Microsoft Windows Server 2016 * Third-party trademarks and copyrights are the property of their respective owners. Feature Summary =============== - Updated driver settings for Virtual Function to receive optimal performance - Added RSS (Rx Steering Mode) monitoring counters support - Added counter for link up/down to count the number of times that the link operative state changes to down. See “Proprietary Mellanox Adapter Diagnostics Counters” in the User Manual. Fixes =============== Ethernet - Fixed an issue that led to non-optimal Out of box performance for virtual function. - Fixed an issue which caused a firmware upgrade failure (error code 9) during installation, when RoCE was disabled. - Fixed an issue where Miniport reset could lead to a driver hang when occurred during driver disabling, or to a system crash when occurred during driver shutdown. - Fixed an issue where firmware burning failed on servers with Connectx-3 and Connectx-4 devices. - Fixed an issue where link load of ports connected to virtual machines took more than 10 seconds. the issue occurred on a hyper-v VMQ setup with several virtual machines, and after running massive traffic on the virtual machines. - Fixed an issue where in a HyperV machine with VMs running, the network interface required a restart after returning from a Low Power State (sleep/hibernate). - Fixed an issue that caused the installation process to hang while checking if the RDSH service is installed. - Fixed a driver crash that occurred when the VPORT-ID on the TX packet was invalid. - Fixed a rare error that caused a freeze in the error flow during the driver’s startup. - Fixed an SR-IOV team failure caused by an unsuccessful adapter parameters update. - Fixed an issue which caused a crash in the driver properties dialog, in case more than 8 teaming ports were defined. - Fixed an issue which reported a false error for successful netsh tcp settings via performance tuning - Fixed a crash on the VF initialization error-flow. - Fixed an issue that caused occasional failures of the execution of OID_QOS_OFFLOAD_CURRENT_CAPABILITIES on Windows 2016. - Fixed the issue which caused the failure of the powershell command Get_MLNXNetAdapterSettings and the command Get_MLNXNetAdapterFlowControlSettings on servers with Connectx3/Pro and ConnectX4/LX devices. - Fixed a crash on the initial stage of the driver’s startup. - Fixed an issue where the SriovPorts settings were not deleted in the process of the driver’s uninstallation. - Fixed an issue where the nd providers were not removed in the process of the driver’s uninstallation. - Fixed an issue which caused the tools’ folder not to be added to the path environment variable when installing the driver on a VM. - Fixed an issue that generated and sent the following message to the Windows event log when using firmware 2.36.5000, whenever "Mellanox WinOF Bus Counters" was selected in Perfmon:SingleFunc_196_0_0: Execution of FW command failed. op 0x68, status 0x2, errno -1 - Fixed an issue that occasionally caused system-hang when TCP offload parameters were updated dynamically (in SR-IOV machines only). - Fixed an issue that occasionally caused system-hang upon bus driver disabling, when the encapsulation parameters were updated dynamically (in SRIOV machines only). - Updated the port type configuration as described below for adapter cards that do not support an IB port type: . ETH_roce_1_25:ETH_roce_1_25 for ConnectX-3 . ETH_rroce_2:ETH_rroce_2 for ConnectX-3 Pro - Fixed an issue which caused the driver to hang during installation process due to a deadlock. - Removed the MUP registry keys for the mlx4_bus INF for Windows Server 2016. - Fixed an issue which caused the SriovPort1NumVFs and SriovPort2NumVFs fields of the Get-MLNXPciDeviceSriovSetting command to show the value from the registry and not the actual value set by the driver. - Fixed an issue which occasionally caused BSOD when enabling VMMQ on a vport while having traffic from the VM's vport to another host. The vmswitch was created with SR-IOV enabled but the vport was not. Important Notes =============== - After the installer completes, the system must be rebooted to complete the device driver installation - Wake on LAN is only supported with Mellanox ConnectX-3 and ConnectX-3 Pro Dual Port 10 GbE KR Blade Mezzanine Ethernet Cards ********************************************************************* Currently known issues and limitations with Mellanox Ethernet drivers and devices ********************************************************************* 1. No current support for Brocade M8428-k IO Module for Dell EMC PowerEdge M1000e chassis 2. No current support for Dell PowerConnect M8024-k IO Module for Dell EMC PowerEdge M1000e chassis 3. DCBX support does not include support for Willing Mode. 5. When installing a Windows Server 2008 R2 operating system, one may see an Event Log error 'Driver PCI returned Invalid ID for child device'. This is a cosmetic error and can be ignored. Mellanox Known General Issues with WinOF 5.35 ============================================= - Activating NC-SI in WinOF v4.90.10541 may cause driver’s loading failure when using an older firmware version than 2.30.8000. Workaround: Do not enable NC-SI in machines that WinOF v4.90.10541 is installed in. - Running Ntttcp without the '-a X' flag (X >1) in a NIC configured with 10GbE, may cause low bandwidth in TCP single stream. Workaround: Run Ntttcp with '-a 8' for best performance - The UI displays the network interface’s state incorrectly even after disabling and re-enabling the "disconnected" network interfaces. Workaround: To see the correct state and link speed, perform one of the following: . Run Get-netadapter Powershell cmdlet or . Right click on that interface from 'Network Connections' and click on status - Clearing the Mellanox device counters through perfmon does not always work. Workaround: Restart the driver - When running in SR-IOV mode and Hyper-V, the same driver version should be used on both the Hypervisor and the virtual machine. - Working with user space RDMA resources (i.e. ND) in parallel with hardware reset may cause unexpected behavior. - In Windows Server 2008 R2, ConnectX-3 Pro is identified as ConnectX-3. This does not affect anything. - According to the hardware architecture, Mellanox NIC devices are presented in the registry as virtual devices and not as physical devices. Workaround: Any customer programs or scripts that look into the NIC registry values should not assume it is a physical device. Mellanox Known Ethernet Issues with WinOF 5.35 ============================================= - Disabling the 'Priority & VLAN tag' in the UI which VLANID is configured, may result in sending packets with the configured VLANID. Workaround: Remove the VLANID before disabling the 'Priority & VLAN tag'. - WakeOnMagicPacket registry key is not added to the registry although WoL is supported by the driver and by the NIC. - High multicast drop rate on multicast storming. Workaround: Use 'Multicast traffic' tuning option under the performance tab.For further information, please refer to section 'Tunable Performance Parameters' in the User Manual. - When there is a stress in TCP connection establishments, some of those connections may fail. Workaround: Increase the Ring queue sizes: . ReceiveBuffers - controls the receive ring size . TransmitBuffers - controls the transmit ring size - The DCB component specifies a default traffic classification that is applied to all egress packets that do not match other classification conditions. In this case, the network adapter assigns the IEEE 802.1p priority level that is associated with the default classification to these egress packets. The default traffic classification has the following attributes: . It has a traffic classification condition of type NDIS_QOS_CONDITION_DEFAULT. . It is the first traffic classification defined in the array of NDIS_QOS_CLASSIFICATION_ELEMENT structures. - Disabling the ethernet adapter with more than 25 VLANs configured over windows server 2008 R2 \ windows server 7, may result with a non-responding server. Workaround: Reduce the number of configured VLANs prior to disabling the ethernet adapter. - Virtual Ethernet Interfaces created by vea_man are not tuned by the automatic performance tuning script. Workaround: For optimal performance need to follow the performance tuning guide and apply relevant changes to the VEA interface - In Windows Server 2008 R2, devices created by the Teaming driver do not show the correct OEM branding in the Device Manager. - Wake on Lan (WoL) cannot be disabled on NICs which supports it. Mellanox Known QoS Issues with WinOF 5.35 ============================================= - Running Quality of Service (QoS) commands without the parameter “-PolicyStore ActiveStore” may cause machines to load without Quality of Service policy. Workaround: Store the QoS policy in the ActiveStore Mellanox Known RoCE NVGRE Issues with WinOF 5.35 ============================================= - RoCE does not support: . Traffic cannot go through the router. It works in the same subnet only. . Multicast traffic . VLAN . Layer 3 feature - In machines with heterogeneous NICs: a NIC which supports RoCE v2, and a NIC which does not support RoCE v2 the following issus might raise: . ConnectX®-3 Pro is loaded with the transport type RoCE v2 . ConnectX®-3 Pro is loaded with the transport type RoCE - When NVGRE off-load is enabled, the GRE traffic cannot be accepted as a regular L2 traffic and requires special L2_TUNNELING steering rules. In such case the GRE packets are dropped or directed to promiscuous queue. - Using different versions of RoCE in your cluster is not supported. Workaround: Use the same RoCE version in all the cluster in the Ethernet ports. - RDMA Activity counters do not count during NetworkDirect RoCE traffic. - GRE traffic steering by inner MAC and by outer MAC simultaneously is currently not supported. Workaround: Configure steering or by inner MAC, or by outer MAC. - If VMQ set filter requests are accepted without a GRE flag (i.e. requested steering by outer MAC), the GRE packets do not reach that VMQ. Workaround: N/A - Set the bus driver registry keyAcceptGREbyOuterMAC_P1/2 per port to accept GRE traffic by outer MAC and to duplicate L2 steering rule to L2_TUNNELING rule for each VMQ set filter request without GRE flag. Note: For regular NVGRE Hyper-V scenarios the value of the registry key below must be set to 0:AcceptGREbyOuterMAC_P1/2 Workaround: N/A Mellanox Known Performance Issues with WinOF 5.35 ============================================= - When using WinOF 4.40 or above, low throughput will be seen on 40GbE adapters when QoS is enabled. Workaround: Disable QoS when it is not in use: . Open a PowerShell prompt. Run: . Disable-NetAdapterQos -name where is e.g. "Ethernet 1" - perf_tuning is supported only when one of the two NUMA nodes are in use. - Running performance benchmarks for a short period of time (< 1 sec) may provide bad latency. Workaround: Set 'Rx Interrupt Moderation Profile' and 'Tx Interrupt Moderation Profile', to 'Low Latency' to avoid bad latency. Note: This may increase CPU utilization. - The driver uses optimal interrupt moderation values for 10 GbE SR-IOV VF scenario. For other scenarios, the optimal values yet to be found. - While running in a Virtual Machine (working with VF) or in a Native Machine, performance counters, if read directly or by using any tool (as Windows Task Manager), may show that no packet has been sent/received. This happens because the driver periodically examines the actual performance counters and caches the results. If the sample rate is too high, the counter values will remain the same. Workaround: Reduce the counters sample rate. Mellanox Known Hyper-V Issues with WinOF 5.35 ============================================= - When the vSwitch is detached from the ETH device while the driver is disabled, the device does not reacquire the static IP it had before the attachment of the vSwitch. When the vSwitch is attached to the ETH device while there is no link, it will not receive the device IP when the link is back up. - After attaching the vSwitch to the ETH device, changing the 'Jumbo Packet' registry key on the ETH device does not affect the vSwitch configuration and vice versa. For example, if the user sets the 'Jumbo Packet' on the ETH device to X, and the 'Jumbo Packet' on the vSwitch to X+Y, X+Y sized packets will be passed from NDIS down to the driver and they will be dropped by it. Workaround: Reattach the vSwitch to sync with the value set in the ETH device. - Unexpected behavior might occur when running in a virtualized environment and creating two virtual switches bound to each of the ports of a dual port NIC and then using both of them with two vNICs from the same VM. Workaround: N/A - In Ethernet to achieve better iperf TCP performance between a Linux VM and a Windows VM on different hosts, when using MS MUX over the Ethernet driver, use the non VMQ mode for the VMs. Workaround: N/A - After disabling and enabling a port on a guest, a ping to it may be renewed after a minute. The ARP requests sent by Windows are less frequent as the time passes. If the guest port was down for a while, it could take time until Windows decides to send another ARP request to it. Workaround: N/A - When VMQ is enabled after reset, the driver loads all the VMQs that existed before the reset. However, it is not guaranteed that each VMQ will receive the same QP number it had before the reset. This can cause some delay as a result of resetting before connectivity is reestablished. The delay is caused by the time it takes for the ARP table to update after initiating the Gratuitous ARP. Workaround: N/A Mellanox Known SR-IOV Issues with WinOF 5.35 ============================================= - Working with Windows guest OS over non-Windows SR-IOV hypervisor may result with higher latency compared to Windows Hypervisor. - When working with OEM adapter cards in SR-IOV mode, the VF is identified as a Mellanox adapter. There is no other effect on the behavior of the VF. Workaround: Verify that the FW INI contains the parameter vf_subsystem_id with the OEM encoding in - In SR-IOV mode over Hyper-V, all ports are in pure Ethernet mode and RDMA is not supported on either port. - In SR-IOV mode, enabling SR-IOV in the BIOS may change the interfaces names. If any VSwitch is bounded to an interface whose name was changed, there will not be any way to perform any operation on them. Workaround: Unbond all Vswitches from the NIC's interfaces before enabling SR-IOV in the BIOS. - In SR-IOV, 40Gbps interfaces of vSwitch interface on HyperV, and the VMNIC are identified as 10Gbps. Despite the incorrect speed identification, they still achieve 40Gbps performance. - Device name in the Device Manager does not show the correct OEM branding for SR-IOV Virtual Function devices without the updated firmware. Mellanox Known Install and Upgrade Issues with WinOF 5.35 ============================================= - Upgrading the driver while the UI is opened with the 'ConnectX NIC device' may cause the installation process to never end. Work-around: Close the UI before driver upgrade. - Canceling the installation process may leave the bus driver in a disable state. The driver appears in a yellow bang containing the following error message: 'Windows cannot start this hardware device because its configuration information (in the registry) is incomplete or damaged. (Code 19)'. Work-around: Scan for new hardware and reboot the machine. - Downgrade is not supported. Work-around: Uninstall the current version and install the older one. - Configuration is not restored when replacing a ConnectX®-3 NIC with a ConnectX®-3 Pro NIC located on the same PCI slot Work-around: Clean the old network adapter configuration prior to upgrade. - Configuration can be restored only in Windows Server 2012 and above - IPv6 configuration restore is not supported - Upon upgrade, the following Registry Key values will be overwritten with the following: *ReceiveBuffers = 512 *MaxRssProcessors = 8 *RssBaseProcNumber = 0 *NumRSSQueues = 8 *RssMaxProcNumber = 63 *RssProfile = 1 DefaultRecvRingProcessor = -1 TxInterruptProcessor = -1 TxForwardingProcessor = -1 RxIntModerationProfile = 1 TxIntModerationProfile = 1 RecvCompletionMethod = 1 SingleStream = 0 TxRingNum = 8 - Upon upgrade the following Ethernet Registry Keys will be deleted: . SendCompletionMethod . UseRSSForRawIP . UseRSSForUDP - Uninstalling the driver on Windows Server 2008 R2 with LBFO configuration results in the appearance of a pop-up window requesting to close several running applications. Work-around: Choose 'ngrade the return code will always be 0. - Uninstalling the driver after upgrade won't remove the directory %ProgramFiles%\Mellanox - Uninstalling the driver when multiple of VLANs are configured never ends. Work-around: Remove the VLANs before uninstallation. - The installation process does not close any applications running in the background, and may cause a BSOD as a result of a stuck cmd. Work-around: It is recommended to close all running applications prior to upgrading the driver. - Installation/upgrade fails due to PNP failure to copy the driver files to the driver store, and the following text is printed in the event logs: Fault bucket, type 0 Event Name: PnPDriverImportError Response: Not available Attached files: C:\Users\\AppData\Local\Temp\DMI151A.tmp.log.xml C:\Program Files\Mellanox\MLNX_VPI\ETH\mlx4eth63.inf Work-around: Reboot the machine and reinstall - Installation/upgrade fails due to failure to stop the WMI service, and the following text is printed in the installation log: 'CustomAction StopWMIService returned actual error code 1603' Work-around: Kill the WMIPrvSE.exe tasks in the task manager and reinstall. - Following the upgrade of Mellanox driver to WinOF-4.60 and above or on servers with no internet access, the first PowerShell command might be stuck for ~2-3 minutes before its completed. According to the following content, this issue is related to .Net framework version or an issue with the internet access: http://www.minasi.com/forum/topic.asp?TOPIC_ID=39253 Work-around: Run the following script on the server to optimizes loading PowerShell DLLs: $Env:PATH = [Runtime.InteropServices.RuntimeEnvironment]::GetRuntimeDirectory() [AppDomain]::CurrentDomain.GetAssemblies() | % {$pt = $_.Location if (! $pt) {continue} if ($cn++) {''} $na = Split-Path -Leaf $pt Write-Host -ForegroundColor Yellow "NGENing $na" ngen install $pt } - Uninstall from the Device Manager is currently not supported - If there are disabled network interfaces or a disabled Mellanox bus driver, they will be enabled after the WinOF upgrade. - Uninstalling mlx4 bus device from the Device Manager results in the removal of performance counters for all WinOF devices. - The ucrtbse.dll file is missing in operating systems older than Windows Server 2016. This leads to failure of the installation. - The UI and installation log will show messages for all missing updates such as "missing windows updates: KB2999226" Work-Around: Install Windows update kb2999226. The update files can be downloaded from https://support.microsoft.com/en-us/kb/2999226. Mellanox Known Utilities Issues with WinOF 5.35 ============================================= - Pcap file issues for RoCE IP Based: . The packets 'capture-time' field is wrong (may be 0 or negative numbers). . For every captured packet, an additional 0 size flame is added. This appears in Wireshark as a 'malformed Ethernet packet'. - When the tracer tool traces many events, it may consume a large amount of memory (up to several GB RAM). Use the button to set maximum number of the displayed lines at a maximum buffer size. Workaround: Verify Ethernet flow control is enable to ensure a lossless link - Pcap file issues for RoCE IP Based: . The packets 'capture-time' field is wrong (may be 0 or negative numbers). . For every captured packet, an additional 0 size flame is added. This appears in Wireshark as a 'malformed Ethernet packet'. - When the tracer tool traces many events, it may consume a large amount of memory (up to several GB RAM). Mellanox Known CIM/WMI Issues with WinOF 5.35 ============================================= - Running Microsoft CIM cmdlets operations and their derived classes on classes MLNX_NetAdapterSettingData and MLNX_NetAdapterRoceSettingData is not supported. Calling those commands may cause the debugger, if connected to the machine, to assert. Work-around: Use DriverCoreSettings instead. - For PCI Gen3, PcieLinkSpeed is reported as 'Unknown' when running Get-NetAdapterHardwareInfo Powershell cmdlet - WMI does not work due to lack of permissions. Work-around: Change the execution policy. Run: Set-ExecutionPolicy AllSigned - The information that is printed in the cmdlets get-netadaptersriov and Get-MlnxPCIDeviceSetting is inconsistent. Work-around: Use only the cmdlet Get-MlnxPCIDeviceSetting. Mellanox Known ND Issues with WinOF 5.35 ============================================= - There is an interoperability problem between NDK and ND in RDMA operations during the write and read operations. However, the send operation resumes working. This happens since ND uses remote token in a network order (big endian) while NDK uses remote token in CPU order (little endian). Therefore, an inconsistency is caused between ND and NDK in RDMA operations. Work-around: An ND application that works with NDK using RDMA operations must handle this issue by changing the remote token to the appropriate byte order before sending it to NDK. - When working with the default NDv1 and NDv2 providers, the following error message might be displayed: 0xC0000120 NT_STATUS_CANCELLED This error does not affect any functionality and can be safely ignored. - Changing the default ND providers may cause random errors, such as: 0xC0000238 (NT_STATUS_ADDRESS_ALREADY_ASSOCIATED) Mellanox Known Misc Issues with WinOF 5.35 ============================================= - Windows 2016 mistakenly recognizes ConnectX-3 Pro card as a USB interface. This may disconnect the network and cause failure of critical network services. - RoCE fallback mode is not working when SR-IOV is enabled. - Errors that may occur during initial driver start in the DriverEntry function do not appear in the event log. These errors will be indicated by a yellow exclamation mark on the device, with code 37. - Several error messages from the mlx4_bus may appear in the VM Event Log. These messages (104, 122, 144) do not indicate any error, and can be safely ignored. - On hypervisor, when one Ethernet port is bind to vmswitch in which SR-IOV is enabled, Network Direct applications do not work on the second port. - Vlan_config tool allows to modify the vlan name to a name which is invalid by the vlan_config addvlan command. - A VF can initialize only one Comm Channel: either MS Comm Channel or MLNX Comm Channel. - Changing RoCE mode in a virtual machine is not supported. ################################################################ # Mellanox Technologies Inc # Mellanox Ethernet Microsoft Windows 64-bit Driver Release # March 13, 2017 # # RELEASE Version 05.22.00 ################################################################ Hardware Compatibility ====================== This release of software supports the following hardware: - Mellanox ConnectX-3 Dual Port 10 GbE KR Blade Mezzanine Ethernet Card - Mellanox ConnectX-3 Dual Port 40 GbE QSFP+ Ethernet Adapter - Mellanox ConnectX-3 Dual Port 10 GbE DA/SFP+ Ethernet Adapter - Mellanox ConnectX-3 Pro Dual Port 10 GbE KR Blade Mezzanine Ethernet Card - Mellanox ConnectX-3 Pro Dual Port 40 GbE QSFP+ Ethernet Adapter - Mellanox ConnectX-3 Pro Dual Port 10 GbE DA/SFP+ Ethernet Adapter Software Compatibility ====================== This release of software supports the following operating systems: - Microsoft Windows Server 2008 R2 w/SP1 - Microsoft Windows Server 2012 - Microsoft Windows Server 2012 R2 Release Contents ================== - Mellanox WinOF 5.10 for Microsoft Windows Server 2008 R2 w/SP1 - Mellanox WinOF 5.10 for Microsoft Windows Server 2012 - Mellanox WinOF 5.10 for Microsoft Windows Server 2012 R2 * Third-party trademarks and copyrights are the property of their respective owners. Feature Summary =============== - Changed default performance settings for Ethernet Virtual Functions to improve latency. Fixes =============== Ethernet - Fixed an issue that caused some drivers when worked with firmware 2.36.5000 to send the following message to the Windows event log: “SingleFunc_196_0_0: Execution of FW command failed. op 0x68, status 0x2, errno -1 These messages are sent when customer runs perfmon and chooses "Mellanox WinOF Bus Counters". - Old SriovEnable registry key is now obsolete. SR-IOV is now enabled by default. - Fixed an issue which caused the system to crash due to a wrong detection of the Forwarding scenario. - Improved the notification about the port type settings events in the Windows systrem event Logs. - Fixed an issue which resulted in BSOD 0x1E (KMODE_EXCEPTION_NOT_HANDLED) when the machine had multiple network port (whether a single physical card with multiple network ports or multiple single-port cards) with some ports configured with SR-IOV enabled, and some with SR-IOV disabled. - Fixed the NetworkDirect API INDAdapter::Listen() return status when the requested port was already in use. Changed the status from the non-compliant ND_INVALID_PARAMETER to the specification-compliant ND_ADDRESS_ALREADY_EXISTS. - Added a mechanism to ensure that listeners will not receive the same ephemeral port. The driver uses the default range of ephemeral ports 49152 to 65535. - Fixed an ND applications return status when a Send request failed with RNR. Changed the status from the non-compliant STATUS_DATA_NOT_-ACCEPTED to the specification-compliant STATUS_IO_TIMEOUT. - Fixed an issue which disabled VXLAN offload when received OID_TCP_OFFLOAD_PARAMETERS with EncapsulationPacketTaskOffload Enabled and the VxlanUDPPortNumber set to 0. - Fixed wrong counting of MTT-pages shown under "Mellanox WinOF Bus Counters". - Fixed an issue which caused the CIM provider installation to fail when another provider which uses the same classes was installed. - Fixed an issue which resulted in BSOD when the NIC received illegal OID_RECEIVE_FILTER_QUEUE_PARAMETERS value with QueueID higher than reported. - Fixed an issue which prevented the TEAM interface and the VLAN interface to send and receive traffic at the same time. Now both the TEAM and the VLAN interfaces work at the same time where the TEAM interface handles all untagged frames and the VLAN interface handles the tagged frames. - Fixed an issue causing BSOD upon driver restart in VM. - Added to VSTAT the ability to show the ports’ state in all port configurations (e.g. Ethernet, RoCE) and for several cards. - Fixed an issue that caused the system to hang when the driver was installed and RDMA applications were opened. - Disabled encapsulation offload capabilities for VFs regardless of the device capability to support it. - Added event-viewer message to identify "PortType" registry in case the port is configured for a unsupported type. For example if the device supports only Ethernet and the port is configured for IB the event-viewer message will be displayed. - Fixed a bluescreen issue caused by calling NDK_FN_DISCONNECT on a not connected NDK_CONNECTOR. - Fixed an issue causing the IP fragmentation not to function properly, consequently discarding large packets, when SR-IOV was turned ON and the "Jumbo Packet" field on the "Advanced" tab of the Ethernet driver's properties (in device manager) was set to 9602 or more. - Modified the event viewer-log message displayed when the PCI virtualization does not support SR-IOV. The new message is: "SR-IOV cannot be enabled due to an error in the PCI_VIRTUALIZATION_INTERFACE. Possible reason, the machine does not support SR-IOV." - Fixed an issue causing the system to crash when creating a team from the GUI. - Fixed an issue where the TTL setting for the RDMA traffic could not be changed. - Fixed an issue which resulted in interface disabling failure, and system crash when disabling the system, when multi-streams of sent fragmented/small packets could not be sent/returned to the OS. - Fixed an issue causing a Black Screen in Windows guest OS running over non-Windows SR-IOV Hypervisor with two IPoIB adapters when the OpenSM was not running. - Fixed wrong packets handling when the number of fragments equals the maximum allowed which resulted in a crash when the fragments array was higher then the allowed value. - Fixed an issue which caused the drivers to not work properly or crash as a result of setting values lower than the their default value. - Fixed an issue causing memory leakage when uninstalling the PF driver when vSwitch is connected to the device. - Fixed a performance tuning issue when using the "Performance" tab in the Advanced property page. Performance tuning did not work as expected on certain systems that have processor groups enabled. Users saw unexpected RSS CPU's assigned. Now the following event log can be seen in logs: “Detected issues with Processor groups on the system that may cause performance degradation. Disable processor groups for best results." - Fixed an issue that caused the system to hang when the driver failed to allocate contiguous memory on driver load. - Fixed wrong MAC creation from GUIDs that start with E4 D1 D2. In addition to applying the GUIDMask, the bug used to alternate the first byte of the created MAC. This alternation may have caused problems in communication with DHCP for IP address acquiring. - Fixed an issue where the ND applications provided with the driver package would fail to run when running on a server with WinOF-2 driver package installed. - Fixed a rare issue where system would crash when miniport driver is reset. - Fixed the issue where in case of multi streams of fragmented/small packets, one or more of the Tx rings might become non-operational, and the packets might not be returned to the OS. This used to result in a failure in disabling the network interface, and caused a system crash during restart/shutdown. - Fixed an issue where driver could falsely indicate that the device is stuck in extreme CPU load conditions. - Fixed the issue of VF miniport reset that used to be caused by OIDtimeout of OID_GEN_CURRENT_PACKET_FILTER. - Fixed RSS functionality in VM guests with SR-IOV enabled and VMs running in VMQ mode with SR-IOV enabled on host. This used to result in poor performance. - Fixed the issue where encapsulation settings (NVGRE+VLXAN) were not restored after reset. - Fixed the issue where it was possible to install the Microsoft Azure 2012R2 package on older Operating Systems. - Fixed an error which could cause a freeze of the drivers in the startup in a VM with several VFs. Important Notes =============== - When installing WinOF v5.22 on Windows Server 2008 R2, Microsoft Security Advisory 3033929 is required. As of January 2016, SHA-1 hashing algorithm is deprecated. Therefore, Microsoft signs certificate driver with the SHA-2 hashing algorithm. To support the SHA-2 algorithm, you must install Microsoft Security Advisory 3033929. - After the installer completes, the system must be rebooted to complete the device driver installation - Wake on LAN is only supported with Mellanox ConnectX-3 and ConnectX-3 Pro Dual Port 10 GbE KR Blade Mezzanine Ethernet Cards ********************************************************************* Currently known issues and limitations with Mellanox Ethernet drivers and devices ********************************************************************* 1. No current support for Brocade M8428-k IO Module for Dell EMC PowerEdge M1000e chassis 2. No current support for Dell PowerConnect M8024-k IO Module for Dell EMC PowerEdge M1000e chassis 3. DCBX support does not include support for Willing Mode. 4. After installing Mellanox WinOF on Windows Server 2012 R2, one may see “WMI Error” in the Event Viewer Event Logs. This is a cosmetic error and can be ignored. 5. When installing a Windows Server 2008 R2 operating system, one may see an Event Log error “Driver PCI returned Invalid ID for child device…”. This is a cosmetic error and can be ignored. Mellanox Known General Issues with WinOF 5.22 ============================================= - Pinning all the physical memory (used by RDMA operations, such as register memory, pin user memory) on the machine, on Operating Systems prior to Windows Server 2012, may cause the machine to hang. Workaround: Avoid pining the whole machine memory in those Operating Systems. - Activating NC-SI in WinOF v4.90.10541 may cause driver’s loading failure when using an older firmware version than 2.30.8000. Workaround: Do not enable NC-SI in machines that WinOF v4.90.10541 is installed in. - Running Ntttcp without the “-a X” flag (X >1) in a NIC configured with 10GbE, may cause low bandwidth in TCP single stream. Workaround: Run Ntttcp with “-a 8” for best performance - The UI displays the network interface’s state incorrectly even after disabling and re-enabling the "disconnected" network interfaces. Workaround: To see the correct state and link speed, perform one of the following: •Run Get-netadapter Powershell cmdlet or •Right click on that interface from “Network Connections” and click on status - Clearing the Mellanox device counters through perfmon does not always work. Workaround: Restart the driver - When running in SR-IOV mode and Hyper-V, the same driver version should be used on both the Hypervisor and the virtual machine. - Working with user space RDMA resources (i.e. ND) in parallel with hardware reset may cause unexpected behavior. - In Windows Server 2008 R2, ConnectX-3 Pro is identified as ConnectX-3. This does not affect anything. - According to the hardware architecture, Mellanox NIC devices are presented in the registry as virtual devices and not as physical devices. Workaround: Any customer programs or scripts that look into the NIC registry values should not assume it is a physical device. - In Windows Server 2008 R2 and Windows 7 Client, ConnectX-3 Pro is identified as ConnectX-3. This does not affect system behavior. Workaround: N/A Mellanox Known Ethernet Issues with WinOF 5.22 ============================================= - Disabling the “Priority & VLAN tag” in the UI which VLANID is configured, may result in sending packets with the configured VLANID. Workaround: Remove the VLANID before disabling the “Priority & VLAN tag”. - WakeOnMagicPacket registry key is not added to the registry although WoL is supported by the driver and by the NIC. - High multicast drop rate on multicast storming. Workaround: Use “Multicast traffic” tuning option under the performance tab. For further information, please refer to section “Tunable Performance Parameters” in the User Manual. - When there is a stress in TCP connection establishments, some of those connections may fail. Workaround: Increase the Ring queue sizes: •ReceiveBuffers - controls the receive ring size •TransmitBuffers - controls the transmit ring size - The DCB component specifies a default traffic classification that is applied to all egress packets that do not match other classification conditions. In this case, the network adapter assigns the IEEE 802.1p priority level that is associated with the default classification to these egress packets. The default traffic classification has the following attributes: •It has a traffic classification condition of type NDIS_QOS_CONDITION_DEFAULT. •It is the first traffic classification defined in the array of NDIS_QOS_CLASSIFICATION_ELEMENT structures. - Disabling the ethernet adapter with more than 25 VLANs configured over windows server 2008 R2 \ windows server 7, may result with a non-responding server. Workaround: Reduce the number of configured VLANs prior to disabling the ethernet adapter. - Virtual Ethernet Interfaces created by vea_man are not tuned by the automatic performance tuning script. Workaround: For optimal performance need to follow the performance tuning guide and apply relevant changes to the VEA interface - In Windows Server 2008 R2, devices created by the Teaming driver do not show the correct OEM branding in the Device Manager. - Wake on Lan (WoL) cannot be disabled on NICs which supports it. Mellanox Known QoS Issues with WinOF 5.22 ============================================= - Running Quality of Service (QoS) commands without the parameter “-PolicyStore ActiveStore” may cause machines to load without Quality of Service policy. Workaround: Store the QoS policy in the ActiveStore Mellanox Known RoCE NVGRE Issues with WinOF 5.22 ============================================= - RoCE does not support: •Traffic cannot go through the router. It works in the same subnet only •Multicast traffic •VLAN •Layer 3 feature - In machines with heterogeneous NICs: a NIC which supports RoCE v2, and a NIC which does not support RoCE v2 the following issus might raise: •ConnectX®-3 Pro is loaded with the transport type RoCE v2 •ConnectX®-3 Pro is loaded with the transport type RoCE - When NVGRE off-load is enabled, the GRE traffic cannot be accepted as a regular L2 traffic and requires special L2_TUNNELING steering rules. In such case the GRE packets are dropped or directed to promiscuous queue. - Using different versions of RoCE in your cluster is not supported. Workaround: Use the same RoCE version in all the cluster in the Ethernet ports. - RDMA Activity counters do not count during NetworkDirect RoCE traffic. - GRE traffic steering by inner MAC and by outer MAC simultaneously is currently not supported. Workaround: Configure steering or by inner MAC, or by outer MAC. - If VMQ set filter requests are accepted without a GRE flag (i.e. requested steering by outer MAC), the GRE packets do not reach that VMQ. Workaround: N/A - Set the bus driver registry keyAcceptGREbyOuterMAC_P1/2 per port to accept GRE traffic by outer MAC and to duplicate L2 steering rule to L2_TUNNELING rule for each VMQ set filter request without GRE flag. Note: For regular NVGRE Hyper-V scenarios the value of the registry key below must be set to 0:AcceptGREbyOuterMAC_P1/2 Workaround: N/A Mellanox Known Performance Issues with WinOF 5.22 ============================================= - When using WinOF 4.40 or above, low throughput will be seen on 40GbE adapters when QoS is enabled. Workaround: Disable QoS when it is not in use: •Open a PowerShell prompt. Run: Disable-NetAdapterQos -name where is e.g. "Ethernet 1" - perf_tuning is supported only when one of the two NUMA nodes are in use. - Running performance benchmarks for a short period of time (< 1 sec) may provide bad latency. Workaround: Set “Rx Interrupt Moderation Profile” and “Tx Interrupt Moderation Profile”, to “Low Latency” to avoid bad latency. Note: This may increase CPU utilization. - The driver uses optimal interrupt moderation values for 10 GbE SR-IOV VF scenario. For other scenarios, the optimal values yet to be found. - While running in a Virtual Machine (working with VF) or in a Native Machine, performance counters, if read directly or by using any tool (as Windows Task Manager), may show that no packet has been sent/received. This happens because the driver periodically examines the actual performance counters and caches the results. If the sample rate is too high, the counter values will remain the same. Workaround: Reduce the counters sample rate. - Perf Tuning does not allocate RSS cores correctly on all profiles when being used on a platform with more than one Processor Group. Workaround: Apply manual tuning via PowerShell. Mellanox Known Hyper-V Issues with WinOF 5.22 ============================================= - When the vSwitch is detached from the ETH device while the driver is disabled, the device does not reacquire the static IP it had before the attachment of the vSwitch. When the vSwitch is attached to the ETH device while there is no link, it will not receive the device IP when the link is back up. - After attaching the vSwitch to the ETH device, changing the “Jumbo Packet” registry key on the ETH device does not affect the vSwitch configuration and vice versa. For example, if the user sets the “Jumbo Packet” on the ETH device to X, and the “Jumbo Packet” on the vSwitch to X+Y, X+Y sized packets will be passed from NDIS down to the driver and they will be dropped by it. Workaround: Reattach the vSwitch to sync with the value set in the ETH device. - Unexpected behavior might occur when running in a virtualized environment and creating two virtual switches bound to each of the ports of a dual port NIC and then using both of them with two vNICs from the same VM. Workaround: N/A - Hyper-V is at low bandwidth on LBFO vSwitch, Windows Server 2012 Workaround: N/A - In Ethernet to achieve better iperf TCP performance between a Linux VM and a Windows VM on different hosts, when using MS MUX over the Ethernet driver, use the non VMQ mode for the VMs. Workaround: N/A - After disabling and enabling a port on a guest, a ping to it may be renewed after a minute. The ARP requests sent by Windows are less frequent as the time passes. If the guest port was down for a while, it could take time until Windows decides to send another ARP request to it. Workaround: N/A - When VMQ is enabled after reset, the driver loads all the VMQs that existed before the reset. However, it is not guaranteed that each VMQ will receive the same QP number it had before the reset. This can cause some delay as a result of resetting before connectivity is reestablished. The delay is caused by the time it takes for the ARP table to update after initiating the Gratuitous ARP. Workaround: N/A Mellanox Known SR-IOV Issues with WinOF 5.22 ============================================= - Working with Windows guest OS over non-Windows SR-IOV hypervisor may result with higher latency compared to Windows Hypervisor. - When working with OEM adapter cards in SR-IOV mode, the VF is identified as a Mellanox adapter. There is no other effect on the behavior of the VF. Workaround: Verify that the FW INI contains the parameter vf_subsystem_id with the OEM encoding in - In SR-IOV mode over Hyper-V, all ports are in pure Ethernet mode and RDMA is not supported on either port. - In SR-IOV mode, enabling SR-IOV in the BIOS may change the interfaces names. If any VSwitch is bounded to an interface whose name was changed, there will not be any way to perform any operation on them. Workaround: Unbond all Vswitches from the NIC's interfaces before enabling SR-IOV in the BIOS. - In SR-IOV, 40Gbps interfaces of vSwitch interface on HyperV, and the VMNIC are identified as 10Gbps. Despite the incorrect speed identification, they still achieve 40Gbps performance. - Device name in the Device Manager does not show the correct OEM branding for SR-IOV Virtual Function devices without the updated firmware. Mellanox Known Install and Upgrade Issues with WinOF 5.22 ============================================= - Upgrading the driver while the UI is opened with the “ConnectX NIC device” may cause the installation process to never end. Work-around: Close the UI before driver upgrade. - Canceling the installation process may leave the bus driver in a disable state. The driver appears in a yellow bang containing the following error message: “Windows cannot start this hardware device because its configuration information (in the registry) is incomplete or damaged. (Code 19)”. Work-around: Scan for new hardware and reboot the machine. - Downgrade is not supported. Work-around: Uninstall the current version and install the older one. - Configuration is not restored when replacing a ConnectX®-3 NIC with a ConnectX®-3 Pro NIC located on the same PCI slot Work-around: Clean the old network adapter configuration prior to upgrade. - Configuration can be restored only in Windows Server 2012 and above - IPv6 configuration restore is not supported - Upon upgrade, the following Registry Key values will be overwritten with the following: •*ReceiveBuffers = 512 •*MaxRssProcessors = 8 •*RssBaseProcNumber = 0 •*NumRSSQueues = 8 •*RssMaxProcNumber = 63 •*RssProfile = 1 •DefaultRecvRingProcessor = -1 •TxInterruptProcessor = -1 •TxForwardingProcessor = -1 •RxIntModerationProfile = 1 •TxIntModerationProfile = 1 •RecvCompletionMethod = 1 •SingleStream = 0 •TxRingNum = 8 - Upon upgrade the following Ethernet Registry Keys will be deleted: •SendCompletionMethod •UseRSSForRawIP •UseRSSForUDP - Uninstalling the driver on Windows Server 2008 R2 with LBFO configuration results in the appearance of a pop-up window requesting to close several running applications. Work-around: Choose “Do not close applications”. This action allows the uninstallation of the driver. A Reboot may be required. Rebooting the server before uninstalling the driver when LBFO is configured will eliminate this pop-up completely. - Running a downgrade in silent mode is not supported. Upon downgrade the return code will always be 0. - Uninstalling the driver after upgrade won't remove the directory %ProgramFiles%\Mellanox - Uninstalling the driver when multiple of VLANs are configured never ends. Work-around: Remove the VLANs before uninstallation. - The installation process does not close any applications running in the background, and may cause a BSOD as a result of a stuck cmd. Work-around: It is recommended to close all running applications prior to upgrading the driver. - Installation/upgrade fails due to PNP failure to copy the driver files to the driver store, and the following text is printed in the event logs: Fault bucket, type 0 Event Name: PnPDriverImportError Response: Not available Attached files: C:\Users\\AppData\Local\Temp\DMI151A.tmp.log.xml C:\Program Files\Mellanox\MLNX_VPI\ETH\mlx4eth63.inf Work-around: Reboot the machine and reinstall - Installation/upgrade fails due to failure to stop the WMI service, and the following text is printed in the installation log: “CustomAction StopWMIService returned actual error code 1603” Work-around: Kill the WMIPrvSE.exe tasks in the task manager and reinstall. - Following the upgrade of Mellanox driver to WinOF-4.60 and above or on servers with no internet access, the first PowerShell command might be stuck for ~2-3 minutes before its completed. According to the following content, this issue is related to .Net framework version or an issue with the internet access: http://www.minasi.com/forum/topic.asp?TOPIC_ID=39253 Work-around: Run the following script on the server to optimizes loading PowerShell DLLs: $Env:PATH = [Runtime.InteropServices.RuntimeEnvironment]::GetRuntimeDirectory() [AppDomain]::CurrentDomain.GetAssemblies() | % {$pt = $_.Location if (! $pt) {continue} if ($cn++) {''} $na = Split-Path -Leaf $pt Write-Host -ForegroundColor Yellow "NGENing $na" ngen install $pt } - On ConnectX®-3 cards only, when upgrading from Windows Server 2012 R2 Inbox driver to WinOF, the RoCE mode setting in the registry is not properly transferred to the new driver. In case a non-default value was used it will not be configured following the upgrade. Work-around: Reconfigure the RoCE Mode setting manually. - Uninstall from the Device Manager is currently not supported - WinOF Inbox driver does not support upgrade. When installing WinOF v4.40 and above on a Windows Server 2012 and above machine, the Inbox driver is uninstalled prior to starting the new installation and any previous configurations is lost. The Inbox driver will be reinstalled automatically when the new driver is uninstalled. - Driver installation requires deletion of the mlx4_bus.sys file in Windows Server 2008 R2 and WLH OSs when using the PXE package. Work-around: Delete the mlx4_bus.sys file and reboot the machine to install the driver. - If there are disabled network interfaces or a disabled Mellanox bus driver, they will be enabled after the WinOF upgrade. - Uninstalling mlx4 bus device from the Device Manager results in the removal of performance counters for all WinOF devices. Mellanox Known Utilities Issues with WinOF 5.22 ============================================= - Pcap file issues for RoCE IP Based: •The packets 'capture-time' field is wrong (may be 0 or negative numbers). •For every captured packet, an additional 0 size flame is added. This appears in Wireshark as a 'malformed Ethernet packet'. - When the tracer tool traces many events, it may consume a large amount of memory (up to several GB RAM). Use the button to set maximum number of the displayed lines at a maximum buffer size. Workaround: Verify Ethernet flow control is enable to ensure a lossless link - Pcap file issues for RoCE IP Based: •The packets 'capture-time' field is wrong (may be 0 or negative numbers). •For every captured packet, an additional 0 size flame is added. This appears in Wireshark as a 'malformed Ethernet packet'. - When the tracer tool traces many events, it may consume a large amount of memory (up to several GB RAM). Mellanox Known CIM/WMI Issues with WinOF 5.22 ============================================= - Running Microsoft CIM cmdlets operations and their derived classes on classes MLNX_NetAdapterSettingData and MLNX_NetAdapterRoceSettingData is not supported. Calling those commands may cause the debugger, if connected to the machine, to assert. Work-around: Use DriverCoreSettings instead. - For PCI Gen3, PcieLinkSpeed is reported as “Unknown” when running Get-NetAdapterHardwareInfo Powershell cmdlet - WMI does not work due to lack of permissions. Work-around: Change the execution policy. Run: Set-ExecutionPolicy AllSigned - The information that is printed in the cmdlets get-netadaptersriov and Get-MlnxPCIDeviceSetting is inconsistent. Work-around: Use only the cmdlet Get-MlnxPCIDeviceSetting. Mellanox Known ND Issues with WinOF 5.22 ============================================= - There is an interoperability problem between NDK and ND in RDMA operations during the write and read operations. However, the send operation resumes working. This happens since ND uses remote token in a network order (big endian) while NDK uses remote token in CPU order (little endian). Therefore, an inconsistency is caused between ND and NDK in RDMA operations. Work-around: An ND application that works with NDK using RDMA operations must handle this issue by changing the remote token to the appropriate byte order before sending it to NDK. - When working with the default NDv1 and NDv2 providers, the following error message might be displayed: 0xC0000120 NT_STATUS_CANCELLED This error does not affect any functionality and can be safely ignored. - Changing the default ND providers may cause random errors, such as: 0xC0000238 (NT_STATUS_ADDRESS_ALREADY_ASSOCIATED) ################################################################ # Mellanox Technologies Inc # Mellanox Ethernet Microsoft Windows 64-bit Driver Release # March 30, 2016 # # RELEASE Version 05.10.00 ################################################################ Hardware Compatibility ====================== This release of software supports the following hardware: - Mellanox ConnectX-3 Dual Port 10 GbE KR Blade Mezzanine Ethernet Card - Mellanox ConnectX-3 Dual Port 40 GbE QSFP+ Ethernet Adapter - Mellanox ConnectX-3 Dual Port 10 GbE DA/SFP+ Ethernet Adapter - Mellanox ConnectX-3 Pro Dual Port 10 GbE KR Blade Mezzanine Ethernet Card - Mellanox ConnectX-3 Pro Dual Port 40 GbE QSFP+ Ethernet Adapter - Mellanox ConnectX-3 Pro Dual Port 10 GbE DA/SFP+ Ethernet Adapter Software Compatibility ====================== This release of software supports the following operating systems: - Microsoft Windows Server 2008 R2 w/SP1 - Microsoft Windows Server 2012 - Microsoft Windows Server 2012 R2 Release Contents ================== - Mellanox WinOF 5.10 for Microsoft Windows Server 2008 R2 w/SP1 - Mellanox WinOF 5.10 for Microsoft Windows Server 2012 - Mellanox WinOF 5.10 for Microsoft Windows Server 2012 R2 * Third-party trademarks and copyrights are the property of their respective owners. Feature Summary =============== - Memory Registration Resources (MTTs) Optimization Driver now uses a minimal number of memory registration resources when physical memory contiguity allows it - Co-installer Allows the installation of counters while installing the driver via INF mechanism - Added an RDMA test to the package to demonstrate Linux interoperability - WPP Tracing Mellanox installer now starts an automatic WPP trace session for drivers upon installation. - Excessive Pauses Prevention - Enabled a device watchdog mechanism that prevents the device from sending excessive pauses to the network for any reason - MAC Address from Primary Interface Windows 2008R2 teaming driver allows selecting the MAC address of the primary interface Fixes =============== Ethernet - Fixed the adapter name in command vea_man to not be case sensitive,. For example: vea_man -a ethernet9 is now the same as vea_man -a Ethernet9. - Fixed the issue when after removing the Virtual Ethernet Adapter, some registry keys would remain. - Enabled a device watchdog mechanism that prevents the device from sending excessive pauses to the network for any reason. - Added support for Windows 2008R2 teaming driver to allow selecting the MAC address of the primary interface. Virtualization - Fixed the issue of when enabling VMQ after SRIOV has been disabled, VMQ would not work, and the VM would fall back to paravirtualization, impacting performance. - Improved isolation of SRIOV host from VM driver issues. - Fixed a race which caused some of the configurations to be badly reinitialized during VM or host transition to VMQ mode. This bug used to cause loss of communication to the affected VM or host. RoCE - Fixed an issue where adding VLANs would fail because the driver’s internal table was not cleaned up correctly upon VLAN removal. - Fixed a memory leak caused by a race between successful finish of a Management Datagram (MAD) and canceling it. QoS - Fixed an issue when after OS initiated reset of the interface, default QoS policies were not properly restored. Important Notes =============== - After the installer completes, the system must be rebooted to complete the device driver installation - Wake on LAN is only supported with Mellanox ConnectX-3 Dual Port 10 GbE KR Blade Mezzanine Ethernet Card ********************************************************************* Currently known issues and limitations with Mellanox Ethernet drivers and devices ********************************************************************* 1. No current support for Brocade M8428-k IO Module for Dell PowerEdge M1000e chassis 2. No current support for Dell PowerConnect M8024-k IO Module for Dell PowerEdge M1000e chassis 3. DCBX support does not include support for Willing Mode. 4. After installing Mellanox WinOF on Windows Server 2012 R2, one may see “WMI Error” in the Event Viewer Event Logs. This is a cosmetic error and can be ignored. 5. When installing a Windows Server 2008 R2 operating system, one may see an Event Log error “Driver PCI returned Invalid ID for child device…”. This is a cosmetic error and can be ignored. Mellanox Known General Issues with WinOF 5.10 ============================================= - Pinning all the physical memory (used by RDMA operations, such as register memory, pin user memory) on the machine, on Operating Systems prior to Windows Server 2012, may cause the machine to hang. Workaround: Avoid pining the whole machine memory in those Operating Systems. - Activating NC-SI in WinOF v4.90.10541 may cause driver’s loading failure when using an older firmware version than 2.30.8000. Workaround: Do not enable NC-SI in machines that WinOF v4.90.10541 is installed in. - Running Ntttcp without the “-a X” flag (X >1) in a NIC configured with 10GbE, may cause low bandwidth in TCP single stream. Workaround: Run Ntttcp with “-a 8” for best performance - The UI displays the network interface’s state incorrectly even after disabling and re-enabling the "disconnected" network interfaces. Workaround: To see the correct state and link speed, perform one of the following: •Run Get-netadapter Powershell cmdlet or •Right click on that interface from “Network Connections” and click on status - Clearing the Mellanox device counters through perfmon does not always work. Workaround: Restart the driver - When running in SR-IOV mode and Hyper-V, the same driver version should be used on both the Hypervisor and the virtual machine. - Working with user space RDMA resources (i.e. ND) in parallel with hardware reset may cause unexpected behavior. - In Windows Server 2008 R2, ConnectX-3 Pro is identified as ConnectX-3. This does not affect anything. - The drivers’ functionality is limited up to 128 cores. - According to the hardware architecture, Mellanox NIC devices are presented in the registry as virtual devices and not as physical devices. Workaround: Any customer programs or scripts that look into the NIC registry values should not assume it is a physical device. Mellanox Known Ethernet Issues with WinOF 5.10 ============================================= - Disabling the “Priority & VLAN tag” in the UI which VLANID is configured, may result in sending packets with the configured VLANID. Workaround: Remove the VLANID before disabling the “Priority & VLAN tag”. - WakeOnMagicPacket registry key is not added to the registry although WoL is supported by the driver and by the NIC. - High multicast drop rate on multicast storming. Workaround: Use “Multicast traffic” tuning option under the performance tab. For further information, please refer to section “Tunable Performance Parameters” in the User Manual. - When there is a stress in TCP connection establishments, some of those connections may fail. Workaround: Increase the Ring queue sizes: •ReceiveBuffers - controls the receive ring size •TransmitBuffers - controls the transmit ring size - The DCB component specifies a default traffic classification that is applied to all egress packets that do not match other classification conditions. In this case, the network adapter assigns the IEEE 802.1p priority level that is associated with the default classification to these egress packets. The default traffic classification has the following attributes: •It has a traffic classification condition of type NDIS_QOS_CONDITION_DEFAULT. •It is the first traffic classification defined in the array of NDIS_QOS_CLASSIFICATION_ELEMENT structures. - Disabling the ethernet adapter with more than 25 VLANs configured over windows server 2008 R2 \ windows server 7, may result with a non-responding server. Workaround: Reduce the number of configured VLANs prior to disabling the ethernet adapter. - Virtual Ethernet Interfaces created by vea_man are not tuned by the automatic performance tuning script. Workaround: For optimal performance need to follow the performance tuning guide and apply relevant changes to the VEA interface - In Windows Server 2008 R2, devices created by the Teaming driver do not show the correct OEM branding in the Device Manager. - Wake on Lan (WoL) cannot be disabled on NICs which supports it. Mellanox Known QoS Issues with WinOF 5.10 ============================================= - Running Quality of Service (QoS) commands without the parameter “-PolicyStore ActiveStore” may cause machines to load without Quality of Service policy. Workaround: Store the QoS policy in the ActiveStore Mellanox Known RoCE NVGRE Issues with WinOF 5.10 ============================================= - RoCE does not support: •Traffic cannot go through the router. It works in the same subnet only •Multicast traffic •VLAN •Layer 3 feature - In machines with heterogeneous NICs: a NIC which supports RoCE v2, and a NIC which does not support RoCE v2 the following issus might raise: •ConnectX®-3 Pro is loaded with the transport type RoCE v2 •ConnectX®-3 Pro is loaded with the transport type RoCE - When NVGRE off-load is enabled, the GRE traffic cannot be accepted as a regular L2 traffic and requires special L2_TUNNELING steering rules. In such case the GRE packets are dropped or directed to promiscuous queue. - Using different versions of RoCE in your cluster is not supported. Workaround: Use the same RoCE version in all the cluster in the Ethernet ports. - RDMA Activity counters do not count during NetworkDirect RoCE traffic. - GRE traffic steering by inner MAC and by outer MAC simultaneously is currently not supported. Workaround: Configure steering or by inner MAC, or by outer MAC. - If VMQ set filter requests are accepted without a GRE flag (i.e. requested steering by outer MAC), the GRE packets do not reach that VMQ. Workaround: Set the bus driver registry keyAcceptGREbyOuterMAC_P1/2 per port to accept GRE traffic by outer MAC and to duplicate L2 steering rule to L2_TUNNELING rule for each VMQ set filter request without GRE flag. Note: For regular NVGRE Hyper-V scenarios the value of the registry key below must be set to 0:AcceptGREbyOuterMAC_P1/2 Mellanox Known Performance Issues with WinOF 5.10 ============================================= - When using WinOF 4.40 or above, low throughput will be seen on 40GbE adapters when QoS is enabled. Workaround: Disable QoS when it is not in use: •Open a PowerShell prompt. Run: Disable-NetAdapterQos -name where is e.g. "Ethernet 1" - perf_tuning is supported only when one of the two NUMA nodes are in use. - Running performance benchmarks for a short period of time (< 1 sec) may provide bad latency. Workaround: Set “Rx Interrupt Moderation Profile” and “Tx Interrupt Moderation Profile”, to “Low Latency” to avoid bad latency. Note: This may increase CPU utilization. - The driver uses optimal interrupt moderation values for 10 GbE SR-IOV VF scenario. For other scenarios, the optimal values yet to be found. - While running in a Virtual Machine (working with VF) or in a Native Machine, performance counters, if read directly or by using any tool (as Windows Task Manager), may show that no packet has been sent/received. This happens because the driver periodically examines the actual performance counters and caches the results. If the sample rate is too high, the counter values will remain the same. Workaround: Reduce the counters sample rate. - Perf Tuning does not allocate RSS cores correctly on all profiles when being used on a platform with more than one Processor Group. Workaround: Apply manual tuning via PowerShell. Mellanox Known Hyper-V Issues with WinOF 5.10 ============================================= - When the vSwitch is detached from the ETH device while the driver is disabled, the device does not reacquire the static IP it had before the attachment of the vSwitch. When the vSwitch is attached to the ETH device while there is no link, it will not receive the device IP when the link is back up. - After attaching the vSwitch to the ETH device, changing the “Jumbo Packet” registry key on the ETH device does not affect the vSwitch configuration and vice versa. For example, if the user sets the “Jumbo Packet” on the ETH device to X, and the “Jumbo Packet” on the vSwitch to X+Y, X+Y sized packets will be passed from NDIS down to the driver and they will be dropped by it. Workaround: Reattach the vSwitch to sync with the value set in the ETH device. - Unexpected behavior might occur when running in a virtualized environment and creating two virtual switches bound to each of the ports of a dual port NIC and then using both of them with two vNICs from the same VM. - Hyper-V is at low bandwidth on LBFO vSwitch, Windows Server 2012 - In Ethernet to achieve better iperf TCP performance between a Linux VM and a Windows VM on different hosts, when using MS MUX over the Ethernet driver, use the non VMQ mode for the VMs. - After disabling and enabling a port on a guest, a ping to it may be renewed after a minute. The ARP requests sent by Windows are less frequent as the time passes. If the guest port was down for a while, it could take time until Windows decides to send another ARP request to it. - When VMQ is enabled after reset, the driver loads all the VMQs that existed before the reset. However, it is not guaranteed that each VMQ will receive the same QP number it had before the reset. This can cause some delay as a result of resetting before connectivity is reestablished. The delay is caused by the time it takes for the ARP table to update after initiating the Gratuitous ARP. Mellanox Known SR-IOV Issues with WinOF 5.10 ============================================= - Working with Windows guest OS over non-Windows SR-IOV hypervisor may result with higher latency compared to Windows Hypervisor. - When working with OEM adapter cards in SR-IOV mode, the VF is identified as a Mellanox adapter. There is no other effect on the behavior of the VF. Workaround: Verify that the FW INI contains the parameter vf_subsystem_id with the OEM encoding in - In SR-IOV mode over Hyper-V, all ports are in pure Ethernet mode and RDMA is not supported on either port. - In SR-IOV mode, enabling SR-IOV in the BIOS may change the interfaces names. If any VSwitch is bounded to an interface whose name was changed, there will not be any way to perform any operation on them. Workaround: Unbond all Vswitches from the NIC's interfaces before enabling SR-IOV in the BIOS. - In SR-IOV, 40Gbps interfaces of vSwitch interface on HyperV, and the VMNIC are identified as 10Gbps. Despite the incorrect speed identification, they still achieve 40Gbps performance. - Device name in the Device Manager does not show the correct OEM branding for SR-IOV Virtual Function devices without the updated firmware. Mellanox Known Install and Upgrade Issues with WinOF 5.10 ============================================= - Upgrading the driver while the UI is opened with the “ConnectX NIC device” may cause the installation process to never end. Work-around: Close the UI before driver upgrade. - Canceling the installation process may leave the bus driver in a disable state. The driver appears in a yellow bang containing the following error message: “Windows cannot start this hardware device because its configuration information (in the registry) is incomplete or damaged. (Code 19)”. Work-around: Scan for new hardware and reboot the machine. - Downgrade is not supported. Work-around: Uninstall the current version and install the older one. - Configuration is not restored when replacing a ConnectX®-3 NIC with a ConnectX®-3 Pro NIC located on the same PCI slot Work-around: Clean the old network adapter configuration prior to upgrade. - Configuration can be restored only in Windows Server 2012 and above - IPv6 configuration restore is not supported - Upon upgrade, the following Registry Key values will be overwritten with the following: •*ReceiveBuffers = 512 •*MaxRssProcessors = 8 •*RssBaseProcNumber = 0 •*NumRSSQueues = 8 •*RssMaxProcNumber = 63 •*RssProfile = 1 •DefaultRecvRingProcessor = -1 •TxInterruptProcessor = -1 •TxForwardingProcessor = -1 •RxIntModerationProfile = 1 •TxIntModerationProfile = 1 •RecvCompletionMethod = 1 •SingleStream = 0 •TxRingNum = 8 - Upon upgrade the following Ethernet Registry Keys will be deleted: •SendCompletionMethod •UseRSSForRawIP •UseRSSForUDP - Uninstalling the driver on Windows Server 2008 R2 with LBFO configuration results in the appearance of a pop-up window requesting to close several running applications. Work-around: Choose “Do not close applications”. This action allows the uninstallation of the driver. A Reboot may be required. Rebooting the server before uninstalling the driver when LBFO is configured will eliminate this pop-up completely. - Running a downgrade in silent mode is not supported. Upon downgrade the return code will always be 0. - Uninstalling the driver after upgrade won't remove the directory %ProgramFiles%\Mellanox - Uninstalling the driver when multiple of VLANs are configured never ends. Work-around: Remove the VLANs before uninstallation. - The installation process does not close any applications running in the background, and may cause a BSOD as a result of a stuck cmd. Work-around: It is recommended to close all running applications prior to upgrading the driver. - Installation/upgrade fails due to PNP failure to copy the driver files to the driver store, and the following text is printed in the event logs: Fault bucket, type 0 Event Name: PnPDriverImportError Response: Not available Attached files: C:\Users\\AppData\Local\Temp\DMI151A.tmp.log.xml C:\Program Files\Mellanox\MLNX_VPI\ETH\mlx4eth63.inf Work-around: Reboot the machine and reinstall - Installation/upgrade fails due to failure to stop the WMI service, and the following text is printed in the installation log: “CustomAction StopWMIService returned actual error code 1603” Work-around: Kill the WMIPrvSE.exe tasks in the task manager and reinstall. - Following the upgrade of Mellanox driver to WinOF-4.60 and above or on servers with no internet access, the first PowerShell command might be stuck for ~2-3 minutes before its completed. According to the following content, this issue is related to .Net framework version or an issue with the internet access: http://www.minasi.com/forum/topic.asp?TOPIC_ID=39253 Work-around: Run the following script on the server to optimizes loading PowerShell DLLs: $Env:PATH = [Runtime.InteropServices.RuntimeEnvironment]::GetRuntimeDirectory() [AppDomain]::CurrentDomain.GetAssemblies() | % {$pt = $_.Location if (! $pt) {continue} if ($cn++) {''} $na = Split-Path -Leaf $pt Write-Host -ForegroundColor Yellow "NGENing $na" ngen install $pt } - On ConnectX®-3 cards only, when upgrading from Windows Server 2012 R2 Inbox driver to WinOF, the RoCE mode setting in the registry is not properly transferred to the new driver. In case a non-default value was used it will not be configured following the upgrade. Work-around: Reconfigure the RoCE Mode setting manually. - Uninstall from the Device Manager is currently not supported - WinOF Inbox driver does not support upgrade. When installing WinOF v4.40 and above on a Windows Server 2012 and above machine, the Inbox driver is uninstalled prior to starting the new installation and any previous configurations is lost. The Inbox driver will be reinstalled automatically when the new driver is uninstalled. - Driver installation requires deletion of the mlx4_bus.sys file in Windows Server 2008 R2 and WLH OSs when using the PXE package. Work-around: Delete the mlx4_bus.sys file and reboot the machine to install the driver. - If there are disabled network interfaces or a disabled Mellanox bus driver, they will be enabled after the WinOF upgrade. - Uninstalling mlx4 bus device from the Device Manager results in the removal of performance counters for all WinOF devices. Mellanox Known Utilities Issues with WinOF 5.10 ============================================= - Pcap file issues for RoCE IP Based: •The packets 'capture-time' field is wrong (may be 0 or negative numbers). •For every captured packet, an additional 0 size flame is added. This appears in Wireshark as a 'malformed Ethernet packet'. - When the tracer tool traces many events, it may consume a large amount of memory (up to several GB RAM). Use the button to set maximum number of the displayed lines at a maximum buffer size. Workaround: Verify Ethernet flow control is enable to ensure a lossless link - Pcap file issues for RoCE IP Based: •The packets 'capture-time' field is wrong (may be 0 or negative numbers). •For every captured packet, an additional 0 size flame is added. This appears in Wireshark as a 'malformed Ethernet packet'. - When the tracer tool traces many events, it may consume a large amount of memory (up to several GB RAM). Mellanox Known CIM/WMI Issues with WinOF 5.10 ============================================= - Running Microsoft CIM cmdlets operations and their derived classes on classes MLNX_NetAdapterSettingData and MLNX_NetAdapterRoceSettingData is not supported. Calling those commands may cause the debugger, if connected to the machine, to assert. Work-around: Use DriverCoreSettings instead. - For PCI Gen3, PcieLinkSpeed is reported as “Unknown” when running Get-NetAdapterHardwareInfo Powershell cmdlet - WMI does not work due to lack of permissions. Work-around: Change the execution policy. Run: Set-ExecutionPolicy AllSigned - The information that is printed in the cmdlets get-netadaptersriov and Get-MlnxPCIDeviceSetting is inconsistent. Work-around: Use only the cmdlet Get-MlnxPCIDeviceSetting. Mellanox Known ND Issues with WinOF 5.10 ============================================= - There is an interoperability problem between NDK and ND in RDMA operations during the write and read operations. However, the send operation resumes working. This happens since ND uses remote token in a network order (big endian) while NDK uses remote token in CPU order (little endian). Therefore, an inconsistency is caused between ND and NDK in RDMA operations. Work-around: An ND application that works with NDK using RDMA operations must handle this issue by changing the remote token to the appropriate byte order before sending it to NDK. - When working with the default NDv1 and NDv2 providers, the following error message might be displayed: 0xC0000120 NT_STATUS_CANCELLED This error does not affect any functionality and can be safely ignored. - Changing the default ND providers may cause random errors, such as: 0xC0000238 (NT_STATUS_ADDRESS_ALREADY_ASSOCIATED) ################################################################ # Mellanox Technologies Inc # Mellanox Ethernet Microsoft Windows 64-bit Driver Release # July 08, 2015 # # RELEASE Version 04.95.00 ################################################################ Hardware Compatibility ====================== This release of software supports the following hardware: - Mellanox ConnectX-3 Dual Port 10 GbE KR Blade Mezzanine Ethernet Card - Mellanox ConnectX-3 Dual Port 40 GbE QSFP+ Ethernet Adapter - Mellanox ConnectX-3 Dual Port 10 GbE DA/SFP+ Ethernet Adapter - Mellanox ConnectX-3 Pro Dual Port 10 GbE KR Blade Mezzanine Ethernet Card - Mellanox ConnectX-3 Pro Dual Port 40 GbE QSFP+ Ethernet Adapter - Mellanox ConnectX-3 Pro Dual Port 10 GbE DA/SFP+ Ethernet Adapter Software Compatibility ====================== This release of software supports the following operating systems: - Microsoft Windows Server 2008 R2 w/SP1 - Microsoft Windows Server 2012 - Microsoft Windows Server 2012 R2 Release Contents ================== - Mellanox WinOF 4.95 for Microsoft Windows Server 2008 R2 w/SP1 - Mellanox WinOF 4.95 for Microsoft Windows Server 2012 - Mellanox WinOF 4.95 for Microsoft Windows Server 2012 R2 * Third-party trademarks and copyrights are the property of their respective owners. Feature Summary =============== - ND Extension for Resource Pools Allows RDMA programmers to create pre-allocated pools of ND resources to reduce resource creation time - New Mode to Ignore Frame Check Sequence (FCS) in Ethernet Packets Ignores FCS warnings and enables the Ethernet packets to be received by the NIC to enable the usage of the FCS field for other applications - 2008R2 Teaming MAC Address MAC address of 2008R2 teaming driver can be set to be taken from the primary interface MAC Address used for teaming is taken from one of the interfaces without manipulation Fixes =============== Ethernet - Modified counter names in perfmon to be more clear: "Sent Pause Frames" --> "Pause Frames Sent" "Sent Pause Duration" --> "Total Transmitter Pause Duration (us)" "Rcv Pause Frames" --> "Pause Frames Received" "Rcv Pause Duration" --> "Total Requested Pause Duration (us)" - Fixed BSoD after the receive buffer’s size changes in VMQ mode - ECN is now configurable via PowerShell - Fixed Powershell setting of RoCE mode when the machine has more than one Mellanox device - Fixed reporting of NVGRE capabilities to the OS - Added a new mode that ignores FCS warnings and enables the Ethernet packets to be received by the NIC Added the option of setting the MAC address of 2008R2 teaming driver to be taken from the primary interface Added the option of taking the MAC Address used for teaming from one of the interfaces without manipulation - Fixed the issue of when raising the Receive Buffers default value (from 512) the system could crash. Changes in UI - Added the option of configuring the team MAC address to be the same as the primary adapter MAC address Troubleshooting - Added an autologger session that dumps WPP traces to file to enable easier analysis of issues - Added logging of performance counters and CPU power saving state to system snapshot tool RDMA - Fixed handling of send request with inline data larger than supported - Added the option to allow RDMA programmers to create pre-allocated pools of ND resources to reduce resource creation time Low Level Driver - Fixed the case in which the system rarely gets unstable after driver restart Important Notes =============== - After the installer completes, the system must be rebooted to complete the device driver installation - Wake on LAN is only supported with Mellanox ConnectX-3 Dual Port 10 GbE KR Blade Mezzanine Ethernet Card ********************************************************************* Currently known issues and limitations with Mellanox Ethernet drivers and devices ********************************************************************* 1. No current support for Brocade M8428-k IO Module for Dell PowerEdge M1000e chassis 2. No current support for Dell PowerConnect M8024-k IO Module for Dell PowerEdge M1000e chassis 3. DCBX support does not include support for Willing Mode. 4. After installing Mellanox WinOF on Windows Server 2012 R2, one may see “WMI Error” in the Event Viewer Event Logs. This is a cosmetic error and can be ignored. 5. When installing a Windows Server 2008 R2 operating system, one may see an Event Log error “Driver PCI returned Invalid ID for child device…”. This is a cosmetic error and can be ignored. Mellanox Known General Issues with WinOF 4.95 - Pinning all the physical memory (used by RDMA operations, such as register memory, pin user memory) on the machine, on Operating Systems prior to Windows Server 2012, may cause the machine to hang. Workaround: Avoid pining the whole machine memory in those Operating Systems. - Activating NC-SI in WinOF v4.90.10541 may cause driver’s loading failure when using an older firmware version than 2.30.8000. Workaround: Do not enable NC-SI in machines that WinOF v4.90.10541 is installed in. - Running Ntttcp without the “-a X” flag (X >1) in a NIC configured with 10GbE, may cause low bandwidth in TCP single stream. Workaround: Run Ntttcp with “-a 8” for best performance - The UI displays the network interface’s state incorrectly even after disabling and re-enabling the "disconnected" network interfaces. Workaround: To see the correct state and link speed, perform one of the following: •Run Get-netadapter Powershell cmdlet or •Right click on that interface from “Network Connections” and click on status - Clearing the Mellanox device counters through perfmon does not always work. Workaround: Restart the driver - When running in SR-IOV mode and Hyper-V, the same driver version should be used on both the Hypervisor and the virtual machine. - Working with user space RDMA resources (i.e. ND) in parallel with hardware reset may cause unexpected behavior. - In Windows Server 2008 R2, ConnectX-3 Pro is identified as ConnectX-3. This does not affect anything. - The drivers’ functionality is limited up to 128 cores. - According to the hardware architecture, Mellanox NIC devices are presented in the registry as virtual devices and not as physical devices. Workaround: Any customer programs or scripts that look into the NIC registry values should not assume it is a physical device. Mellanox Known Ethernet Issues with WinOF 4.95 - Disabling the “Priority & VLAN tag” in the UI which VLANID is configured, may result in sending packets with the configured VLANID. Workaround: Remove the VLANID before disabling the “Priority & VLAN tag”. - WakeOnMagicPacket registry key is not added to the registry although WoL is supported by the driver and by the NIC. - High multicast drop rate on multicast storming. Workaround: Use “Multicast traffic” tuning option under the performance tab. For further information, please refer to section “Tunable Performance Parameters” in the User Manual. - When there is a stress in TCP connection establishments, some of those connections may fail. Workaround: Increase the Ring queue sizes: •ReceiveBuffers - controls the receive ring size •TransmitBuffers - controls the transmit ring size - The DCB component specifies a default traffic classification that is applied to all egress packets that do not match other classification conditions. In this case, the network adapter assigns the IEEE 802.1p priority level that is associated with the default classification to these egress packets. The default traffic classification has the following attributes: •It has a traffic classification condition of type NDIS_QOS_CONDITION_DEFAULT. •It is the first traffic classification defined in the array of NDIS_QOS_CLASSIFICATION_ELEMENT structures. - Disabling the ethernet adapter with more than 25 VLANs configured over windows server 2008 R2 \ windows server 7, may result with a non-responding server. Workaround: Reduce the number of configured VLANs prior to disabling the ethernet adapter. - Virtual Ethernet Interfaces created by vea_man are not tuned by the automatic performance tuning script. Workaround: For optimal performance need to follow the performance tuning guide and apply relevant changes to the VEA interface - In Windows Server 2008 R2, devices created by the Teaming driver do not show the correct OEM branding in the Device Manager. - Wake on Lan (WoL) cannot be disabled on NICs which supports it. Mellanox Known QoS Issues with WinOF 4.95 - Running Quality of Service (QoS) commands without the parameter “-PolicyStore ActiveStore” may cause machines to load without Quality of Service policy. Workaround: Store the QoS policy in the ActiveStore Mellanox Known RoCE NVGRE Issues with WinOF 4.95 - RoCE does not support: •Traffic cannot go through the router. It works in the same subnet only •Multicast traffic •VLAN •Layer 3 feature - In machines with heterogeneous NICs: a NIC which supports RoCE v2, and a NIC which does not support RoCE v2 the following issus might raise: •ConnectX®-3 Pro is loaded with the transport type RoCE v2 •ConnectX®-3 Pro is loaded with the transport type RoCE - When NVGRE off-load is enabled, the GRE traffic cannot be accepted as a regular L2 traffic and requires special L2_TUNNELING steering rules. In such case the GRE packets are dropped or directed to promiscuous queue. - Using different versions of RoCE in your cluster is not supported. Workaround: Use the same RoCE version in all the cluster in the Ethernet ports. - RDMA Activity counters do not count during NetworkDirect RoCE traffic. - GRE traffic steering by inner MAC and by outer MAC simultaneously is currently not supported. Workaround: Configure steering or by inner MAC, or by outer MAC. - If VMQ set filter requests are accepted without a GRE flag (i.e. requested steering by outer MAC), the GRE packets do not reach that VMQ. Workaround: Set the bus driver registry keyAcceptGREbyOuterMAC_P1/2 per port to accept GRE traffic by outer MAC and to duplicate L2 steering rule to L2_TUNNELING rule for each VMQ set filter request without GRE flag. Note: For regular NVGRE Hyper-V scenarios the value of the registry key below must be set to 0:AcceptGREbyOuterMAC_P1/2 Mellanox Known Performance Issues with WinOF 4.95 - When using WinOF 4.40 or above, low throughput will be seen on 40GbE adapters when QoS is enabled. Workaround: Disable QoS when it is not in use: •Open a PowerShell prompt. Run: Disable-NetAdapterQos -name where is e.g. "Ethernet 1" - perf_tuning is supported only when one of the two NUMA nodes are in use. - Running performance benchmarks for a short period of time (< 1 sec) may provide bad latency. Workaround: Set “Rx Interrupt Moderation Profile” and “Tx Interrupt Moderation Profile”, to “Low Latency” to avoid bad latency. Note: This may increase CPU utilization. - The driver uses optimal interrupt moderation values for 10 GbE SR-IOV VF scenario. For other scenarios, the optimal values yet to be found. - While running in a Virtual Machine (working with VF) or in a Native Machine, performance counters, if read directly or by using any tool (as Windows Task Manager), may show that no packet has been sent/received. This happens because the driver periodically examines the actual performance counters and caches the results. If the sample rate is too high, the counter values will remain the same. Workaround: Reduce the counters sample rate. Mellanox Known Hyper-V Issues with WinOF 4.95 - When the vSwitch is detached from the ETH device while the driver is disabled, the device does not reacquire the static IP it had before the attachment of the vSwitch. When the vSwitch is attached to the ETH device while there is no link, it will not receive the device IP when the link is back up. - After attaching the vSwitch to the ETH device, changing the “Jumbo Packet” registry key on the ETH device does not affect the vSwitch configuration and vice versa. For example, if the user sets the “Jumbo Packet” on the ETH device to X, and the “Jumbo Packet” on the vSwitch to X+Y, X+Y sized packets will be passed from NDIS down to the driver and they will be dropped by it. Workaround: Reattach the vSwitch to sync with the value set in the ETH device. - Unexpected behavior might occur when running in a virtualized environment and creating two virtual switches bound to each of the ports of a dual port NIC and then using both of them with two vNICs from the same VM. - In IPoIB when using long Multicast traffic from a Virtual Machine (VM) to an external host there might be up to 0.5% loss in 5% bursts - Hyper-V is at low bandwidth on LBFO vSwitch, Windows Server 2012 - In Ethernet to achieve better iperf TCP performance between a Linux VM and a Windows VM on different hosts, when using MS MUX over the Ethernet driver, use the non VMQ mode for the VMs. - After disabling and enabling a port on a guest, a ping to it may be renewed after a minute. The ARP requests sent by Windows are less frequent as the time passes. If the guest port was down for a while, it could take time until Windows decides to send another ARP request to it. - When VMQ is enabled after reset, the driver loads all the VMQs that existed before the reset. However, it is not guaranteed that each VMQ will receive the same QP number it had before the reset. This can cause some delay as a result of resetting before connectivity is reestablished. The delay is caused by the time it takes for the ARP table to update after initiating the Gratuitous ARP. Mellanox Known SR-IOV Issues with WinOF 4.95 - Working with Windows guest OS over non-Windows SR-IOV hypervisor may result with higher latency compared to Windows Hypervisor. - When working with OEM adapter cards in SR-IOV mode, the VF is identified as a Mellanox adapter. There is no other effect on the behavior of the VF. Workaround: Verify that the FW INI contains the parameter vf_subsystem_id with the OEM encoding in - In SR-IOV mode over Hyper-V, all ports are in pure Ethernet mode and RDMA is not supported on either port. - In SR-IOV mode, enabling SR-IOV in the BIOS may change the interfaces names. If any VSwitch is bounded to an interface whose name was changed, there will not be any way to perform any operation on them. Workaround: Unbond all Vswitches from the NIC's interfaces before enabling SR-IOV in the BIOS. - In SR-IOV, 40Gbps interfaces of vSwitch interface on HyperV, and the VMNIC are identified as 10Gbps. Despite the incorrect speed identification, they still achieve 40Gbps performance. - Device name in the Device Manager does not show the correct OEM branding for SR-IOV Virtual Function devices without the updated firmware. Mellanox Known Install and Upgrade Issues with WinOF 4.95 - Upgrading the driver while the UI is opened with the “ConnectX NIC device” may cause the installation process to never end. Work-around: Close the UI before driver upgrade. - Rebooting the machine while uninstalling WinOF may result in installation failure. Work-around: Delete Mellanox components from HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\DIFxApp\Components. The Mellanox components are mlx4eth63, ipoib6x and mlx4_bus - Canceling the installation process may leave the bus driver in a disable state. The driver appears in a yellow bang containing the following error message: “Windows cannot start this hardware device because its configuration information (in the registry) is incomplete or damaged. (Code 19)”. Work-around: Scan for new hardware and reboot the machine. - Downgrade is not supported. Work-around: Uninstall the current version and install the older one. - Configuration is not restored when replacing a ConnectX®-3 NIC with a ConnectX®-3 Pro NIC located on the same PCI slot Work-around: Clean the old network adapter configuration prior to upgrade. - Configuration can be restored only in Windows Server 2012 and above - IPv6 configuration restore is not supported - Upon upgrade, the following Registry Key values will be overwritten with the following: •*ReceiveBuffers = 512 •*MaxRssProcessors = 8 •*RssBaseProcNumber = 0 •*NumRSSQueues = 8 •*RssMaxProcNumber = 63 •*RssProfile = 1 •DefaultRecvRingProcessor = -1 •TxInterruptProcessor = -1 •TxForwardingProcessor = -1 •RxIntModerationProfile = 1 •TxIntModerationProfile = 1 •RecvCompletionMethod = 1 •SingleStream = 0 •TxRingNum = 8 - Upon upgrade the following Ethernet Registry Keys will be deleted: •SendCompletionMethod •UseRSSForRawIP •UseRSSForUDP - Upon upgrade the SendCompletionMethod IPoIB Registry Key value will be modified as follow: •SendCompletionMethod = 0 - Upon upgrade the following IPoIB Registry Keys will be deleted: •UseRSSForRawIP •UseRSSForUDP - Uninstalling the driver on Windows Server 2008 R2 with LBFO configuration results in the appearance of a pop-up window requesting to close several running applications. Work-around: Choose “Do not close applications”. This action allows the uninstallation of the driver. A Reboot may be required. Rebooting the server before uninstalling the driver when LBFO is configured will eliminate this pop-up completely. - Running a downgrade in silent mode is not supported. Upon downgrade the return code will always be 0. - Uninstalling the driver after upgrade won't remove the directory %ProgramFiles%\Mellanox - Uninstalling the driver when multiple of VLANs are configured never ends. Work-around: Remove the VLANs before uninstallation. - The installation process does not close any applications running in the background, and may cause a BSOD as a result of a stuck cmd. Work-around: It is recommended to close all running applications prior to upgrading the driver. - Installation/upgrade fails due to PNP failure to copy the driver files to the driver store, and the following text is printed in the event logs: Fault bucket, type 0 Event Name: PnPDriverImportError Response: Not available Attached files: C:\Users\\AppData\Local\Temp\DMI151A.tmp.log.xml C:\Program Files\Mellanox\MLNX_VPI\ETH\mlx4eth63.inf Work-around: Reboot the machine and reinstall - Installation/upgrade fails due to failure to stop the WMI service, and the following text is printed in the installation log: “CustomAction StopWMIService returned actual error code 1603” Work-around: Kill the WMIPrvSE.exe tasks in the task manager and reinstall. - Following the upgrade of Mellanox driver to WinOF-4.60 and above or on servers with no internet access, the first PowerShell command might be stuck for ~2-3 minutes before its completed. According to the following content, this issue is related to .Net framework version or an issue with the internet access: http://www.minasi.com/forum/topic.asp?TOPIC_ID=39253 Work-around: Run the following script on the server to optimizes loading PowerShell DLLs: $Env:PATH = [Runtime.InteropServices.RuntimeEnvironment]::GetRuntimeDirectory() [AppDomain]::CurrentDomain.GetAssemblies() | % {$pt = $_.Location if (! $pt) {continue} if ($cn++) {''} $na = Split-Path -Leaf $pt Write-Host -ForegroundColor Yellow "NGENing $na" ngen install $pt } - On ConnectX®-3 cards only, when upgrading from Windows Server 2012 R2 Inbox driver to WinOF, the RoCE mode setting in the registry is not properly transferred to the new driver. In case a non-default value was used it will not be configured following the upgrade. Work-around: Reconfigure the RoCE Mode setting manually. - Uninstall from the Device Manager is currently not supported - WinOF Inbox driver does not support upgrade. When installing WinOF v4.40 and above on a Windows Server 2012 and above machine, the Inbox driver is uninstalled prior to starting the new installation and any previous configurations is lost. The Inbox driver will be reinstalled automatically when the new driver is uninstalled. - Driver installation requires deletion of the mlx4_bus.sys file in Windows Server 2008 R2 and WLH OSs when using the PXE package. Work-around: Delete the mlx4_bus.sys file and reboot the machine to install the driver. - If there are disabled network interfaces or a disabled Mellanox bus driver, they will be enabled after the WinOF upgrade. - The upgrade from WinOF v4.58 to WinOF v4.90 saved only the configuration of the Ethernet interface, and did not save the configuration of the IPoIB interface. Mellanox Known Utilities Issues with WinOF 4.95 - Running ibdump on a RoCE Ethernet port may decrease the functional bandwidth due to the overhead of creating extra copy for each packet. This may lead to packet drops on the link. Work-around: Verify Ethernet flow control is enable to ensure a lossless link - Pcap file issues for RoCE IP Based: •The packets 'capture-time' field is wrong (may be 0 or negative numbers). •For every captured packet, an additional 0 size flame is added. This appears in Wireshark as a 'malformed Ethernet packet'. - Creating virtual adapters over IPoIB using part_man is currently not supported in all OEM adapter cards. - When the tracer tool traces many events, it may consume a large amount of memory (up to several GB RAM). Use the button to set maximum number of the displayed lines at a maximum buffer size. - Packets loss is not reported by ibdump. - Running ibdump on a RoCE Ethernet port may decrease the functional bandwidth due to the overhead of creating extra copy for each packet. This may lead to packet drops on the link. Workaround: Verify Ethernet flow control is enable to ensure a lossless link Mellanox Known CIM/WMI Issues with WinOF 4.95 - Running Microsoft CIM cmdlets operations and their derived classes on classes MLNX_NetAdapterSettingData and MLNX_NetAdapterRoceSettingData is not supported. Calling those commands may cause the debugger, if connected to the machine, to assert. Work-around: Use DriverCoreSettings instead. - For PCI Gen3, PcieLinkSpeed is reported as “Unknown” when running Get-NetAdapterHardwareInfo Powershell cmdlet - WMI does not work due to lack of permissions. Work-around: Change the execution policy. Run: Set-ExecutionPolicy AllSigned - The information that is printed in the cmdlets get-netadaptersriov and Get-MlnxPCIDeviceSetting is inconsistent. Work-around: Use only the cmdlet Get-MlnxPCIDeviceSetting. Mellanox Known ND Issues with WinOF 4.95 - There is an interoperability problem between NDK and ND in RDMA operations during the write and read operations. However, the send operation resumes working. This happens since ND uses remote token in a network order (big endian) while NDK uses remote token in CPU order (little endian). Therefore, an inconsistency is caused between ND and NDK in RDMA operations. Work-around: An ND application that works with NDK using RDMA operations must handle this issue by changing the remote token to the appropriate byte order before sending it to NDK. - When working with the default NDv1 and NDv2 providers, the following error message might be displayed: 0xC0000120 NT_STATUS_CANCELLED This error does not affect any functionality and can be safely ignored. - Changing the default ND providers may cause random errors, such as: 0xC0000238 (NT_STATUS_ADDRESS_ALREADY_ASSOCIATED) on Connect() or with 0xC0000236 (NT_STATUS_CONNECTION_REFUSED) on Accept(). These errors can be safely ignored ################################################################ # Mellanox Technologies Inc # Mellanox Ethernet Microsoft Windows 64-bit Driver Release # January 30th, 2015 # # RELEASE Version 04.90.00 ################################################################ Hardware Compatibility ====================== This release of software supports the following hardware: - Mellanox ConnectX-3 Dual Port 10 GbE KR Blade Mezzanine Ethernet Card - Mellanox ConnectX-3 Dual Port 40 GbE QSFP+ Ethernet Adapter - Mellanox ConnectX-3 Dual Port 10 GbE DA/SFP+ Ethernet Adapter Software Compatibility ====================== This release of software supports the following operating systems: - Microsoft Windows Server 2008 R2 w/SP1 - Microsoft Windows Server 2012 - Microsoft Windows Server 2012 R2 Release Contents ================== - Mellanox WinOF 4.90 for Microsoft Windows Server 2008 R2 w/SP1 - Mellanox WinOF 4.90 for Microsoft Windows Server 2012 - Mellanox WinOF 4.90 for Microsoft Windows Server 2012 R2 * Third-party trademarks and copyrights are the property of their respective owners. Feature Summary =============== - UDP Source port - ETS with DSCP Fixes =============== Generic - Added support for new Mellanox OUI: E4:1D:2D. Mellanox card with this OUI would not work with the previous versions of WinOF - Fixed CM CPU consumption for a long period of time while in DPC context - Added to MLNX_System_Snapshot Mellanox specific counters and data from Get-Mlnx* Cmdlets Resiliency - Reset Flow improvements: redesigned and simplified it to resolve race conditions with parallel resets Ethernet - Added ETS with DSCP support - Fixed stability issues - Fixed failure when deleting Virtual Ethernet Interfaces right after creating it - Fixed duplicated values of Receive Completion Method in UI on Windows Server 2012 R2 - Improved completion memory access speed - Improved RX descriptors memory access - Improved the scalability of VMQ to allow running more VMs reliably - Enabled ETS per flow according to the DSCP value in the IP header - Reduced memory footprints of Ethernet interfaces - Changed default VMQ/VPort affinity to use the first RSS CPU - Changed behavior in VMQ/SR-IOV mode by making all traffic of the default VPORT/VMQ use the same CPU NDK - Improved CPU utilization - Changed ndkgetremotetokenfrommr() to return value in network byte order Performance - Fixed the UI crash when working with a single port - Fixed wrong ND_* performance tests duration calculation - Fixed the higher limit of ThreadPoll to be 200,000 Installation - Fixed CIM failure after installation in maintenance mode - Fixed loading of old driver after driver upgrade that requires system reboot to complete the process - Fixed RoCE disable by default after installation of WinOF in Windows 8.1 Client ND - Fixed seg fault when executing ND application with no device installed or when a wrong device identifier is used - Fixed wrong reported value of supported number of SGE in 32 bit DLLs - Increased the number of supported SGEs in 32 bit DLLs to 2 Changes in UI - Replaced the terms “LBFO” and “Bundle” with “Teaming” and “team” respectively. CIM/WMI - Fixed failure when calling Get-MlnxFirmwareIdentity if not all devices are up Important Notes =============== - After the installer completes, the system must be rebooted to complete the device driver installation - Wake on LAN is only supported with Mellanox ConnectX-3 Dual Port 10 GbE KR Blade Mezzanine Ethernet Card ********************************************************************* Currently known issues and limitations with Mellanox Ethernet drivers and devices ********************************************************************* 1. No current support for Brocade M8428-k IO Module for Dell PowerEdge M1000e chassis 2. No current support for Dell PowerConnect M8024-k IO Module for Dell PowerEdge M1000e chassis 3. DCBX support does not include support for Willing Mode. 4. After installing Mellanox WinOF on Windows Server 2012 R2, one may see “WMI Error” in the Event Viewer Event Logs. This is a cosmetic error and can be ignored. 5. When installing a Windows Server 2008 R2 operating system, one may see an Event Log error “Driver PCI returned Invalid ID for child device…”. This is a cosmetic error and can be ignored. Mellanox Known General Issues with WinOF 4.90 - Pinning all the physical memory (used by RDMA operations, such as register memory, pin user memory) on the machine, on Operating Systems prior to Windows Server 2012, may cause the machine to hang. Work-around: Avoid pining the whole machine memory in those Operating Systems. - When running applications that use ND or libibumad (such as OpenSM) the system might get to an unstable state when trying to shutdown/restart/hibernate it. Work-around: Close all applications that use ND or libibumad before performing shutdown/restart/hibernate. - Activating NC-SI in WinOF v4.90.10541 may cause driver’s loading failure when using an older firmware version than 2.30.8000. Work-around: Do not enable NC-SI in machines that WinOF v4.90.10541 is installed in. - The maximum values returned by the ib_query_ca() function (for example: max_qp, max_mr) are the upper limits of the supported resources by the device. However, it may be impossible to use these maximum values, since the actual number of any resource that can be created may be limited by the machine configuration, the amount of host memory, user permissions, and the amount of resources already in use by other users/processes. - Running Ntttcp without the “-a X” flag (X >1) in a NIC configured with 10GbE, may cause low bandwidth in TCP single stream. Work-around: Run Ntttcp with “-a 8” for best performance - Active links disappear after changing the cable connectivity from Ethernet to InfiniBand or vice versa. Work-around: Disable and enable the mlx4_bus interface from the Device Manager. - On ConnectX®-2/ConnectX®-3 Ethernet adapter cards, there is a mismatch between the GUID value returned by firmware management tools and that returned by fabric/driver utilities that read the GUID via device firmware (e.g., using ibstat). Mlxburn/flint return 0xffff as GUID while the utilities return a value derived from the MAC address. For all driver/firmware/software purposes, the latter value should be used. Work-around: Please use the GUID value returned by the fabric/driver utilities (not 0xfffff). - The UI displays the network interface’s state incorrectly even after disabling and re-enabling the "disconnected" network interfaces. Work-around: To see the correct state and link speed, perform one of the following: •Run Get-netadapter Powershell cmdlet or •Right click on that interface from “Network Connections” and click on status - WoL is not supported on Windows Server 2008 R2. - iSCSI boot over Windows Server 2008 R2 is not supported. - Clearing the Mellanox device counters through perfmon does not always work. Work-around: Restart the driver - When running in SR-IOV mode and Hyper-V, the same driver version should be used on both the Hypervisor and the virtual machine. - Working with user space RDMA resources (i.e. ND) in parallel with hardware reset may cause unexpected behavior. - In Windows Server 2008 R2 and Windows 7 Client, ConnectX-3 Pro is identified as ConnectX-3. This does not affect anything. - On rare occasions, as a result of port configuration change (IB/ETH) the UI may get stuck for up to a few minutes. This effect does not require any user action. The UI returns to its proper functionality after a few minutes. - The drivers’ functionality is limited up to 128 cores. Mellanox Known Ethernet Issues with WinOF 4.90 - Disabling the “Priority & VLAN tag” in the UI which VLANID is configured, may result in sending packets with the configured VLANID. Work-around: Remove the VLANID before disabling the “Priority & VLAN tag”. - When working with LBFO, the teamed interface disappears after machine reboot. The issue applies to Windows Server 2012 R2 Work-around: Delete the existing teamed interface and create a new one. - WakeOnMagicPacket registry key is not added to the registry although WoL is supported by the driver and by the NIC. - When the ports of the device are configured as Ethernet only, ibstat/vstat may display wrong information. - High multicast drop rate on multicast storming. Work-aroung: Use “Multicast traffic” tuning option under the performance tab. For further information, please refer to section “Tunable Performance Parameters” in the User Manual. - When there is a stress in TCP connection establishments, some of those connections may fail. Work-around: Increase the Ring queue sizes: •ReceiveBuffers - controls the receive ring size •TransmitBuffers - controls the transmit ring size - The DCB component specifies a default traffic classification that is applied to all egress packets that do not match other classification conditions. In this case, the network adapter assigns the IEEE 802.1p priority level that is associated with the default classification to these egress packets. The default traffic classification has the following attributes: •It has a traffic classification condition of type NDIS_QOS_CONDITION_DEFAULT. •It is the first traffic classification defined in the array of NDIS_QOS_CLASSIFICATION_ELEMENT structures. - Disabling the ethernet adapter with more than 25 VLANs configured over windows server 2008 R2 \ windows server 7, may result with a non-responding server. Work-around: Reduce the number of configured VLANs prior to disabling the ethernet adapter. - Virtual Ethernet Interfaces created by vea_man are not tuned by the automatic performance tuning script. Work-around: For optimal performance need to follow the performance tuning guide and apply relevant changes to the VEA interface - In Windows Server 2008 R2, devices created by the Teaming driver do not show the correct OEM branding in the Device Manager. - Wake on Lan (WoL) cannot be disabled on NICs which supports it. Mellanox Known QoS Issues with WinOF 4.90 - Running Quality of Service (QoS) commands without the parameter “-PolicyStore ActiveStore” may cause machines to load without Quality of Service policy. Work-around: Store the QoS policy in the ActiveStore Mellanox Known RoCE NVGRE Issues with WinOF 4.90 - RoCE does not support: •Traffic cannot go through the router. It works in the same subnet only •Multicast traffic •VLAN •Layer 3 feature - In machines with heterogeneous NICs: a NIC which supports RoCE v2, and a NIC which does not support RoCE v2 the following issus might raise: •ConnectX®-3 Pro is loaded with the transport type RoCE v2 •ConnectX®-3 Pro is loaded with the transport type RoCE - When NVGRE off-load is enabled, the GRE traffic cannot be accepted as a regular L2 traffic and requires special L2_TUNNELING steering rules. In such case the GRE packets are dropped or directed to promiscuous queue. - Using different versions of RoCE in your cluster is not supported. Work-around: Use the same RoCE version in all the cluster in the Ethernet ports. - RDMA Activity counters do not count during NetworkDirect RoCE traffic. - GRE traffic steering by inner MAC and by outer MAC simultaneously is currently not supported. Work-around: Configure steering or by inner MAC, or by outer MAC. - If VMQ set filter requests are accepted without a GRE flag (i.e. requested steering by outer MAC), the GRE packets do not reach that VMQ. Work-around: Set the bus driver registry keyAcceptGREbyOuterMAC_P1/2 per port to accept GRE traffic by outer MAC and to duplicate L2 steering rule to L2_TUNNELING rule for each VMQ set filter request without GRE flag. Note: For regular NVGRE Hyper-V scenarios the value of the registry key below must be set to 0:AcceptGREbyOuterMAC_P1/2 Mellanox Known Performance Issues with WinOF 4.90 - When using WinOF 4.40 or above, low throughput will be seen on 40GbE adapters when QoS is enabled. Work-around: Disable QoS when it is not in use: •Open a PowerShell prompt. Run: Disable-NetAdapterQos -name where is e.g. "Ethernet 1" - perf_tuning is supported only when one of the two NUMA nodes are in use. - Running performance benchmarks for a short period of time (< 1 sec) may provide bad latency in IPoIB and Ethernet. Work-around: Set “Rx Interrupt Moderation Profile” and “Tx Interrupt Moderation Profile”, to “Low Latency” to avoid bad latency. Note: This may increase CPU utilization. - The driver uses optimal interrupt moderation values for 10 GbE SR-IOV VF scenario. For other scenarios, the optimal values yet to be found. - While running in a Virtual Machine (working with VF) or in a Native Machine, performance counters, if read directly or by using any tool (as Windows Task Manager), may show that no packet has been sent/received. This happens because the driver periodically examines the actual performance counters and caches the results. If the sample rate is too high, the counter values will remain the same. Work-around: Reduce the counters sample rate. Mellanox Known Hyper-V Issues with WinOF 4.90 - When the vSwitch is detached from the ETH\IPoIB device while the driver is disabled, the device does not reacquire the static IP it had before the attachment of the vSwitch. When the vSwitch is attached to the ETH\IPoIB device while there is no link, it will not receive the device IP when the link is back up. - After attaching the vSwitch to the ETH\IPoIB device, changing the “Jumbo Packet” registry key on the ETH\IPoIB device does not affect the vSwitch configuration and vice versa. For example, if the user sets the “Jumbo Packet” on the ETH\IPoIB device to X, and the “Jumbo Packet” on the vSwitch to X+Y, X+Y sized packets will be passed from NDIS down to the driver and they will be dropped by it. Work-around: Reattach the vSwitch to sync with the value set in the ETH\IPoIB device. - Unexpected behavior might occur when running in a virtualized environment and creating two virtual switches bound to each of the ports of a dual port NIC and then using both of them with two vNICs from the same VM. - Hyper-V is at low bandwidth on LBFO vSwitch, Windows Server 2012 - In Ethernet to achieve better iperf TCP performance between a Linux VM and a Windows VM on different hosts, when using MS MUX over the Ethernet driver, use the non VMQ mode for the VMs. - After disabling and enabling a port on a guest, a ping to it may be renewed after a minute. The ARP requests sent by Windows are less frequent as the time passes. If the guest port was down for a while, it could take time until Windows decides to send another ARP request to it. - When VMQ is enabled after reset, the driver loads all the VMQs that existed before the reset. However, it is not guaranteed that each VMQ will receive the same QP number it had before the reset. This can cause some delay as a result of resetting before connectivity is reestablished. The delay is caused by the time it takes for the ARP table to update after initiating the Gratuitous ARP. Mellanox Known SR-IOV Issues with WinOF 4.90 - Working with Windows guest OS over non-Windows SR-IOV hypervisor may result with higher latency compared to Windows Hypervisor. - When working with OEM adapter cards in SR-IOV mode, the VF is identified as a Mellanox adapter. There is no other effect on the behavior of the VF. Work-around: Verify that the FW INI contains the parameter vf_subsystem_id with the OEM encoding in the [HCA] section. - In SR-IOV mode over Hyper-V, all ports are in pure Ethernet mode and RDMA is not supported on either port. - In SR-IOV mode, enabling SR-IOV in the BIOS may change the interfaces names. If any VSwitch is bounded to an interface whose name was changed, there will not be any way to perform any operation on them. Wprk-around: Unbond all Vswitches from the NIC's interfaces before enabling SR-IOV in the BIOS. - In SR-IOV, 40Gbps interfaces of vSwitch interface on HyperV, and the VMNIC are identified as 10Gbps. Despite the incorrect speed identification, they still achieve 40Gbps performance. - Device name in the Device Manager does not show the correct OEM branding for SR-IOV Virtual Function devices without the updated firmware. Mellanox Known Install and Upgrade Issues with WinOF 4.90 - Upgrading the driver while the UI is opened with the “ConnectX NIC device” may cause the installation process to never end. Work-around: Close the UI before driver upgrade. - Rebooting the machine while uninstalling WinOF may result in installation failure. Work-around: Delete Mellanox components from HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\DIFxApp\Components. The Mellanox components are mlx4eth63, ipoib6x and mlx4_bus - Canceling the installation process may leave the bus driver in a disable state. The driver appears in a yellow bang containing the following error message: “Windows cannot start this hardware device because its configuration information (in the registry) is incomplete or damaged. (Code 19)”. Work-around: Scan for new hardware and reboot the machine. - Downgrade is not supported. Work-around: Uninstall the current version and install the older one. - Configuration is not restored when replacing a ConnectX®-3 NIC with a ConnectX®-3 Pro NIC located on the same PCI slot Work-around: Clean the old network adapter configuration prior to upgrade. - Configuration can be restored only in Windows Server 2012 and above - IPv6 configuration restore is not supported - Upon upgrade, the following Registry Key values will be overwritten with the following: •*ReceiveBuffers = 512 •*MaxRssProcessors = 8 •*RssBaseProcNumber = 0 •*NumRSSQueues = 8 •*RssMaxProcNumber = 63 •*RssProfile = 1 •DefaultRecvRingProcessor = -1 •TxInterruptProcessor = -1 •TxForwardingProcessor = -1 •RxIntModerationProfile = 1 •TxIntModerationProfile = 1 •RecvCompletionMethod = 1 •SingleStream = 0 •TxRingNum = 8 - Upon upgrade the following Ethernet Registry Keys will be deleted: •SendCompletionMethod •UseRSSForRawIP •UseRSSForUDP - Upon upgrade the SendCompletionMethod IPoIB Registry Key value will be modified as follow: •SendCompletionMethod = 0 - Upon upgrade the following IPoIB Registry Keys will be deleted: •UseRSSForRawIP •UseRSSForUDP - Uninstalling the driver on Windows Server 2008 R2 with LBFO configuration results in the appearance of a pop-up window requesting to close several running applications. Work-around: Choose “Do not close applications”. This action allows the uninstallation of the driver. A Reboot may be required. Rebooting the server before uninstalling the driver when LBFO is configured will eliminate this pop-up completely. - Running a downgrade in silent mode is not supported. Upon downgrade the return code will always be 0. - Uninstalling the driver after upgrade won't remove the directory %ProgramFiles%\Mellanox - Uninstalling the driver when multiple of VLANs are configured never ends. Work-around: Remove the VLANs before uninstallation. - The installation process does not close any applications running in the background, and may cause a BSOD as a result of a stuck cmd. Work-around: It is recommended to close all running applications prior to upgrading the driver. - Installation/upgrade fails due to PNP failure to copy the driver files to the driver store, and the following text is printed in the event logs: Fault bucket, type 0 Event Name: PnPDriverImportError Response: Not available Attached files: C:\Users\\AppData\Local\Temp\DMI151A.tmp.log.xml C:\Program Files\Mellanox\MLNX_VPI\ETH\mlx4eth63.inf Work-around: Reboot the machine and reinstall - Installation/upgrade fails due to failure to stop the WMI service, and the following text is printed in the installation log: “CustomAction StopWMIService returned actual error code 1603” Work-around: Kill the WMIPrvSE.exe tasks in the task manager and reinstall. - Following the upgrade of Mellanox driver to WinOF-4.60 and above or on servers with no internet access, the first PowerShell command might be stuck for ~2-3 minutes before its completed. According to the following content, this issue is related to .Net framework version or an issue with the internet access: http://www.minasi.com/forum/topic.asp?TOPIC_ID=39253 Work-around: Run the following script on the server to optimizes loading PowerShell DLLs: $Env:PATH = [Runtime.InteropServices.RuntimeEnvironment]::GetRuntimeDirectory() [AppDomain]::CurrentDomain.GetAssemblies() | % {$pt = $_.Location if (! $pt) {continue} if ($cn++) {''} $na = Split-Path -Leaf $pt Write-Host -ForegroundColor Yellow "NGENing $na" ngen install $pt } - On ConnectX®-3 cards only, when upgrading from Windows Server 2012 R2 Inbox driver to WinOF, the RoCE mode setting in the registry is not properly transferred to the new driver. In case a non-default value was used it will not be configured following the upgrade. Work-around: Reconfigure the RoCE Mode setting manually. - Uninstall from the Device Manager is currently not supported - WinOF Inbox driver does not support upgrade. When installing WinOF v4.40 and above on a Windows Server 2012 and above machine, the Inbox driver is uninstalled prior to starting the new installation and any previous configurations is lost. The Inbox driver will be reinstalled automatically when the new driver is uninstalled. - Driver installation requires deletion of the mlx4_bus.sys file in Windows Server 2008 R2 and WLH OSs when using the PXE package. Work-around: Delete the mlx4_bus.sys file and reboot the machine to install the driver. - If there are disabled network interfaces or a disabled Mellanox bus driver, they will be enabled after the WinOF upgrade. - The upgrade from WinOF v4.58 to WinOF v4.90 saved only the configuration of the Ethernet interface, and did not save the configuration of the IPoIB interface. Mellanox Known Utilities Issues with WinOF 4.90 - Running ibdump on a RoCE Ethernet port may decrease the functional bandwidth due to the overhead of creating extra copy for each packet. This may lead to packet drops on the link. Work-around: Verify Ethernet flow control is enable to ensure a lossless link - Pcap file issues for RoCE IP Based: •The packets 'capture-time' field is wrong (may be 0 or negative numbers). •For every captured packet, an additional 0 size flame is added. This appears in Wireshark as a 'malformed Ethernet packet'. - Creating virtual adapters over IPoIB using part_man is currently not supported in all OEM adapter cards. - When the tracer tool traces many events, it may consume a large amount of memory (up to several GB RAM). Use the button to set maximum number of the displayed lines at a maximum buffer size. Mellanox Known CIM/WMI Issues with WinOF 4.90 - Running Microsoft CIM cmdlets operations and their derived classes on classes MLNX_NetAdapterSettingData and MLNX_NetAdapterRoceSettingData is not supported. Calling those commands may cause the debugger, if connected to the machine, to assert. Work-around: Use DriverCoreSettings instead. - For PCI Gen3, PcieLinkSpeed is reported as “Unknown” when running Get-NetAdapterHardwareInfo Powershell cmdlet - WMI does not work due to lack of permissions. Work-around: Change the execution policy. Run: Set-ExecutionPolicy AllSigned - The information that is printed in the cmdlets get-netadaptersriov and Get-MlnxPCIDeviceSetting is inconsistent. Work-around: Use only the cmdlet Get-MlnxPCIDeviceSetting. Mellanox Known ND Issues with WinOF 4.90 - There is an interoperability problem between NDK and ND in RDMA operations during the write and read operations. However, the send operation resumes working. This happens since ND uses remote token in a network order (big endian) while NDK uses remote token in CPU order (little endian). Therefore, an inconsistency is caused between ND and NDK in RDMA operations. Work-around: An ND application that works with NDK using RDMA operations must handle this issue by changing the remote token to the appropriate byte order before sending it to NDK. - When working with the default NDv1 and NDv2 providers, the following error message might be displayed: 0xC0000120 NT_STATUS_CANCELLED This error does not affect any functionality and can be safely ignored. - Changing the default ND providers may cause random errors, such as: 0xC0000238 (NT_STATUS_ADDRESS_ALREADY_ASSOCIATED) on Connect() or with 0xC0000236 (NT_STATUS_CONNECTION_REFUSED) on Accept(). These errors can be safely ignored ################################################################ # Mellanox Technologies Inc # Mellanox Ethernet Microsoft Windows 64-bit Driver Release # June 19th, 2014 # # RELEASE Version 04.60.00 ################################################################ Hardware Compatibility ====================== This release of software supports the following hardware: - Mellanox ConnectX-3 Dual Port 10 GbE KR Blade Mezzanine Ethernet Card - Mellanox ConnectX-3 Dual Port 40 GbE QSFP+ Ethernet Adapter - Mellanox ConnectX-3 Dual Port 10 GbE DA/SFP+ Ethernet Adapter Software Compatibility ====================== This release of software supports the following operating systems: - Microsoft Windows Server 2008 R2 w/SP1 - Microsoft Windows Server 2012 - Microsoft Windows Server 2012 R2 Release Contents ================== - Mellanox WinOF 4.60 for Microsoft Windows Server 2008 R2 w/SP1 - Mellanox WinOF 4.60 for Microsoft Windows Server 2012 - Mellanox WinOF 4.60 for Microsoft Windows Server 2012 R2 * Third-party trademarks and copyrights are the property of their respective owners. Feature Summary =============== - 2nd Dell release of Mellanox Ethernet Drivers supporting Mellanox ConnectX-3 Adapters on Dell PowerEdge Servers. Performance - Added support to IPv6-to-all nd_*_* tests ND - Added the ability to rearm a CQ in the kernel - Added the ability to handle LID changes - Changed connection timeout behavior. Added the STATUS_CONNECTION_REFUSED return value upon connection timeout. Ethenet - Added DSCP support over IPv4 - Added traffic profile - Changed the report link speed zero in case of disconnected network adapter - LBFO: Enabled creation of more than one team with different MAC addresses in Windows Server 2008 R2 WMI/CIM - Added ControlledBy association to IBPort Fixes =============== Generic - Random parsing failures of string registry entries - Compilation failure of "Hello_world" in the SDK - The return value of ib_query_ca() if failed to allocate resources for operation Performance - CPU utilization report in nd_*_* tests - Correct bandwidth peak results in ibv_send_bw with UD QP - Sync problems of bidirectional mode in ibv_read_bw/ibv_write_bw RoCE - RoCE mode parsing ND - Missing completions when working with Completion Queue with single entry Ethernet Ethernet - Modified the CQ size to prevent CQ overrun - LBFO: Fixed port channel teaming with CISCO switch and Fabric Extenders traffic loose in Windows Server 2008 R2 - Issue related to packets sent with corrupted VLAN header when they were meant to be untagged - Unexpected behavior upon QP asynchronous event - Stability issues - Performance improvements WMI/CIM - ConformsToProfiles association for SoftwareIdentity and DriverIdentity - Execution of all tests which were running when executing Diagnostic tests on one instance - Failure occurred when running MLNX_Card - Printing of diagnostics log - Issue preventing from get-event to show information after disabling the PCI device Important Notes =============== - After the installer completes, the system must be rebooted to complete the device driver installation - Wake on LAN is only supported with Mellanox ConnectX-3 Dual Port 10 GbE KR Blade Mezzanine Ethernet Card ********************************************************************* Currently known issues and limitations with Mellanox Ethernet drivers and devices ********************************************************************* 1. No current support for SR-IOV in Windows operating systems 2. No current support for Brocade M8428-k IO Module for Dell PowerEdge M1000e chassis 3. No current support for Dell PowerConnect M8024-k IO Module for Dell PowerEdge M1000e chassis 4. DCBX support does not include support for Willing Mode. Mellanox Known General Issues with WinOF 4.60 - Pinning all the physical memory (used by RDMA operations, such as register memory, pin user memory) on the machine, on Operating Systems prior to Windows Server 2012, may cause the machine to hang. Suggested work around: Avoid pining the whole machine memory in those Operating Systems. - When the tracer tool traces many events, it may consume a large amount of memory (up to several GB ram). Suggested work around: Reduce the verbose level. - When trying to use the following WMI classes and operations, an error or empty messages will be displayed: - MLNX_Card - MLNX_FirmwareIdentity - MLNX_FirmwareInstallationServic - MLNX_Realizes - Running Ntttcp without the “-a X” flag (X >1) in a NIC configured with 10GbE, may cause low bandwidth in TCP single stream. Suggested work around: Run Ntttcp with “-a 8” for best performance - When working with the default NDv1 and NDv2 providers, the following error message might be displayed: 0xC0000120 NT_STATUS_CANCELLED This error does not affect any functionality and can be safely ignored. - Changing the default ND providers may cause random errors, such as: 0xC0000238 (NT_STATUS_ADDRESS_ALREADY_ASSOCIATED) on Connect() or with 0xC0000236 (NT_STATUS_CONNECTION_REFUSED) on Accept(). These errors can be safely ignored. - The UI displays the network interface’s state incorrectly even after disabling and re-enabling the "disconnected" network interfaces. Suggested work around: To see the correct state and link speed, perform one of the following: - Run Get-netadapter Powershell cmdlet - Right click on that interface from "Network Connections" and click on status - Upon bad Cable Recoverable the driver will print in the event log “Unsupported cable message” and upon unsupported cable the message be “Bad Cable”. Mellanox Known Ethernet Issues with WinOF 4.60 - Disabling the "Priority & VLAN tag" in the UI which VLANID is configured, may result in sending packets with the configured VLANID. Suggested work around: Remove the VLANID before disabling the "Priority & VLAN tag". - When working with LBFO, the teamed interface disappears after machine reboot. The issue applies to Windows Server 2012 R2 Suggest work around: Delete the existing teamed interface and create a new one. - WakeOnMagicPacket registry key is not added to the registry although WoL is supported by the driver and by the NIC. - When the ports of the device are configured as Ethernet only, ibstat/vstat may display wrong information. - High multicast drop rate on multicast storming. Suggested work around: Use "Multicast traffic" tuning option under the performance tab. For further information, please refer to section “Tunable Performance Parameters” in the User Manual. - Driver installation requires deletion of the mlx4_bus.sys file in Windows Server 2008 R and WLH OSs when using the PXE package. Suggested work around: Delete the mlx4_bus.sys file and reboot the machine to install the driver - When there is a stress in TCP connection establishments, some of those connections may fail. Suggested work around: Increase the Ring queue sizes: - ReceiveBuffers - controls the receive ring size - TransmitBuffers - controls the transmit ring size - The DCB component specifies a default traffic classification that is applied to all egress packets that do not match other classification conditions. In this case, the network adapter assigns the IEEE 802.1p priority level that is associated with the default classification to these egress packets. The default traffic classification has the following attributes: - It has a traffic classification condition of type NDIS_QOS_CONDITION_DEFAULT. - It is the first traffic classification defined in the array of NDIS_QOS_CLASSIFICATION_ELEMENT structures. Mellanox Known QoS Issues with WinOF 4.60 - Running Quality of Service (QoS) commands without the parameter "-PolicyStore ActiveStore” may cause machines to load without Quality of Service policy. Suggested work around: Store the QoS policy in the ActiveStore Mellanox Known RoCE Issues with WinOF 4.60 - RoCE traffic cannot be routed. It works in the same subnet only - RoCE does not support: Multicast traffic, VLAN, Layer 3 feature - Using different versions of RoCE in your cluster is not supported. Suggested work around: Use the same RoCE version in all the cluster in the Ethernet ports Mellanox Known Performance Issues with WinOF 4.60 - When using WinOF 4.40 or later, low throughput will be seen on 40GbE adapters when QoS is enabled. Suggested work around: Disable QoS when it is not in use. Open a PowerShell prompt. Run: Disable-NetAdapter-Qos -name where is, for example, "Ethernet 1" - Perf_tuning is supported only when one of the two NUMA nodes are in use. - Execution of nd_write_lat over mlx4ndv1 or mlx4ndv2 may never end. Suggested work around: Press CTRL-C and kill the test. - Running performance benchmarks for a short period of time (< 1 sec) may provide bad latency in IPoIB and Ethernet. Suggested work around: Set “Rx Interrupt Moderation Profile” and “Tx Interrupt Moderation Profile”, to “Low Latency” to avoid bad latency. Note: This may increase CPU utilization. ################################################################ # Mellanox Technologies Inc # Mellanox Ethernet Microsoft Windows 64-bit Driver Release # September 20th, 2013 # # RELEASE Version 04.40.00 ################################################################ Software Compatibility ====================== This release of software supports the following hardware: - Mellanox ConnectX-3 Dual Port 10 GbE KR Blade Mezzanine Ethernet Card - Mellanox ConnectX-3 Dual Port 40 GbE QSFP+ Ethernet Adapter - Mellanox ConnectX-3 Dual Port 10 GbE DA/SFP+ Ethernet Adapter This release of software supports the following operating systems: - Microsoft Windows Server 2008 SP2 x64 - Microsoft Windows Server 2008 R2 w/SP1 - Microsoft Windows Small Business Server 2011 Standard edition - Microsoft Windows Server 2012 Release Contents ================== - Mellanox WinOF 4.3 for Microsoft Windows Server 2008 SP2 x64 - Mellanox WinOF 4.40 for Microsoft Windows Server 2008 R2 w/SP1 - Mellanox WinOF 4.40 for Microsoft Windows Small Business Server 2011 Standard edition - Mellanox WinOF 4.40 for Microsoft Windows Server 2012 * Third-party trademarks and copyrights are the property of their respective owners. Feature Summary =============== - Initial Dell release of Mellanox Ethernet Drivers supporting Mellanox ConnectX-3 Adapters on Dell PowerEdge Servers. Fixes =============== - Initial Release Important Notes =============== - After the installer completes, the system must be rebooted to complete the device driver installation - Wake on LAN is only supported with Mellanox ConnectX-3 Dual Port 10 GbE KR Blade Mezzanine Ethernet Card ********************************************************************* Currently known issues and limitations with Mellanox Ethernet drivers and devices ********************************************************************* 1. Disabling network device corresponding to port 1 of the Mellanox adapter may not cause the physical link to drop. No network traffic will actually be transmitted on that link. 2. No current support for SR-IOV in Windows operating systems 3. No current support for Brocade M8428-k IO Module for Dell PowerEdge M1000e chassis 4. No current support for Dell PowerConnect M8024-k IO Module for Dell PowerEdge M1000e chassis Mellanox Known General Issues with WinOF 4.40 - Pinning all the physical memory (used by RDMA operations, such as register memory, pin user memory) on the machine, on Operating Systems prior to Windows Server 2012, may cause the machine to hang. Avoid pining the whole machine memory in those Operating Systems. - When the tracer tool traces many events, it may consume a large amount of memory (up to several GB ram). Reduce the verbose level. Mellanox Known Ethernet Issues with WinOF 4.40 - Disabling the "Priority & VLAN tag" in the UI which VLANID is configured, may result in sending packets with the configured VLANID. Remove the VLANID before disabling the "Priority & VLAN tag". - WakeOnMagicPacket registry key is not added to the registry although WoL is supported by the driver and by the NIC. - When the ports of the device are configured as Ethernet only, ibstat/vstat may display wrong information. - High multicast drop rate on multicast storming. Use "Multicast traffic" tuning option under the performance tab. For further information, please refer to section “Tunable Performance Parameters” in the User Manual. - Driver installation requires deletion of the mlx4_bus.sys file in Windows Server 2008 R and WLH OSs when using the PXE package. Delete the mlx4_bus.sys file and reboot the machine to install the driver Mellanox Known RoCE Issues with WinOF 4.40 - RoCE traffic cannot go through the router. It works in the same subnet only - RoCE does not support: Multicast traffic, VLAN, Layer 3 feature - ND tests nd_write_bw/lat, nd_read_bw/lat are not functional in Windows 2008 R2. When running them, they fail and the following message is displayed: "NDOpenAdapter failed" Mellanox Known Performance Issues with WinOF 4.40 - When using WinOF 4.40 or later, low throughput will be seen on 40GbE adapters when QoS is enabled. Disable QoS when it is not in use. Open a PowerShell prompt. Run: Disable-NetAdapter-Qos -name where is, for example, "Ethernet 1" - Perf_tuning is supported only when one of the two NUMA nodes are in use.