#####################################################################
#	Mellanox Technologies Inc
#	Mellanox OFED for Linux Release Notes
#	Jan 30th, 2015
#
#	RELEASE Version 2.4-1.0.0
#####################################################################

Hardware Compatibility
======================
This release of software supports the following hardware:
- Mellanox ConnectX-3 Dual Port 10 GbE KR Blade Mezzanine Ethernet Card
- Mellanox ConnectX-3 Dual Port 40 GbE QSFP+ Ethernet Adapter
- Mellanox ConnectX-3 Dual Port 10 GbE DA/SFP+ Ethernet Adapter


Software Compatibility
======================
This release of software supports the following operating systems:
- Red Hat Enterprise Linux 6.4, x86_64
- Red Hat Enterprise Linux 6.5, x86_64
- Red Hat Enterprise Linux 7.0, x86_64
- Novell - SUSE Linux ES 11 SP3, x86_64
- Novell - SUSE Linux ES 12 SP0, x86_64


Release Contents
==================
MLNX_OFED_LINUX-2.4-1.0.0 installer

* Third-party trademarks and copyrights are the property of their respective owners.

Feature Summary 
===============
- Dell release of Mellanox OFED for Linux OSes, supporting Mellanox ConnectX-3 Ethernet Adapters on Dell PowerEdge Servers.

MLNX_OFED Rev 2.4-1.0.0 provides the following new features: 
- Verbs performance optimization 
- EQ pool management
- MAC Forwarding DataBase (FDB) status reporting
- Ethtool Settings
- Adaptive Interrupt Moderation Scheme 
- RSS support for fragmented IP datagrams
- iSER Target driver compatible with upstream Linux IO Target


Fixes
===============
Ethernet  
- LRO fixes and improvements for jumbo MTU. 
- Fixed a crash occurred when changing the number of rings (ethtool set-channels) when interface con-nected to netconsole. 
- Fixed ping issues with IP fragmented datagrams in MTUs 1600-1700. 
- The default priority to TC mapping assigns all prior-ities to TC0. This configuration achieves fairness in transmission between priorities but may cause undesirable PFC behavior where pause request for prior-ity “n” affects all other priorities. 
- Fixed dmesg warnings: "NOHZ: local_softirq_pending 08". 

mlx4_core
- Fixed "sleeping while atomic" error occurred when the driver ran many firmware commands simultaneously.

mlx4_en
- Fixed dmesg warnings: "NOHZ: local_softirq_pending 08".
- Fixed erratic report of hardware clock which caused bad report of PTP hardware Time Stamping.

mlnx.conf
- Defined mlnx.conf as a configuration file in mlnxofa_kernel RPM

SR-IOV 
- Fixed counter index allocation for VFs which enables Ethernet port statistics.

iSER 
- Fixed iSER DIX sporadic false DIF errors caused in large transfers when block merges were enabled.

Verbs 
- Fixed registration memory failure when fork was enabled and contiguous pages or ODP were used.


Important Notes
===============
- Wake on LAN is only supported with Mellanox ConnectX-3 Dual Port 10 GbE KR Blade Mezzanine Ethernet Card

*********************************************************************
Currently known issues and limitations with Mellanox Ethernet drivers and devices
*********************************************************************
1. No current support for Brocade M8428-k IO Module for Dell PowerEdge M1000e chassis

2. No current support for Dell PowerConnect M8024-k IO Module for Dell PowerEdge M1000e chassis

3. DCBX support does not include support for Willing Mode.

Mellanox Known Issues with Mellanox MLNX_OFED Rev 2.4-1.0.0

Ethernet 
- When creating more than 125 VLANs and SR-IOV mode is enabled, a kernel warning message will be printed indicating that the native VLAN is created but will not work with RoCE traffic. kernel warning: mlx4_core 0000:07:00.0: vhcr command ALLOC_RES (0xf00) slave:0 in_param 0x7e in_mod=0x107, op_mod=0x1 failed with error:0, status -28
- Kernel panic might occur during fio splice in kernels before 2.6.34-rc4. Work around: Use kernel v2.6.34-rc4 which provides the following solution: baff42a net: Fix oops from tcp_collapse() when using splice()
- In PPC systems when QoS is enabled a harmless Kernel DMA mapping error messages might appear in kernel log (iommu related issue). 
- Transmit timeout might occur on RH6.3 as a result of lost interrupt (OS issue). In this case, the following message will be shown in dmesg: do_IRQ: 0.203 No irq handler for vector (irq -1) 
- The default priority to TC mapping assigns all priorities to TC0. This configuration achieves fairness in transmission between priorities but may cause undesirable PFC behavior where pause request for priority “n” affects all other priorities.  Work-around: Run: mlnx_qos –i <dev> -p 0,1,2,3,4,5,6,7 -s ets,ets,ets,ets,ets,ets,ets,ets -t 12,13,12,13,12,13,12,13 This needs to be applied every time after loading the mlx4_en driver.
- Mixing ETS and strict QoS policies for TCs in 40GbE ports may cause inaccurate results in bandwidth division among TCs.
- Creating a VLAN with user priority >= 4 on ConnectX®-2 HCA is not supported.
- Affinity hints are not supported in Xen Hypervisor (an irqblancer issue). This causes a non-optimal IRQ affinity. Work-around: To overcome this issues, run: set_irq_affinity.sh eth<x>
- Reboot might hang in SR-IOV when using the “probe_vf” parameter with many Virtual Functions. The following message is logged in the kernel log: "waiting for eth to become free. Usage count =1"
- VXLAN may not be functional when configured over Linux bridge in RH7.0. The issue is within the bridge modules in those kernels. In Vanilla kernels above 3.16 issues is fixed.
- In RH6.4, ping may not work over VLANs that are configured over Linux bridge when the bridge has a mlx4_en interface attached to it.
- The interfaces LRO needs to be set to "OFF" manually when there is a bond configured on Mellanox interfaces with a Bridge over that bond.  Work-around: Run: ethtool -K ethX lro off 

General 
- On ConnectX-3 Ethernet adapter cards, there is a mismatch between the GUID value returned by firmware management tools and that returned by fabric/driver utilities that read the GUID via device firmware (e.g., using ibstat). Mlxburn/flint return 0xffff as GUID while the utilities return a value derived from the MAC address. For all driver/firmware/software purposes, the latter value should be used.  Work-around: Please use the GUID value returned by the fabric/driver utilities (not 0xfffff). 

VGT+
- Before adding a VLAN on the VM, the parent interface should be brought up. Otherwise VLAN creation will fail and the following message will be presented:"Fail to register network rule."
- Bringing down and up the parent interface with VLANs configured over it, may result in traffic over VLANs being lost.
- On some of the OSes the callback ndo_vlan_rx_add/kill_vid returns void, therefore ethX.Y is created. However, if VLAN Y is not listed in the set of the allowed VLANs, no traffic will pass.
- When untagged traffic is not allowed, the below message will appear on Dom0 after driver restart on DomU:
mlx4_core 0000:16:00.0: vhcr command ALLOC_RES (0xf00) slave:1 in_param 0x0 in_mod=0x207, op_mod=0x1 failed with error:0, status -1

ABI Compatibility Known Issues
- MLNX_OFED v2.3-1.0.1 is not ABI compatible with previous MLNX_OFED/OFED versions.  Work-around: Recompile the application over the new MLNX_OFED version

ConnectX®-3 Adapter Cards Family
- Using RDMA READ with a higher value than 30 SGEs in the WR might lead to "local length error".  Work-around: Do not set the value of SGEs higher than 30 when RDMA READ is used.

Verbs
- Using libnl1_1_3~26 or earlier, requires ibv_create_ah protection by a lock for multi-threaded applications.
- In MLNX_OFED v2.4-1.0.0, if several CQEs are received on a CQ, they will be coalesced and a user-space event will be triggered only once. Work-around: When getting an event, poll the CQ until it is empty.

Resiliency
- Reset Flow can run on XenServer 6 only after the active user space applications running verbs are terminated.
- SR-IOV non persistent configuration (such as VGT, VST, Host assigned GUIDs, and QP0-enabled VFs) may be lost upon Reset Flow. Work-around: Reset Admin configuration post Reset Flow 
- Upon Reset Flow or after running restart driver, Ethernet VLANs are lost. Work-around: Reset the VLANs using the ifup command.
- Restarting the driver or running connectx_port_config when Reset Flow is running might result in a kernel panic
- Networking configuration (e.g. VLANs, IPv6) should be statically defined in order to have them set after Reset Flow as of after restart driver.

Driver Start
- "Out of memory" issues may rise during drivers load depending on the values of the driver module parameters set (e.g. log_num_cq).
- When reloading/starting the driver using the /etc/init.d/openibd the following messages are displayed if there is a third party RPM or driver installed: "Module mlx4_core does not belong to MLNX_OFED" or "Module mlx4_core belong to <rpm name> which is not a part of MLNX_OFED"  work-around: Remove the third party RPM/non MLNX_OFED drivers directory, run:
"depmod" and then rerun "/etc/init.d/openibd restart"
- Occasionally, when trying to repetitively reload the nes hardware driver on SLES11 SP2, a soft lockups occurs that required reboot.
- “openibd start” unloads kernel modules that were loaded from initrd/initramfs upon boot. This affects only kernel modules which come with MLNX_OFED and are included in initrd/initramfs.
- If a Lustre storage is used, it must be fully unloaded before restarting the driver or rebooting the machine, otherwise machine might get stuck/panic. Work-around: 
1.Unmount any mounted Lustre storages: # umount<lustre_mount_point>  
2.Unload all Lustre modules: # lustre_rmmod

Performance Tools
- perftest package in MLNX_OFED v2.2-1.0.1 and onwards does not work with older versions of the driver.

Performance
- On machines with irqbalancer daemon turned off, the default InfiniBand interrupts will be routed to a single core which may cause overload and software/hardware lockups. Work-around: Execute the following script as root: set_irq_affinity.sh <interface or IB device> [2nd interface or IB device] 
- UDP receiver throughput may be lower then expected, when running over mlx4_en Ethernet driver. This is caused by the adaptive interrupt moderation routine, which sets high values of interrupt coalesc-ing, causing the driver to process large number of packets in the same interrupt, leading UDP to drop packets due to overflow in its buffers. Work-around: Disable adaptive interrupt moderation and set lower values for the interrupt coalescing manu-ally. ethtool -C <eth>X adaptive-rx off rx-usecs 64 rx-frames 24 Values above may need tuning, depending the system, configuration and link speed. 

SR-IOV
- When using legacy VMs with MLNX_OFED 2.x hypervisor, you may need to set the 'enable_64b_cqe_eqe' parameter to zero on the hypervisor. It should be set in the same way that other module parameters are set for mlx4_core at module load time. For example, add “options mlx4_core enable_64b_cqe_eqe=0” as a line in the file /etc/modprobe.d/mlx4_core.conf.
- mlx4_port1_mtu sysfs entry shows a wrong MTU number in the VM. 
- When working with a bonding device to enslave the Ethernet devices in active-backup mode and failover MAC policy in a Virtual Machine (VM), establishment of RoCE connections may fail.  Work-around: Unload the module mlx4_ib and reload it in the VM.
- Attaching or detaching a Virtual Function on SLES11 SP3 to a guest Virtual Machine while the mlx4_core driver is loaded in the Virtual Machine may cause a kernel panic in the hypervisor. work-around: Unload the mlx4_core module in the hypervisor before attaching or detaching a function to or from the guest.
- When detaching a VF without shutting down the driver from a VM and reattaching it to another VM with the same IP address for the Mellanox NIC, RoCE connections will fail  Work-around: Shut down the driver in the VM before detaching the VF.
- Enabling SR-IOV requires appending the “intel_iommu=on” option to the relevant OS in file /boot/grub/grub.conf. Without that SR-IOV cannot be loaded. 
- On various combinations of Hypervisor/OSes and Guest/OSes, an issue might occur when attaching/detaching VFs to a guest while that guest is up and running.  Work-around: Attach/detach VFs to/from a VM only while that VM is down.
- When working with SR-IOV in Xen-4.2 virtualization platform, only the built-in xen_pciback driver should be loaded. The xen_pciback module in dom0 should not be loaded, as loading them simultaneously may cause interrupts loss and cause the driver to enter the reset flow.
- The known PCI BDFs for all VFs in kernel command line should be specified by adding xen-pciback.hide For further information, please refer to http://wiki.xen.org/wiki/Xen_PCI_Passthrough
- SR-IOV UD QPs are forced by the Hypervisor to use the base GID (i.e., the GID that the VF sees in its GID entry at its paravirtualized index 0). This is needed for security, since UD QPs use Address Vectors, and any GID index may be placed in such a vector, including indices not belonging to that VF.
- Attempting to attach a PF to a VM when SR-IOV is already enabled on that PF may result in a kernel panic.
- osmtest on the Hypervisor fails when SR-IOV is enabled. However, only the test fails, OpenSM will operate correctly with the host. The failure reason is that if an mcg is already joined by the host, a subse-quent join request for that group succeeds automati-cally (even if the join parameters in the request are not correct). This success does no harm. 
-  If a VM does not support PCI hot plug, detaching an mlx4 VF and probing it to the hypervisor may cause the hypervisor to crash. 

Port Type Management
- A working IP connectivity between the RoCE devices is required when creating an address handle or modifying a QP with an address vector.
- IPv4 multicast over RoCE requires the MGID format to be as follow ::ffff:<Multicast IPv4 Address>
- DIF: When running IO over FS over DM during unstable ports, block layer BIOs merges may cause false DIF error.

Flow Steering
- IPv4 rule with source IP cannot be created in SLES 11.x OSes.
- RFS does not support UDP

Quality of Service
- QoS is not supported in XenServer, Debian 6.0 and 6.2 with uek kernel
- When QoS features are not supported by the kernel, mlnx_qos tool may crash.

Installation
- When upgrading from an earlier Mellanox OFED version, the installation script does not stop the earlier version prior to uninstalling it. Work-around: Stop the old OFED stack (/etc/init.d/openibd stop) before upgrading to this new version.
- Upgrading from the previous OFED installation to this release, does not unload the kernel module ipoib_helper. Work-around: Reboot after installing the driver

UEFI Secure Boot
- On RHEL7 and SLES12, the following error is displayed in dmesg if the Mellanox's x.509 Public Key is not added to the system:
[4671958.383506] Request for unknown module key 'Mellanox Technologies signing key: 61feb074fc7292f958419386ffdd9d5ca999e403' err -11
This error can be safely ignored as long as Secure Boot is disabled on the system.  Work-around: For further information, please refer to the User Manual section "Enrolling Mellanox's x.509 Public Key On your Systems".
- The initiator does not respect interface parameter while logging in.  Work-around: Configure each interface on a different subnet

Fork Support
- Fork support from kernel 2.6.12 and above is available provided that applications do not use threads. fork() is supported as long as the parent process does not run before the child exits or calls exec().
The former can be achieved by calling wait(childpid), and the latter can be achieved by application specific means. The Posix system() call is supported.

Ethernet Performance Counters
- In a system with more than 61 VFs, the 62nd VF and onwards is assigned with the SINKQP counter, and as a result will have no statistics, and loopback pre­vention functionality for SINK counter.
- Since each VF tries to allocate 2 more QP counter for its RoCE traffic statistics, in a system with less than 61 VFs, if there is free resources it receives new counter otherwise receives the default counter which is shared with Ethernet. In this case RoCE statistics is not available.
- In ConnectX®-3, when we enable function-based loopback prevention for Ethernet port by default (i.e., based on the QP counter index), the dropped self-loopback packets increase the IfRxErrorFrames/Octets counters.

Uplinks 
- On rare occasions, ConnectX®-3 Pro adapter card may fail to link up when performing parallel detect to 40GbE.  Work-around: Restart the driver

Resources Limitation
- The device capabilities reported may not be reached as it depends on the system on which the device is installed and whether the resource is allocated in the kernel or the userspace.
- mlx4_core can allocate up to 64 MSI-X vectors, an MSI-X vector per CPU.
- Setting more IP addresses than the available GID entries in the table results in failure and the "update_gid_table error message is displayed: GID table of port 1 is full. Can't add <address>" message.
- Registering a large amount of Memory Regions (MR) may fail because of DMA mapping issues on RHEL 7.0.
- Occasionally, a user process might experience some memory shortage and not function properly due to Linux kernel occupation of the system’s free memory for its internal cache.  Work-around: To free memory to allow it to be allocated in a user process, run the drop_caches procedure below.
Performing the following steps will cause the kernel to flush and free pages, dentries and inodes caches from memory, causing that memory to become free.
Note: As this is a non-destructive operation and dirty objects are not freeable, run `sync' first.
•To free the pagecache: echo 1 > /proc/sys/vm/drop_caches
•To free dentries and inodes: echo 2 > /proc/sys/vm/drop_caches
•To free pagecache, dentries and inodes: echo 3 > /proc/sys/vm/drop_caches

RoCE
- Not configuring the Ethernet devices or independent VMs with a unique IP address in the physical port, may result in RoCE GID table corruption.  Work-around: Restart the driver
- If RDMA_CM is not used for connection management, then the source and destination GIDs used to modify a QP or create AH should be of the same type - IPv4 or IPv6.
- MLNX_OFED v2.1-1.0.0 and onwards is not interoperable with older versions of MLNX_OFED.
- Since the number of GIDs per port is limited to 128, there cannot be more than the allowed IP addresses configured to Ethernet devices that are associated with the port. Allowed number is:
•“127” for a single function machine
•“15” for a hypervisor in a multifunction machine
•“(127-15)/n” for a guest in a multifunction machine (where n is the number of virtual functions)
- A working IP connectivity between the RoCE devices is required when creating an address handle or modifying a QP with an address vector.
- IPv4 multicast over RoCE requires the MGID format to be as follow ::ffff:<Multicast IPv4 Address>
- IP routable RoCE does not support Multicast Lis­tener Discovery (MLD) therefore, multicast traffic over IPv6 may not work as expected.
- Using GID index 0 (the default GID) is possible only if the matching IPv6 link local address is configured on the net device of the port. This behavior is possible even though the default GID is configured regardless the presence of the IPv6 address.
- Using IPv6 link local address (GID0) when VLANs are configured is not supported.

Storage
- Older versions of rescan_scsi_bus.sh may not recognize some newly created LUNs. Work-around: If encountering such issues, it is recommended to use the '-c' flag.

SRP
- MLNX_OFED SRP installation breaks the ibmvstgt and ibmvscsi symbol resolution in RHEL7.0

SRP Interop
- The driver is tested with Storage target vendors recommendations for multipath.conf extensions (ZFS, DDN, TMS, Nimbus, NetApp).

DDN Storage Fusion 10000 Target
- DDN does not accept non-default P_Key connection establishment.

Oracle Sun ZFS Storage 7420
- Ungraceful power cycle of an initiator connected with Targets DDN, Nimbus, NetApp may result in temporary "stale connection" messages when initiator reconnects.

iSER Initiator
- On SLES OSs, the ib_iser module does not load on boot  Work-around: Add a dummy interface using iscsiadm:
•# iscsiadm -m iface -I ib_iser -o new
•# iscsiadm -m iface -I ib_iser -o update -n iface.transport_name -v ib_iser
- The initiator does not respect interface parameter while logging in.  Work-around: Configure each interface on a different subnet.
- iSCSID v2.0.873 can enter an endless loop on bind error
- iSCSID may hang if target crashes during logout sequence (reproducible with TCP)

iSER Target 
- Currently only the following OSs are supported: RHEL/ContOS 7.0, SLES12
- Stress login/logout from multiple initiators may cause iSER target to panic.
- RHEL 7.0: Discovery over RDMA is not supported.
- ib_isert is unavailable on custom kernels after running the mlnx_add_kernel_support.sh script. Work-around: 
1.Add "isert=y" to the mlnx_add_kernel_support.sh script after "cat << EOF > ofed.conf".
2.Use the updated script to build MLNX_OFED for the custom kernel.

ZFS Appliance
- Connection establishment occurs twice which may cause iSER to log a stack trace.





#####################################################################
#	Mellanox Technologies Inc
#	Mellanox OFED for Linux Release Notes
#	November 14, 2014
#
#	RELEASE Version 2.3-1.0.1
#####################################################################

Hardware Compatibility
======================
This release of software supports the following hardware:
- Mellanox ConnectX-3 Dual Port 10 GbE KR Blade Mezzanine Ethernet Card
- Mellanox ConnectX-3 Dual Port 40 GbE QSFP+ Ethernet Adapter
- Mellanox ConnectX-3 Dual Port 10 GbE DA/SFP+ Ethernet Adapter


Software Compatibility
======================
This release of software supports the following operating systems:
- Red Hat Enterprise Linux 6.5,x86_64
- Red Hat Enterprise Linux 6.4,x86_64
- Novell - SUSE Linux ES 11 SP2, x86_64
- Novell - SUSE Linux ES 11 SP3, x86_64

Release Contents
==================
MLNX_OFED_LINUX-2.3-1.0.1 installer

* Third-party trademarks and copyrights are the property of their respective owners.

Feature Summary 
===============
- Dell release of Mellanox OFED for Linux OSes, supporting Mellanox ConnectX-3 Ethernet Adapters on Dell PowerEdge Servers.

MLNX_OFED Rev 2.3-1.0.1 provides the following new features:
- Secure Host
- Virtual Guest Tagging (VGT+)
- User-Mode Memory Registration (UMR)
- On-Demand-Paging (ODP)
- Reset Flow for ConnectX®-3 (+SR-IOV)
- Checksum offload for packets without L4 header
- Flow Steering: A0 simplified steering
- Memory re-registration
- Cable EEPROM reporting
- 128 Byte Completion Queue Entry (CQE)
- Explicit Congestion Notification (ECN)
- Disable/Enable ethernet RX VLAN tag striping offload via ethtool
- Windows Virtual Machine over Linux KVM Hypervisor (SR-IOV with InfiniBand only)

Fixes
===============
Installation 
- Added apt-get installation support.

Ethernet
- MLNX_OFED no longer changes the OS sysctl TCP parameters.
- Added Explicit Congestion Notification (ECN) support
- Added Flow Steering: A0 simplified steering support
- RoCE port bonding issue, error handling when a connection is dropped when the port is coming up and SW is not yet ready to receive

Resiliency
- Added Reset Flow for ConnectX®-3 (+SR-IOV) support
SR-IOV
- Added Virtual Guest Tagging (VGT+), an advanced mode of Virtual Guest Tagging (VGT), in which a VF is allowed to tag its own packets as in VGT, but is still subject to an administrative VLAN trunk policy.

Ethtool
- Added Cable EEPROM reporting support
- Disable/Enable ethernet RX VLAN tag striping offload via ethtool
- 128 Byte Completion Queue Entry (CQE)

Non-Linux Virtual Machines
- Added Windows Virtual Machine over Linux KVM Hypervisor (SR-IOV with InfiniBand only) support


Important Notes
===============
- Wake on LAN is only supported with Mellanox ConnectX-3 Dual Port 10 GbE KR Blade Mezzanine Ethernet Card

*********************************************************************
Currently known issues and limitations with Mellanox Ethernet drivers and devices
*********************************************************************
1. No current support for Brocade M8428-k IO Module for Dell PowerEdge M1000e chassis

2. No current support for Dell PowerConnect M8024-k IO Module for Dell PowerEdge M1000e chassis

3. DCBX support does not include support for Willing Mode.

Mellanox Known Issues with Mellanox MLNX_OFED Rev 2.3-1.0.1

Ethernet 
- When creating more than 125 VLANs and SR-IOV mode is enabled, a kernel warning message will be printed indicating that the native VLAN is created but will not work with RoCE traffic. kernel warning: mlx4_core 0000:07:00.0: vhcr command ALLOC_RES (0xf00) slave:0 in_param 0x7e in_mod=0x107, op_mod=0x1 failed with error:0, status -28
- Kernel panic might occur during fio splice in kernels before 2.6.34-rc4. Work around: Use kernel v2.6.34-rc4 which provides the following solution: baff42a net: Fix oops from tcp_collapse() when using splice()
- In PPC systems when QoS is enabled a harmless Kernel DMA mapping error messages might appear in kernel log (iommu related issue). 
- Transmit timeout might occur on RH6.3 as a result of lost interrupt (OS issue). In this case, the following message will be shown in dmesg: do_IRQ: 0.203 No irq handler for vector (irq -1) 
- The default priority to TC mapping assigns all priorities to TC0. This configuration achieves fairness in transmission between priorities but may cause undesirable PFC behavior where pause request for priority “n” affects all other priorities.  Work-around: Run: mlnx_qos –i <dev> -p 0,1,2,3,4,5,6,7 -s ets,ets,ets,ets,ets,ets,ets,ets -t 12,13,12,13,12,13,12,13 This needs to be applied every time after loading the mlx4_en driver.
- Mixing ETS and strict QoS policies for TCs in 40GbE ports may cause inaccurate results in bandwidth division among TCs.
- Affinity hints are not supported in Xen Hypervisor (an irqblancer issue). This causes a non-optimal IRQ affinity. Work-around: To overcome this issues, run: set_irq_affinity.sh eth<x>

General 
- On ConnectX-2/ConnectX-3 Ethernet adapter cards, there is a mismatch between the GUID value returned by firmware management tools and that returned by fabric/driver utilities that read the GUID via device firmware (e.g., using ibstat). Mlxburn/flint return 0xffff as GUID while the utilities return a value derived from the MAC address. For all driver/firmware/software purposes, the latter value should be used.  Work-around: Please use the GUID value returned by the fabric/driver utilities (not 0xfffff). 

VGT+
- Before adding a VLAN on the VM, the parent interface should be brought up. Otherwise VLAN creation will fail and the following message will be presented:
"Fail to register network rule."
- Bringing down and up the parent interface with VLANs configured over it, may result in traffic over VLANs being lost.
- On some of the OSes the callback ndo_vlan_rx_add/kill_vid returns void, therefore ethX.Y is created. However, if VLAN Y is not listed in the set of the allowed VLANs, no traffic will pass.
- When untagged traffic is not allowed, the below message will appear on Dom0 after driver restart on DomU:
mlx4_core 0000:16:00.0: vhcr command ALLOC_RES (0xf00) slave:1 in_param 0x0 in_mod=0x207, op_mod=0x1 failed with error:0, status -1

ABI Compatibility
- MLNX_OFED Rev 2.3-1.0.1 is not ABI compatible with previous MLNX_OFED/OFED versions. Work-around:Recompile the application over the new MLNX_OFED version 

ConnectX®-3 Adapter Cards Family
- Using RDMA READ with a higher value than 30 SGEs in the WR might lead to "local length error".  Work-around: Do not set the value of SGEs higher than 30 when RDMA READ is used.

Verbs
- Using libnl1_1_3~26 or earlier, requires ibv_create_ah protection by a lock for multi-threaded applications.

Resiliency
- Reset Flow can run on XenServer 6 only after the active user space applications running verbs are terminated.
- SR-IOV non persistent configuration (such as VGT, VST, Host assigned GUIDs, and QP0-enabled VFs) may be lost upon Reset Flow. Work-around: Reset Admin configuration post Reset Flow 
- Upon Reset Flow or after running restart driver, Ethernet VLANs are lost. Work-around: Reset the VLANs using the ifup command.
- Restarting the driver or running connectx_port_config when Reset Flow is running might result in a kernel panic
- Networking configuration (e.g. VLANs, IPv6) should be statically defined in order to have them set after Reset Flow as of after restart driver.

Driver Start
- "Out of memory" issues may rise during drivers load depending on the values of the driver module parameters set (e.g. log_num_cq).
- When reloading/starting the driver using the /etc/init.d/openibd the following messages are displayed if there is a third party RPM or driver installed: "Module mlx4_core does not belong to MLNX_OFED" or "Module mlx4_core belong to <rpm name> which is not a part of MLNX_OFED"  work-around: Remove the third party RPM/non MLNX_OFED drivers directory, run:
"depmod" and then rerun "/etc/init.d/openibd restart"
- Occasionally, when trying to repetitively reload the nes hardware driver on SLES11 SP2, a soft lockups occurs that required reboot.
- “openibd start” unloads kernel modules that were loaded from initrd/initramfs upon boot. This affects only kernel modules which come with MLNX_OFED and are included in initrd/initramfs.

Performance Tools
- perftest package in MLNX_OFED v2.2-1.0.1 and onwards does not work with older versions of the driver.
- On machines with irqbalancer daemon turned off, the default InfiniBand interrupts will be routed to a single core which may cause overload and software/hardware lockups. Work-around: To avoid this issue copy the following script to /etc/infiniband/post-start-hook.sh and execute it as root:
#!/usr/bin/perl
use strict;
if ( $< != 0 ) {
print "This script must be run as root\n";
exit (0);
}
open(F, "/proc/interrupts") or die "$!";
my $n = `cat /proc/cpuinfo | grep processor | wc -l`;
chomp($n);
print "Spreading over $n cpus\n";
while(<F>) {
#print $_;
my ($irq,$chan,$dev);
if (/(\d+):.*mlx5_comp(\d+)/) {
($irq,$chan,$dev) = ($1,$2,"mlx5");
} elsif (/(\d+):.*(mlx4-ib-\d)-(\d+)/) {
($irq,$chan,$dev) = ($1,$3,$2);
} else {
next;
}
my $place = ($chan % $n);
my $mask = 1 << $place;
printf ("$dev irq=%d chan=%d bit=%d mask=%0x\n", $irq, $chan, $place, $mask);
my $cmd = sprintf("echo %0x > /proc/irq/$irq/smp_affinity", $mask);
print "\t$cmd\n";
system($cmd);
}

SR-IOV
- When using legacy VMs with MLNX_OFED 2.x hypervisor, you may need to set the 'enable_64b_cqe_eqe' parameter to zero on the hypervisor. It should be set in the same way that other module parameters are set for mlx4_core at module load time. For example, add “options mlx4_core enable_64b_cqe_eqe=0” as a line in the file /etc/modprobe.d/mlx4_core.conf.
- When working with a bonding device to enslave the Ethernet devices in active-backup mode and failover MAC policy in a Virtual Machine (VM), establishment of RoCE connections may fail.  Work-around: Unload the module mlx4_ib and reload it in the VM.
- Attaching or detaching a Virtual Function on SLES11 SP3 to a guest Virtual Machine while the mlx4_core driver is loaded in the Virtual Machine may cause a kernel panic in the hypervisor. work-around: Unload the mlx4_core module in the hypervisor before attaching or detaching a function to or from the guest.
- When detaching a VF without shutting down the driver from a VM and reattaching it to another VM with the same IP address for the Mellanox NIC, RoCE connections will fail  Work-around: Shut down the driver in the VM before detaching the VF.
- Enabling SR-IOV requires appending the “intel_iommu=on” option to the relevant OS in file /boot/grub/grub.conf. Without that SR-IOV cannot be loaded. 
- On various combinations of Hypervisor/OSes and Guest/OSes, an issue might occur when attaching/detaching VFs to a guest while that guest is up and running.  Work-around: Attach/detach VFs to/from a VM only while that VM is down.
- When working with SR-IOV in Xen-4.2 virtualization platform, only the built-in xen_pciback driver should be loaded. The xen_pciback module in dom0 should not be loaded, as loading them simultaneously may cause interrupts loss and cause the driver to enter the reset flow.
- The known PCI BDFs for all VFs in kernel command line should be specified by adding xen-pciback.hide For further information, please refer to http://wiki.xen.org/wiki/Xen_PCI_Passthrough
- SR-IOV UD QPs are forced by the Hypervisor to use the base GID (i.e., the GID that the VF sees in its GID entry at its paravirtualized index 0). This is needed for security, since UD QPs use Address Vectors, and any GID index may be placed in such a vector, including indices not belonging to that VF.
- Attempting to attach a PF to a VM when SR-IOV is already enabled on that PF may result in a kernel panic.

Port Type
- A working IP connectivity between the RoCE devices is required when creating an address handle or modifying a QP with an address vector.
- IPv4 multicast over RoCE requires the MGID format to be as follow ::ffff:<Multicast IPv4 Address>
- DIF: When running IO over FS over DM during unstable ports, block layer BIOs merges may cause false DIF error.

Flow Steering
- IPv4 rule with source IP cannot be created in SLES 11.x OSes.
- RFS does not support UDP

Quality of Service
- QoS is not supported in XenServer, Debian 6.0 and 6.2 with uek kernel
- When QoS features are not supported by the kernel, mlnx_qos tool may crash.

Installation
- When upgrading from an earlier Mellanox OFED version, the installation script does not stop the earlier version prior to uninstalling it. Work-around: Stop the old OFED stack (/etc/init.d/openibd stop) before upgrading to this new version.

UEFI Secure Boot
- On RHEL7 and SLES12, the following error is displayed in dmesg if the Mellanox's x.509 Public Key is not added to the system:
[4671958.383506] Request for unknown module key 'Mellanox Technologies signing key: 61feb074fc7292f958419386ffdd9d5ca999e403' err -11
This error can be safely ignored as long as Secure Boot is disabled on the system.  Work-around: For further information, please refer to the User Manual section "Enrolling Mellanox's x.509 Public Key On your Systems".
- The initiator does not respect interface parameter while logging in.  Work-around: Configure each interface on a different subnet

Fork Support
- Fork support from kernel 2.6.12 and above is available provided that applications do not use threads. fork() is supported as long as the parent process does not run before the child exits or calls exec().
The former can be achieved by calling wait(childpid), and the latter can be achieved by application specific means. The Posix system() call is supported.

Ethernet Performance Counters
- Since each VF tries to allocate 2 more QP counter for its RoCE traffic statistics, in a system with less than 61 VFs, if there is free resources it receives new counter otherwise receives the default counter which is shared with Ethernet. In this case RoCE statistics is not available.
- In ConnectX®-3, when we enable function-based loopback prevention for Ethernet port by default (i.e., based on the QP counter index), the dropped self-loopback packets increase the IfRxErrorFrames/Octets counters.

Uplinks 
- On rare occasions, ConnectX®-3 Pro adapter card may fail to link up when performing parallel detect to 40GbE.  Work-around: Restart the driver

Resources Limitation
- The device capabilities reported may not be reached as it depends on the system on which the device is installed and whether the resource is allocated in the kernel or the userspace.
- mlx4_core can allocate up to 64 MSI-X vectors, an MSI-X vector per CPU.
- Setting more IP addresses than the available GID entries in the table results in failure and the "update_gid_table error message is displayed:
GID table of port 1 is full. Can't add <address>" message.
- Registering a large amount of Memory Regions (MR) may fail because of DMA mapping issues on RHEL 7.0.
- Occasionally, a user process might experience some memory shortage and not function properly due to Linux kernel occupation of the system’s free memory for its internal cache.  Work-around: To free memory to allow it to be allocated in a user process, run the drop_caches procedure below.
Performing the following steps will cause the kernel to flush and free pages, dentries and inodes caches from memory, causing that memory to become free.
Note: As this is a non-destructive operation and dirty objects are not freeable, run `sync' first.
•To free the pagecache:
echo 1 > /proc/sys/vm/drop_caches
•To free dentries and inodes:
echo 2 > /proc/sys/vm/drop_caches
•To free pagecache, dentries and inodes:
echo 3 > /proc/sys/vm/drop_caches

RoCE
- Not configuring the Ethernet devices or independent VMs with a unique IP address in the physical port, may result in RoCE GID table corruption.  Work-around: Restart the driver
- If RDMA_CM is not used for connection management, then the source and destination GIDs used to modify a QP or create AH should be of the same type - IPv4 or IPv6.
- MLNX_OFED v2.1-1.0.0 and onwards is not interoperable with older versions of MLNX_OFED.
- Since the number of GIDs per port is limited to 128, there cannot be more than the allowed IP addresses configured to Ethernet devices that are associated with the port. Allowed number is:
•“127” for a single function machine
•“15” for a hypervisor in a multifunction machine
•“(127-15)/n” for a guest in a multifunction machine (where n is the number of virtual functions)
- A working IP connectivity between the RoCE devices is required when creating an address handle or modifying a QP with an address vector.
- IPv4 multicast over RoCE requires the MGID format to be as follow ::ffff:<Multicast IPv4 Address>
- Using GID index 0 (the default GID) is possible only if the matching IPv6 link local address is configured on the net device of the port. This behavior is possible even though the default GID is configured regardless the presence of the IPv6 address.
- Using IPv6 link local address (GID0) when VLANs are configured is not supported.

Storage
- Older versions of rescan_scsi_bus.sh may not recognize some newly created LUNs. Work-around: If encountering such issues, it is recommended to use the '-c' flag.
- RHEL7.0: The rescan-scsi-bus.sh script does not rediscover provisioned LUNs both on iSER and SRP. Work-around: Use older version of the script from RHEL6.4
•iSER: Use "iscsiadm -m session --rescan" 

SRP
- In a high stress IO with unstable links, SRP Initiator may generate a call trace that can be safely ignored.
- MLNX_OFED SRP installation breaks the ibmvstgt and ibmvscsi symbol resolution in RHEL7.0

SRP Interop
- The driver is tested with Storage target vendors recommendations for multipath.conf extensions (ZFS, DDN, TMS, Nimbus, NetApp).

DDN Storage Fusion 10000 Target
- DDN does not accept non-default P_Key connection establishment.

Oracle Sun ZFS Storage 7420
- Occasionally the first command to a LUN may not be serviced, aborted, and cause a successful re-connection to the target
- Ungraceful power cycle of an initiator connected with Targets DDN, Nimbus, NetApp may result in temporary "stale connection" messages when initiator reconnects.

iSER
- On SLES OSs, the ib_iser module does not load on boot  Work-around: Add a dummy interface using iscsiadm:
•# iscsiadm -m iface -I ib_iser -o new
•# iscsiadm -m iface -I ib_iser -o update -n iface.transport_name -v ib_iser
- DIF: When running IO over FS over DM during unstable ports, block layer BIOs merges may cause false DIF error.
- The initiator does not respect interface parameter while logging in.  Work-around: Configure each interface on a different subnet.
- iSCSID v2.0.873 can enter an endless loop on bind error
- DIX: Under heavy IO stress with large block size, the HCA card might generate error Completions in the log and traffic might be affected. Work-around: Disable block merges.
- iSCSID may hang if target crashes during logout sequence (reproducible with TCP)

ZFS Appliance
- Connection establishment occurs twice which may cause iSER to log a stack trace.

Bug Fixes
Ethernet
- Fixed kernel panic on Debian-6.0.7 which occurred when the number of TX channels was set above the default value. 
- Fixed a crash incidence which occurred when enabling Ethernet Time-stamping and running VLAN traffic.

mlx4_en
- Fixed wrong calculation of packet true-size reporting in LRO flow.




######################################################################
#	Mellanox Technologies Inc
#	Mellanox OFED for Linux Release Notes
#	August 18th, 2014
#
#	RELEASE Version 2.1-1.0.0.1
#####################################################################

Hardware Compatibility
======================
This release of software supports the following hardware:
- Mellanox ConnectX-3 Dual Port 10 GbE KR Blade Mezzanine Ethernet Card
- Mellanox ConnectX-3 Dual Port 40 GbE QSFP+ Ethernet Adapter
- Mellanox ConnectX-3 Dual Port 10 GbE DA/SFP+ Ethernet Adapter


Software Compatibility
======================
This release of software supports the following operating systems:
- Red Hat Enterprise Linux 6.5,x86_64
- Red Hat Enterprise Linux 6.4,x86_64
- Novell - SUSE Linux ES 11 SP2, x86_64
- Novell - SUSE Linux ES 11 SP3, x86_64

Release Contents
==================
MLNX_OFED_LINUX-2.1-1.0.0.1 installer

* Third-party trademarks and copyrights are the property of their respective owners.

Feature Summary 
===============
- 2nd Dell release of Mellanox OFED for Linux OSes, supporting Mellanox ConnectX-3 Adapters on Dell PowerEdge Servers.

MLNX_OFED Rev 2.1-1.0.0.1 provides the following new features:
- Ethernet Flow Steering (beta level)
- Ethernet Performance Counters
- Ethernet Performance Enhancements (NUMA related and others) for 10G and 40G
- Raw Ethernet QPs:
  - Checksum TX/RX
  - Flow Steering
- Errata Kernel upgrade support
- YUM update support
- Storage iSER (at Beta level) and SRP
- 64bit wide counters (port xmit/recv data/packets unicast/mcast)
- VERSION query API: library and headers

Fixes
===============
mlnx_en
- Added reporting auto-negotiation support
- Added Transmit Packet Steering (XPS) support
- Added Receive Flow Steering (RFS) support in UDP
- Added Low Latency Socket (LLS) support
- Added check for dma_mapping errors

Important Notes
===============
- Wake on LAN is only supported with Mellanox ConnectX-3 Dual Port 10 GbE KR Blade Mezzanine Ethernet Card

*********************************************************************
Currently known issues and limitations with Mellanox Ethernet drivers and devices
*********************************************************************
1. No current support for Brocade M8428-k IO Module for Dell PowerEdge M1000e chassis

2. No current support for Dell PowerConnect M8024-k IO Module for Dell PowerEdge M1000e chassis

3. DCBX support does not include support for Willing Mode.

Mellanox Known Issues with Mellanox MLNX_OFED Rev 2.1-1.0.0.1

Ethernet 
- PV VLAN Guest transparent Tagging(VGT) is only supported in openvswitch and not in standard Linux vBridges and libvirt. For more information please see : http://libvirt.org/formatnetwork.html (Setting VLAN tag section)
- Changing the ring size on 32-bit system may result in failure due to lack of memory. Therefore, mlx4_en will not be able to vmap enough memory and the below message will be printed in dmesg: vmap allocation for size 528384 failed: use vmalloc=<size> to increase size
In this case user can enlarge the vmalloc memory by adding vmalloc=<size> to grub.conf Default vmalloc setting is 128M. 
It is recommended to add each time 64M of memory until desired ring size can be allocated. Please note, that in case vmalloc size is too big, the OS will fail to boot, so please use caution when adding additional memory. For more info refer to: http://www.mythtv.org/wiki/Common_Problem:_vmalloc_too_small
- Kernel panic might occur during traffic over IPv6 on kernels between 3.12-rc7 and 3.13-rc1 (kernel issue)
- Kernel panic might occur during fio splice in kernels before 2.6.34-rc4. Suggested work around: Use kernel v2.6.34-rc4 which provides the following solution: baff42a net: Fix oops from tcp_collapse() when using splice()
- On kernels that do not support multiqueues, the number of TX channels represents the number of TX rings. The maximal number of TX channels is 16.
- Transmit timeout might occur on RH6.3 as a result of lost interrupt (OS issue). In this case, the following message will be shown in dmesg: do_IRQ: 0.203 No irq handler for vector (irq -1)

XRC
- Legacy API is deprecated, thus when recompiling applications over MLNX_OFED v2.0-3.x.x, warn-ings such as the below are displayed. rdma.c:1699: warning: 'ibv_open_xrc_domain' is deprecated (declared at /usr/include/infiniband/ ofa_verbs.h:72) rdma.c:1706: warning: 'ibv_create_xrc_srq' is deprecated (declared at /usr/include/infiniband/ ofa_verbs.h:89) These warnings can be safely ignored. 
- XRC is not functional in heterogeneous clusters con-taining non Mellanox HCAs. 
- XRC options do not work when using qperf tool. Suggested work-around Use perftest instead 
- XRC over ROCE in SR-IOV mode is not functional
- Out-of memory issue might occur due to overload of XRC receive QP with non-zero receive queue size created. XRC QPs do not have receive queues. 

ABI Compatibility 
- MLNX_OFED Rev 2.1-1.0.0.1 is not ABI compatible with previous MLNX_OFED/OFED versions. Suggested work-around: Recompile the application over the new MLNX_OFED version 

System Time 
- Loading the driver using the openibd script when no InfiniBand vendor module is selected (for example mlx4_ib), may cause the execution of the /sbin/start_udev’ script. In RedHat 6.x and OEL6.x this may change the local system time. 

Verbs 
- Verbs for the following features are subject to change: Shared memory region, Contiguous pages, Flow steering

Driver Start 
- "Out of memory" issues may rise during drivers load depending on the values of the driver module parameters set (for example, log_num_cq). 
- When reloading/starting the driver using the /etc/ init.d/openibd the following messages are dis-played if there is a third party RPM or driver installed: "Module mlx4_core does not belong to MLNX_OFED" or "Module mlx4_core belong to <rpm name> which is not a part of MLNX_OFED" . Suggested work-around: Remove the third party RPM/non MLNX_OFED drivers directory, run: "depmod" and then rerun "/etc/init.d/ openibd restart" 
- Occasionally, when trying to repetitively reload the nes hardware driver on SLES11 SP2, a soft lockups occurs that required reboot. 

SR-IOV 
- When using legacy VMs with OFED 2.0-2.0.5 hypervisor, the 'enable_64b_cqe_eqe' parameter must be set to zero on the hypervisor. It should be set in the same way that other module parameters are set for mlx4_core at module load time. For example, add “options mlx4_core enable_64b_cqe_eqe=0” as a line in the file / etc/modprobe.d/mlx4_core.conf.
- Enabling SR-IOV requires appending the “intel_iommu=on” option to the relevant OS in file /boot/grub/grub.conf/. Without that SR-IOV cannot be loaded.


Flow Steering 
- Flow Steering is disabled by default. To enable it, set the parameter below as follow: log_num_mgm_entry_ size should set to -1

Driver Uninstall 
- A Kernel panic occurs if you uninstall the driver without deleting the SR-IOV module params (mlx4_core’s num_vfs) in the file /etc/mod-probe.d/mlx4_core.conf. On the next boot, you will get the panic, and machine will boot up. Work-around: Remove the module after uninstalling and prior to restarting the driver.

Installation
- When upgrading from an earlier Mellanox OFED version, the installation script does not stop the earlier version prior to uninstalling it. Suggested work around: Stop the old OFED stack (/etc/init.d/openibd stop) before upgrading to this new version.

Fork Support
- Fork support from kernel 2.6.12 and above is available provided that applications do not use threads. fork() is supported as long as the parent process does not run before the child exits or calls exec(). The former can be achieved by calling wait(childpid), and the latter can be achieved by application specific means. The Posix system() call is supported.

Resources Limitation
- Setting more IP addresses than the available GID entries in the table results in failure and the "update_gid_table error message is displayed: GID table of port 1 is full. Can't add <address>" message.

Ethernet Performance Counters
- In a system with more than 61 VFs, the 62nd VF and onwards is assigned with the SINKQP counter, and as a result will have no statistics, and loopback prevention functionality for SINK counter.
- Since each VF tries to allocate 2 more QP counter for its RoCE traffic statistics, in a system with less than 61 VFs, if there is free resources it receives new counter otherwise receives the default counter which is shared with Ethernet. In this case RoCE statistics is not available.
- In ConnectX(R)-3, when we enable function-based loopback prevention for Ethernet port by default (i.e., based on the QP counter index), the dropped self-loopback packets increase the IfRxError-Frames/Octets counters.

RoCE
- Not configuring the Ethernet devices or independent VMs with a unique IP address in the physical port, may result in RoCE GID table corruption. Suggested work around: Restart the driver
- If RDMA_CM is not used for connection management, then the source and destination GIDs used to modify a QP or create AH should be of the same type - IPv4 or IPv6.
- Since the number of GIDs per port is limited to 128, there cannot be more than the allowed IP addresses configured to Ethernet devices that are associated with the port. Allowed number is:
  - “127” for a single function machine  
  - “15” for a hypervisor in a multifunction machine
  - “n” for a guest in a multifunction machine (where n is the number of virtual functions)
- A working IP connectivity between the RoCE devices is required when creating an address handle or modifying a QP with an address vector.
- MLNX_OFED v2.1-1.0.0.1 is not interoperable with older versions of MLNX_OFED.
- Unloading mlx4_en while a rdma_cm session is established can cause a kernel panic.

Storage
- Older versions of rescan_scsi_bus.sh may not recognize some newly created LUNs. Suggest work around: If encountering such issues, it is recommended to use the '-c' flag.

iSER
- On SLES11, the ib_iser module does not load on boot. Suggested work around: Add a dummy interface using iscsiadm:
• # iscsiadm -m iface -I ib_iser -o new
• # iscsiadm -m iface -I ib_iser -o update -n iface.transport_name -v ib_iser
- Trying to disconnect a session while the session is undergoing a reconnect flow may result in disconnection hang.  Suggested work around: Restart iscsid. Note: Please be aware that doing so might cause rmmod process to hang as the
ib_iser module will not be unloaded.


#####################################################################
#                                                                    
#	Mellanox Technologies Inc                                        
#	Mellanox OFED for Linux Release Notes
#	September 20th, 2013
#
#	RELEASE Version 2.0-2.6.8
#####################################################################		
Hardware Compatibility
======================
This release of software supports the following hardware:
- Mellanox ConnectX-3 Dual Port 10 GbE KR Blade Mezzanine Ethernet Card
- Mellanox ConnectX-3 Dual Port 40 GbE QSFP+ Ethernet Adapter
- Mellanox ConnectX-3 Dual Port 10 GbE DA/SFP+ Ethernet Adapter

Software Compatibility
======================

This release of software supports the following operating systems:
- Red Hat Enterprise Linux 6.4,x86_64
- Novell - SUSE Linux ES 11 SP3, x86_64
- Novell - SUSE Linux ES 11 SP2, x86_64

Release Contents
==================
MLNX_OFED_LINUX-2.0-2.6.8 installer

* Third-party trademarks and copyrights are the property of their respective owners.

Feature Summary 
===============
- Initial Dell release of Mellanox OFED for Linux OSes, supporting Mellanox ConnectX-3 Adapters on Dell PowerEdge Servers.

- SR-IOV for Ethernet
- RoCE over SR-IOV    
- Port based QoS for Ethernet
- Flow Steering

Fixes
===============
- Initial Release

Important Notes
===============
- Wake on LAN is only supported with Mellanox ConnectX-3 Dual Port 10 GbE KR Blade Mezzanine Ethernet Card

*********************************************************************
Currently known issues and limitations with Mellanox Ethernet drivers and devices
*********************************************************************

1. Disabling network device corresponding to port 1 of the Mellanox adapter may not cause the physical link to drop.
No network traffic will actually be transmitted on that link.

2. No current support for Brocade M8428-k IO Module for Dell PowerEdge M1000e chassis

3. No current support for Dell PowerConnect M8024-k IO Module for Dell PowerEdge M1000e chassis

Mellanox Known Issues with Mellanox MLNX_OFED Rev 2.0-2.6.8

XRC
- Legacy API is deprecated, thus when recompiling applications over MLNX_OFED v2.0-3.x.x, warn-ings such as the below are displayed. rdma.c:1699: warning: 'ibv_open_xrc_domain' is deprecated (declared at /usr/include/infiniband/ ofa_verbs.h:72) rdma.c:1706: warning: 'ibv_create_xrc_srq' is deprecated (declared at /usr/include/infiniband/ ofa_verbs.h:89) These warnings can be safely ignored. 
- XRC is not functional in heterogeneous clusters con-taining non Mellanox HCAs. 
- XRC options do not work when using qperf tool. Suggested work-around Use perftest instead 
- XRC over ROCE in SR-IOV mode is not functional
- Out-of memory issue might occur due to overload of XRC receive QP with non zero receive queue size created. XRC QPs do not have receive queues. 

ABI Compatibility 
- MLNX_OFED Rev 2.0-2.6.8 is not ABI compatible with previous MLNX_OFED/OFED versions. Suggested work-around: Recompile the application over the new MLNX_OFED version 

System Time 
- Loading the driver using the openibd script when no InfiniBand vendor module is selected (for example mlx4_ib), may cause the execution of the /sbin/start_udev’ script. In RedHat 6.x and OEL6.x this may change the local system time. 

Verbs 
- Verbs for the following features are subject to change: Shared memory region, Contiguous pages, Flow steering

Driver Start 
- "Out of memory" issues may rise during drivers load depending on the values of the driver module parameters set (for example, log_num_cq). 
- When reloading/starting the driver using the /etc/ init.d/openibd the following messages are dis-played if there is a third party RPM or driver installed: "Module mlx4_core does not belong to MLNX_OFED" or "Module mlx4_core belong to <rpm name> which is not a part of MLNX_OFED" . Suggested work-around: Remove the third party RPM/non MLNX_OFED drivers directory, run: "depmod" and then rerun "/etc/init.d/ openibd restart" 
- Occasionally, when trying to repetitively reload the nes hardware driver on SLES11 SP2, a soft lockups occurs that required reboot. 

SR-IOV 
- When using legacy VMs with OFED 2.0-2.0.5 hypervisor, the 'enable_64b_cqe_eqe' parameter must be set to zero on the hypervisor. It should be set in the same way that other module parameters are set for mlx4_core at module load time. For example, add “options mlx4_core enable_64b_cqe_eqe=0” as a line in the file / etc/modprobe.d/mlx4_core.conf.
- Enabling SR-IOV requires appending the “intel_iommu=on” option to the relevant OS in file /boot/grub/grub.conf/. Without that SR-IOV cannot be loaded.
- rdma_cm does not support UD QPs 
- SR-IOV can be enabled only when using the firm-ware version embedded in the MLNX_OFED v2.0-2.6.8 driver.

Flow Steering 
- Flow Steering is disabled by default. To enable it, set the parameter below as follow: log_num_mgm_entry_ size should set to -1

Driver Uninstall 
- A Kernel panic occurs if you uninstall the driver without deleting the SR-IOV module params (mlx4_core’s num_vfs) in the file /etc/mod-probe.d/mlx4_core.conf. On the next boot, you will get the panic, and machine will boot up. Work-around: Remove the midule after uninstalling and prior to restarting the driver.

Fork Support 
- Fork support from kernel 2.6.12 and above is available provided that applications do not use threads. fork() is supported as long as the parent process does not run before the child exits or calls exec(). The former can be achieved by calling wait(childpid), and the latter can be achieved by application specific means. The Posix system() call is supported. 

Resources Limitation 
- The device capabilities reported may not be reached as it depends on the system on which the device is installed and whether the resource is allocated in the kernel or the userspace.
- Occasionally, a user process might experience some memory shortage and not function properly due to Linux kernel occupation of the system’s free memory for its internal cache. Work-around: To free memory to allow it to be allocated in a user process, run the drop_caches pro-cedure below. Performing the follow-ing steps will cause the kernel to flush and free pages, dentries and inodes caches from memory, causing that memory to become free. Note: As this is a non-destructive operation and dirty objects are not freeable, run `sync' first. 
- To free the pagecache: echo 1 > /proc/ sys/vm/ drop_caches 
- To free dentries and inodes: echo 2 > /proc/ sys/vm/ drop_caches 
- To free pagecache, dentries and inodes: echo 3 > /proc/ sys/vm/ drop_caches