NAME

SYNOPSIS

Alternatively, to load the driver as a module at boot time, place the following line in loader.conf(5):

if_ena_load="YES"

DESCRIPTION

The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue.

The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set.

Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices.

The ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement.

When RSS is enabled, each Tx/Rx queue pair is bound to a corresponding CPU core and its NUMA domain. The order of those bindings is based on the RSS bucket mapping. For builds with RSS support disabled, the CPU and NUMA management is left to the kernel.

The ena driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling.

The ena driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs.

Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds.

Support for the netmap(4) framework is provided by the ena driver. Kernel must be built with the DEV_NETMAP option to be able to use this feature.

HARDWARE

• 1d0f:0ec2 - ENA PF
• 1d0f:1ec2 - ENA PF with LLQ support
• 1d0f:ec20 - ENA VF
• 1d0f:ec21 - ENA VF with LLQ support

LOADER TUNABLES

Boot-time tunables:

hw.ena.enable_9k_mbufs

Use 9k mbufs for the Rx descriptors. The default is 0. If the node value is set to 1, 9k mbufs will be used for the Rx buffers. If set to 0, page size mbufs will be used instead.

Using 9k buffers for Rx can improve Rx throughput, but in low memory conditions it might increase allocation time, as the system has to look for 3 contiguous pages. This can further lead to OS instability, together with ENA driver reset and NVMe timeouts. If network performance is critical and memory capacity is sufficient, the 9k mbufs can be used.

hw.ena.force_large_llq_headers

Force the driver to use large LLQ headers (224 bytes). The default is 0. If the node value is set to 0, the regular size LLQ header will be used, which is 96B. In some cases, the packet header can be bigger than this (for example - IPv6 with multiple extensions). In such a situation, the large LLQ headers should be used by setting this node value to 1. This will take effect only if the device supports both LLQ and large LLQ headers. Otherwise, it will fallback to the no LLQ mode or regular header size.

Increasing LLQ header size reduces the size of the Tx queue by half, so it may affect the number of dropped Tx packets.

Run-time tunables:

hw.ena.log_level

Controls extra logging verbosity of the driver. The default is 2. The higher the logging level, the more logs will be printed out. 0 means all extra logs are disabled and only error logs will be printed out. Default value (2) reports errors, warnings and is verbose about driver operation.

The possible flags are:

• 0 - ENA_ERR - Enable driver error messages and ena_com error logs.
• 1 - ENA_WARN - Enable logs for non-critical errors.
• 2 - ENA_INFO - Make the driver more verbose about its actions.
• 3 - ENA_DBG - Enable debug logs.

NOTE: In order to enable logging on the Tx/Rx data path, driver must be compiled with ENA_LOG_IO_ENABLE compilation flag.

Example: To enable logs for errors and warnings, the following command should be used:

sysctl hw.ena.log_level=1
    

dev.ena.X.io_queues_nb

Number of the currently allocated and used IO queues. The default is max_num_io_queues. Controls the number of IO queue pairs (Tx/Rx). As this call has to reallocate the queues, it will reset the interface and restart all the queues - this means that everything, which was currently held in the queue, will be lost, leading to potential packet drops.

This call can fail if the system isn't able to provide the driver with enough resources. In that situation, the driver will try to revert the previous number of the IO queues. If this also fails, the device reset will be triggered.

Example: To use only 2 Tx and Rx queues for the device ena1, the following command should be used:

sysctl dev.ena.1.io_queues_nb=2
    

dev.ena.X.rx_queue_size

Size of the Rx queue. The default is 1024. Controls the number of IO descriptors for each Rx queue. The user may want to increase the Rx queue size if they observe a high number of Rx drops in the driver's statistics. For performance reasons, the Rx queue size must be a power of 2.

This call can fail if the system isn't able to provide the driver with enough resources. In that situation, the driver will try to revert to the previous number of the descriptors. If this also fails, the device reset will be triggered.

Example: To increase Rx ring size to 8K descriptors for the device ena0, the following command should be used:

sysctl dev.ena.0.rx_queue_size=8192
    

dev.ena.X.buf_ring_size

Size of the Tx buffer ring (drbr). The default is 4096. Input must be a power of 2. Controls the number of mbufs that can be held in the Tx buffer ring. The drbr is used as a multiple-producer, single-consumer lockless ring for buffering extra mbufs coming from the stack in case the Tx procedure is busy sending the packets, or the Tx ring is full. Increasing the size of the buffer ring may reduce the number of Tx packets being dropped in case of a big Tx burst, which cannot be handled by the IO queue immediately. Each Tx queue has its own drbr.

It is recommended to keep the drbr with at least the default value, but in case the system lacks the resources, it can be reduced. This call can fail if the system is not able to provide the driver with enough resources. In that situation, the driver will try to revert to the previous number of the drbr and trigger the device reset.

Example: To set drbr size for interface ena0 to 2048, the following command should be used:

sysctl dev.ena.0.buf_ring_size=2048
    

dev.ena.X.eni_metrics.sample_interval

Interval in seconds for updating ENI metrics. The default is 0. Determines how often (if ever) the ENI metrics should be updated. The ENI metrics are being updated asynchronously in a timer service in order to avoid admin queue overload by sysctl node reading. The value in this node controls the interval between issuing admin commands to the device, which will update the ENI metrics values.

If some application is periodically monitoring the eni_metrics, then the ENI metrics interval can be adjusted accordingly. Value 0 turns off the update completely. Value 1 is the minimum interval and is equal to 1 second. The maximum allowed update interval is 1 hour.

Example: To update ENI metrics for the device ena1 every 10 seconds, the following command should be used:

sysctl dev.ena.1.eni_metrics.sample_interval=10
    

dev.ena.X.rss.indir_table_size

RSS indirection table size. The default is 128. Returns the number of entries in the RSS indirection table.

Example: To read the RSS indirection table size, the following command should be used:

sysctl dev.ena.0.rss.indir_table_size
    

dev.ena.X.rss.indir_table

RSS indirection table mapping. The default is x:y key-pairs of indir_table_size length. Updates selected indices of the RSS indirection table.

The entry string consists of one or more x:y keypairs, where x stands for the table index and y for its new value. Table indices that don't need to be updated can be omitted from the string and will retain their existing values.

If an index is entered more than once, the last value is used.

Example: To update two selected indices in the RSS indirection table, e.g. setting index 0 to queue 5 and then index 5 to queue 0, the following command should be used:

sysctl dev.ena.0.rss.indir_table="0:5 5:0"
    

dev.ena.X.rss.key

RSS hash key. The default is 40 bytes long randomly generated hash key. Controls the RSS Toeplitz hash algorithm key value.

Only available when driver compiled without the kernel side RSS support.

Example: To change the RSS hash key value to

0x6d, 0x5a, 0x56, 0xda, 0x25, 0x5b, 0x0e, 0xc2,
0x41, 0x67, 0x25, 0x3d, 0x43, 0xa3, 0x8f, 0xb0,
0xd0, 0xca, 0x2b, 0xcb, 0xae, 0x7b, 0x30, 0xb4,
0x77, 0xcb, 0x2d, 0xa3, 0x80, 0x30, 0xf2, 0x0c,
0x6a, 0x42, 0xb7, 0x3b, 0xbe, 0xac, 0x01, 0xfa

the following command should be used:

sysctl dev.ena.0.rss.key=6d5a56da255b0ec24167253d43a38fb0d0ca2bcbae7b30b477cb2da38030f20c6a42b73bbeac01fa
    

DIAGNOSTICS

SUPPORT

SEE ALSO

HISTORY

AUTHORS