Solarflare-specific Features

Solarflare Onload

Solarflare Onload is a sockets acceleration technology for the transparent acceleration of sockets-based applications.

On Linux, the Handler uses Solarflare Onload Extensions API to support additional features of Solarflare network cards. If one uses such network cards, in your system, you can benefit from advanced features of such network cards. B3 BOE Handler automatically detects if the corresponding Onload library installed in your system and, if so, then loads it in run-time and provides an ability to use additional features. In this case, there should be the record, in the Handler log, that the corresponding version of the Onload extension library is used.

Note

If one wants to test Onload features on the loopback network interface, EF_TCP_CLIENT_LOOPBACK and EF_TCP_SERVER_LOOPBACK parameters should be used, e.g.: EF_TCP_CLIENT_LOOPBACK=2 EF_TCP_SERVER_LOOPBACK=2 onload ...

Please see the "Onload User Guide".

Currently, the Handler supports the following Onload features:

• Onload message warmup

When one uses the OnixS::B3::BOE::Session::warmUp method, the Handler automatically enables this feature. As a result, one can get the maximal warm-up effect, including the complete warm-up of the TCP stack's sending path.

The onload_stackdump utility could be used to monitor the usage of this feature.

For example:

onload_stackdump lots | grep warm

---

Solarflare TCPDirect

Solarflare TCPDirect is an ultra-low latency network stack.

Creating OnixS TcpDirectStack

Before TCPDirect sessions can be created, the application must first create a TCPDirect stack (OnixS::B3::BOE::TcpDirectStack):

TcpDirectAttr attr;
// The default interface name could be provided by environment variable
//     ZF_ATTR="interface=<interface-name>", see TCPDirect specs,
//
// or could be set manually:
//     attr.set("interface", "interface-name");

TcpDirectStack tcpDirectStack(attr);

Attributes (OnixS::B3::BOE::TcpDirectAttr) control the configuration of the stack and its behavior, please see the Solarflare TCPDirect User Guide for a complete list of available attributes and their description.

TCPDirect sessions should be created in the same thread where TcpDirectStack has been created. The reference to the stack is provided in Session's constructor, for example:

Session session(tcpDirectStack, settings, &listener, storageType);

By default, each TCPDirect stack can handle up to 64 TCP endpoints, so those number of sessions could be created. Maximum number of TCP endpoints can be set by max_tcp_endpoints attribute. Please refer to Solarflare TCPDirect documentation. See also max_tcp_listen_endpoints, max_tcp_syn_backlog, max_udp_rx_endpoints, max_udp_tx_endpoints attributes. These attributes could be set to a minimum value to save hardware and software resources.

Dispatching network events, asynchronous connect and disconnect

External code on top of TCPDirect can be implemented as an event loop.

The need to scroll through the loop means the inability to use blocking calls because blocking calls will stop the event's dispatching. Therefore, user code must call asynchronous Session's methods to connect and disconnect, for example:

// some application-managed variable that could be changed by some events
bool finished = false;

Threading::SharedFuture<void> connected = session.connectAsync(host, port);

// The TCPDirect technology is NOT thread-safe.
// Therefore, the event dispatching and message sending should be performed from the same thread.
while(!connected.is_ready())
    tcpDirectStack.dispatchEvents();

if(connected.has_exception())
    connected.get();    // will throw an exception

while(!finished)
    tcpDirectStack.dispatchEvents();

TcpDirectStack shutdown

In general, TCPDirect stack requires to finish all outstanding work and to handle all outstanding events. Before destroying the TcpDirectStack instance, the following code should be executed:

while(!tcpDirectStack.isQuiescent())
    tcpDirectStack.dispatchEvents();

Method TcpDirectStack::isQuiescent() (OnixS::B3::BOE::TcpDirectStack::isQuiescent) returns a boolean value indicating whether a stack is quiescent.

This can be used to ensure that all connections have been closed gracefully before destroying a stack (or exiting the application). Destroying a stack while it is not quiescent is permitted by the API, but when doing so there is no guarantee that sent data has been acknowledged by the peer or even transmitted, and there is the possibility that peers' connections will be reset.

TCPDirect threading model

Warning

Access to the stack instance at each moment is possible from only one thread. The TCPDirect middle-ware does not use inter-threaded synchronization, does not serialize calls, and does not check the correctness of access to its API by the user. Therefore, the event dispatching and message sending should be performed from the same thread.

TCPDirect emulator for Windows

The TCPDirect API does not support loopback connections as opposed to regular sockets.

To facilitate the development and debugging of applications based on TCPDirect OnixS B3 BOE Handler contains the TCPDirect emulator for Windows on top of regular sockets with call correctness verification.

Unlike the original technology, the TCPDirect emulator allows you to create a loopback connection, which also helps the application debugging process.

TCPDirect emulator does not require additional configuration, it activated automatically under Windows when TcpDirectStack object is created and used.

Native TCPDirect-connections should belong to the corresponding Solarflare hardware interface. In Windows emulation mode it could be any existing network interface that allows binding the sockets to it. In most common debugging scenarios is recommended to use the same interface (and its IP address) both for an exchange emulator (OnixS::B3::BOE::Testing::Gateway) and TCPDirect-emulated client connection.

Warning

Under Windows TCPDirect emulator, destroying a stack while it is not quiescent may cause a memory leak.

See also

TCPDirect Getting Started Sample

TCPDirect Benchmark Sample