Low Latency Best Practices

Session Tuning-up

Selecting Right Storage

Using OnixS::FIX::SessionStorageType::MemoryBased instead of OnixS::FIX::SessionStorageType::FileBased boosts performance, since FIX messages are stored directly in memory. Alternatively, it's possible to use a custom pluggable storage (Using OnixS::FIX::SessionStorageType::Pluggable ) which does nothing on FIX message-related operations, as soon as no resend requests are to be supported. Also, you can use the Asynchronous File-Based Session Storage if you need to keep the file-based storage functionality and good performance.

Manipulating Threads Affinity

By default, threads are used by FIX session to send and receive FIX messages; they can be executed on any of available processors/cores. Specifying CPU affinity for each session thread may give a significant performance boost:

Session::receivingThreadAffinity(...);
Session::sendingThreadAffinity(..);

The typical approach is to set different CPU affinities for each thread. However, when the number of threads is greater than the CPU cores, one can use the following approach:

• Each high-loaded session that receives/sends messages at a high rate should be pinned to a separate CPU core.
• All other less-loaded sessions can be pinned to the same CPU core(s).

Also, actual performance benefits for the affinity depend on the particular system and application architecture, so it makes sense to try different settings to find optimal ones.

Using receive spinning timeout

OnixS::FIX::Session::receiveSpinningTimeoutUsec method can be used to decrease the latency of the data receiving. If the value is zero (by default), the receiving thread will wait for a new FIX message in the blocking wait mode. If the value greater than zero, the receiving thread will wait for a new FIX message in the spin loop mode before switching to the blocking wait mode. The method parameter specifies the spin loop period in microseconds. OnixS::FIX::Session::receiveSpinningTimeoutUsec using makes sense when your session receives FIX messages frequently, in this case, waiting in the loop is cheaper than the thread context switch to the blocking wait mode.

Note

The spin wait increases the CPU usage, so the spin wait period should not be too long. Also, for these reasons, the spin wait can negatively impact the performance when an application uses too many high-loaded FIX sessions. Therefore, in such cases, it can be better not to use the spin wait.

Using send spinning timeout

OnixS::FIX::Session::sendSpinningTimeoutUsec method can be used to decrease the latency of the data sending. If the value is zero (by default) and the outgoing message cannot be sent immediately it is placed to the outgoing queue. If the value greater than zero, the OnixS::FIX::Session::send method waits for the socket sending buffer availability in the spin loop mode before placing the message to the outgoing queue (to be sent later by the sending thread). The method parameter specifies the spin loop period in microseconds. OnixS::FIX::Session::sendSpinningTimeoutUsec using makes sense when your session sends FIX messages frequently, in this case, waiting in the loop is cheaper than the thread context switch.

Note

The spin wait increases CPU usage and blocks the thread from which OnixS::FIX::Session::send method is called, so the spin wait period should not be too long. Also, for these reasons, the spin wait can negatively impact the performance when an application uses too many high-loaded FIX sessions. Therefore, in such cases, it can be better not to use the spin wait.

Using spinlock

The OnixS::FIX::Session::useSpinLock option can decrease sending and receiving latency using a spinlock mutex instead of the standard one.

Note

The spin wait increases CPU usage. It is not recommended to be used with the ThreadPool threading model.

This parameter is ignored in the TCPDirect mode.

Using session warmup

OnixS::FIX::Session::warmUp method can be used to warm up the sending path. It makes sense if your session sends FIX messages infrequently, in this case, the sending path and associated data structures will not be in a cache and this can increase the send latency. You can periodically (a recommended period is 500 usec and less) call the OnixS::FIX::Session::warmUp to avoid cache misses and keep the sending path fast. OnixS::FIX::FlatMessage object is required to warm up the sending path including the FIX message assembling and it will not be actually sent.

Additionally, you can pass warm-up flags to this method to turn on the warmup feature for a specific NIC. Some of such NICs have its own TCP stack implementation and introduce such flags to warm up the sending path of the TCP stack. In order to check that such a flag works as expected you can use the OnixS::FIX::Session::checkWarmupFlags function. This function creates server/client test sockets and tries to send data with the given flag. The OnixS::FIX::Session::checkWarmupFlags function should be called only once during an application initialization or once per a session local network interface if there are sessions that work via different network interfaces, e.g.:

    // Some code to initialize an application.
    const int SomeSpecificWarmupFlag = 0x100000;
    const unsigned short BaseListenPort = 5000;
    const std::string SessionLocalNetworkInterface = "127.0.0.1";
    bool someSpecificWarmupFlagIsSupported = Session::checkWarmupFlags(SomeSpecificWarmupFlag, BaseListenPort, SessionLocalNetworkInterface);
    //..
    // Warm-up code for a particular session.
    if(someSpecificWarmupFlagIsSupported)
        initiator.warmUp(&warmupMsg, SomeSpecificWarmupFlag);
    else
        initiator.warmUp(&warmupMsg);
    //..

Using session send batch

OnixS::FIX::Session::send method can be used to send messages in a batch. It can decrease the send latency because all batch messages will be sent in one TCP packet. This ability is available for OnixS::FIX::Message and OnixS::FIX::FlatMessage classes. The OnixS::FIX::Session::MsgBatch class represents the message batch. This class instance should be created in advance on the non-critical path, and required messages should be added. After that, one can send the batch on the critical path:

    // Create the MsgBatch object on the non-critical path.
    const size_t MessageCount = 10;
    Session::FlatMessageBatch batch;
    // Add messages to the batch.
    for(size_t messageCounter = 0; messageCounter < MessageCount; ++messageCounter)
    {
        Session::FlatMessageBatch::MessagePtr msgPtr(createMessage());
        batch.add(msgPtr);
    }
    // Prepare keys for updating serialized messages in the batch.
    std::vector<FlatFieldKey> orderQtyKeys;
    for(size_t messageCounter = 0; messageCounter < MessageCount; ++messageCounter)
    {
        orderQtyKeys.push_back(batch[messageCounter].allocateKey(FIX42::Tags::OrderQty));
    }

    // On the critical path one can update field values in the batch and send messages to the wire.
    for(size_t messageCounter = 0; messageCounter < MessageCount; ++messageCounter)
    {
        batch[messageCounter].set(orderQtyKeys[messageCounter], 10);
    }
    session.send(batch);

Also, you can decrease the send latency a little more by using OnixS::FIX::Session::sendAsIs methods. These methods send serialized message(s) without any fields updating, you can prepare message(s) before sending by setting all necessary fields and using the OnixS::FIX::Session::preFill methods.

Note

The OnixS::FIX::Session::messageGrouping setting conflicts with the usage of the session send batch and can affect it. For the batch sending, the session uses the "gathering" socket send system call, which accepts an array of buffers to send all of them at once. When one sets the OnixS::FIX::Session::messageGrouping setting, the session tries to combine (copy) outgoing messages (when available) to the one outgoing buffer until the given grouping value is achieved and then uses the regular socket send system call for sending. Therefore, one should not use the OnixS::FIX::Session::messageGrouping option and batch sending simultaneously.

Consider to switch off the logging before sending

By default, the logging of an outgoing message to the session storage is performed before sending to the wire. This is more reliable because we guarantee that an outgoing message is stored before going to the counterparty and if the application is shut down after sending, for some reasons, the sent message can be resent afterward. However, this approach adds the logging latency to the FIX Engine sending latency. As a result, it increases the tick-to-trade latency. When the latency is more important, one can switch off the logging before sending, by setting the OnixS::FIX::Session::logBeforeSending option to false. In this case, the logging of outgoing messages to the session storage will be performed after sending to the wire. This way, one can exclude the logging latency from the FIX Engine sending latency and as a result, decrease the tick-to-trade latency.

Updating Engine Configuration

Disabling Resend Requests Functionality

If no messages are re-sent on counterparty's Resend Request messages, then it is possible to set the Resending Queue size to zero to increase overall performance.

    EngineSettings settings;
    settings.resendingQueueSize(0);

Disabling FIX Messages Validation

Validation significantly reduces performance due to miscellaneous checks, performed on each FIX message. To disable FIX messages validation the following settings should be used:

    EngineSettings settings;
    settings.validateFieldValues(false)
            .validateRequiredFields(false)
            .validateUnknownFields(false)
            .validateUnknownMessages(false);

Optimizing FIX Dictionaries

When FIX Message is constructed, a space for all the fields, defined for the message, is allocated. The latest FIX specifications define a lot of fields for each message type. Even when most of the fields are not used, FIX Engine reserves space for them, and this has a negative effect on FIX Engine's performance.

Editing Dictionaries Descriptions and excluding messages and fields (especially repeating groups) which are not used by an application, have a direct influence onto the FIX messages processing speed. In this way, it decreases the general latency of the FIX message related processing.

Manipulating FIX messages

While constructing FIX messages, each time message, that has to be sent, is not efficient, because it involves memory allocation and subsequent destruction. The OnixS::FIX::Message class provides the ability to reuse a message of a particular type. It exposes the OnixS::FIX::Message::clear method, which wipes out all the FIX fields, assigned to the given instance. This method does not deallocate all memory occupied, it just brings the message to the initial state as it was just constructed. However, after the previously allocated memory is reused, thus, the second use of the object allows to setup FIX fields faster.

Reusing FIX message instance

A common strategy is to use the same message instance for sending multiple times. As soon as a message of a certain type is constructed, it is updated with common fields for the subsequent re-use. Afterwards, before sending it to the counterparty, it is updated with mutable fields (like price or quantity) exactly for the case and sent to the counterparty.

As mutable fields are updated, each time message is about to be sent to the counterparty.

Using Flat FIX messages

As an advance for reusing FIX messages, the concept of Flat FIX Message has been developed. In general, it follows the same approach of updating only mutable fields, before sending a message to the counterpart.

The only difference is that the message is serialized into 'tag=value' presentation in advance, and the serialization is not required, when the message is actually sent.

Using Solarflare-specific Features

Please see the Solarflare Specific Features.