Low Latency Best Practices

This section summarizes our findings and recommends best practices to tune the different layers of the FIX Engine for latency-sensitive workloads. By latency-sensitive, we mean workloads that are looking at optimizing for a few microseconds to a few tens of microseconds end-to-end latencies; we don’t mean workloads in the hundreds of microseconds to tens of milliseconds end-to-end-latencies. Many of the recommendations that can help with the microsecond level latency, can hurt the performance of applications that are tolerant of higher latency. Please note that the specific benefits and effects of each of these configuration choices will be highly dependent upon the particular applications and workloads, so we strongly recommend experimenting with the different configuration options with your workload before deploying them in a production environment.

Setting Process.PriorityClass property

A process priority class encompasses a range of thread priority levels. We recommend setting ProcessPriorityClass.RealTime. For more details, please see Process.PriorityClass Property.

C#:

Process.GetCurrentProcess().PriorityClass = ProcessPriorityClass.RealTime;

Session Tuning

Selecting Right Session Storage

Using MemoryBasedStorage instead of FileBasedStorage boosts performance since FIX messages are stored directly in memory. Alternatively, it's possible to use a custom pluggable storage (using PluggableStorage ) which does nothing on FIX message-related operations as soon as no resend requests are to be supported. Also, you can use the Async File-based Session Storage if you need to keep the file-based storage functionality and excellent performance.

Manipulating Threads Affinity

By default, threads are used by FIX session to send and receive FIX messages. They can be executed on any of the available processors/cores. Specifying CPU affinity for each session thread may give a significant performance boost: ReceivingThreadAffinity, SendingThreadAffinity.

Receive Spinning Timeout

ReceiveSpinningTimeout property can be used to decrease the latency of the data receiving. If the value is zero (by default), the receiving thread will wait for a new FIX message in the blocking wait mode. If the value greater than zero, the receiving thread will wait for a new FIX message in the spin loop mode before switching to the blocking wait mode.

The property specifies the spin loop period in microseconds.

ReceiveSpinningTimeout property using makes sense when your session receives FIX messages frequently - in this case, waiting in the loop is cheaper than the thread context switch to the blocking wait mode. However please note that the spin wait increases the CPU usage so the spin wait period should not be too long.

Send Spinning Timeout

SendSpinningTimeout property can be used to decrease the latency of the data sending. If the value is 0 (the default value) and the outgoing message cannot be sent to the network immediately, it is placed in the outgoing queue. If the value greater than zero, the Send(IMessage) method waits for the socket sending buffer availability in the spin loop mode before placing the message to the outgoing queue (to be sent later by the sending thread).

The property specifies the spin loop period in microseconds.

SendSpinningTimeout property using makes sense when your session sends FIX messages frequently - in this case, waiting in the loop is cheaper than the thread context switch. However please note that the spin wait increases the CPU usage and blocks the thread from which OnixS::FIX::Session::send method is called, so the spin wait period should not be too long.

Session Warmup

WarmUp(SerializedMessage) method can be used to warm up the sending path. It makes sense if your session sends FIX messages infrequently, the sending path and associated data structures will not be in a cache, and this can increase the send latency. You can periodically (a recommended period is 500 usec and less) call the WarmUp(SerializedMessage) to avoid cache misses and keep the sending path fast. SerializedMessage object is required to warm up the sending path including the FIX message assembling and it will not be sent.

Reusing Message Instances and Event Arguments in Event Handlers

Object creation is an expensive operation in .NET, with impact on both performance and memory consumption. The cost varies depending on the amount of initialization that needs to be performed when the object is to be created. OnixS .NET FIX Engine exposes an ability to reuse message instances and event arguments in event handlers by the Session. We highly recommend to turn on ReuseInboundMessage and ReuseEventArguments to minimize the excess object creation and garbage collection overhead.

Note

If ReuseInboundMessage turns on, the client's code must copy a message for using outside of inbound callbacks. If ReuseEventArguments turns on, the client's code must copy event arguments for using out of callbacks.

Updating Engine configuration

Disabling Resend Requests functionality

If no messages are re-sent on counterparty's Resend Request messages, then it is possible to set the Resending Queue size to zero, to increase overall performance.

EngineSettings settings = new EngineSettings()
{
    ResendingQueueSize = 0
};

Disabling FIX messages validation

Validation significantly reduces performance due to miscellaneous checks that are performed on each FIX message.

The code sample below shows how to disable FIX messages validation:

EngineSettings settings = new EngineSettings()
{
    ValidateCheckSum = false,
    ValidateDuplicatedFields = false,
    ValidateEmptyFieldValues = false,
    ValidateFieldValues = false,
    ValidateRepeatingGroupEntryCount = false,
    ValidateRepeatingGroupLeadingField = false,
    ValidateRequiredFields = false,
    ValidateUnknownFields = false,
    ValidateUnknownMessages = false
};

Optimizing FIX Dictionaries

When FIX Message is constructed, the space for all the fields, that are defined for the message, is allocated. The latest FIX specifications define a lot of fields for each message type. Even when most of fields are not used, FIX Engine reserves the space for them, and this hurts the performance.

Editing dictionaries descriptions and excluding FIX Protocol versions, messages and fields (especially repeating groups), which are not used by the application, have direct influence onto FIX messages processing speed and thus decrease overall latency of FIX message related processing.

For example:

<?xml version="1.0" encoding="utf-8"?>
<Dialect xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="https://ref.onixs.biz/fix/dialects https://ref.onixs.biz/fix/dialects/dialects-2.19.xsd" xmlns="https://onixs.biz/fix/dialects">

  <FIX version="4.0" mode="remove"/>

  <FIX version="4.1" mode="remove"/>

  <FIX version="4.2" mode="remove"/>

  <FIX version="4.3" mode="remove"/>

  <FIX version="4.4" mode="remove"/>

  <FIX version="5.0" mode="remove"/>

  <FIX version="5.0_SP1" mode="remove"/>

  <FIX version="5.0_SP2" mode="remove"/>

  <FIX version="4.2" id="LowLatencyDictionaryId" mode="override" revision="1.0.0.0">

    <Message type="D" mode="override">
      <Field tag="11" name="ClOrdID"/>
      <Field tag="109" name="ClientID"/>
      <Field tag="21" name="HandlInst"/>
      <Field tag="55" name="Symbol"/>
      <Field tag="54" name="Side"/>
      <Field tag="38" name="OrderQty"/>
      <Field tag="40" name="OrdType"/>
      <Field tag="60" name="TransactTime"/>
    </Message>

  </FIX>
</Dialect>

Manipulating Messages

Constructing FIX messages, each time message has to be sent, is not efficient because it involves memory allocation and subsequent destruction. Message class provides the ability to reuse message of a particular type. It exposes the Reset() method, which wipes out all the FIX fields that are assigned to the given instance. This method does not deallocate all memory occupied; it just brings the message to the initial state, as it was just constructed. However, the previously allocated memory is reused, and thus second use of the object allows to setup FIX fields faster.

Reusing Message Instance

A common strategy is to use the same message instance for sending multiple times. As soon as a message of a particular type is constructed, it is updated with common fields for subsequent reuse. Afterward, before sending it to the counterparty, it is updated with mutable fields (like price or quantity) precisely for the case and sent to the counterparty.

Mutable fields are updated each time message is about to be sent to the counterparty.

Reusing Message With Groups Instance

To reuse the same message with repeating groups, use the MessageMemoryPool instance. It contains pre-allocated objects to be reused by messages. Depending on the message structure and data, the pool must be properly initialized. The following sample demonstrates the pool for the message with one repeating group and less than 41 group instances.


var messageInfo = dictionary.GetMessageInfo(MsgType.MarketDataSnapshotFullRefresh);

const int MaxNumberOfGroupInstances = 40;
var memoryPool = new MessageMemoryPool(messageInfo, MaxNumberOfGroupInstances);
memoryPool.Preallocate();

const int MaxMessageContentSize = 1000;
var snapshot = new Message(MsgType.MarketDataSnapshotFullRefresh, dictionary, MaxMessageContentSize, memoryPool);

//preparing before the first sending
{
    snapshot
        .Set(Tag.MDReqID, "1")
        .Set(Tag.Symbol, "A");

    var group = snapshot.SetGroup(Tag.NoMDEntries, MaxNumberOfGroupInstances );
    for (int i = 0; i < group.NumberOfInstances; i++)
    {
        group[i].Set(Tag.MDEntryType, 1);
        group[i].Set(Tag.TradeID, "1");
    }
}

//preparing before the next sending
{
    snapshot.Reset();

    snapshot
        .Set(Tag.MDReqID, "2")
        .Set(Tag.Symbol, "B");

    var group = snapshot.SetGroup(Tag.NoMDEntries, MaxNumberOfGroupInstances);
    foreach (var groupInstance in group)
    {
        groupInstance.Set(Tag.MDEntryType, 2);
        groupInstance.Set(Tag.TradeID, "2");
    }
}
Note

For proper usage of allocated objects it is necessary to use Message.Reset() before changing the number of group entries. If the pool contains fewer objects than required, additional objects will be allocated, but they will not be reused.

Using Serialized Messages

As an advance for reusing FIX messages, the concept of SerializedMessage was developed. In general, it follows the same approach of updating only mutable fields, before sending a message to the counterparty.

The only difference is that the message is serialized into 'tag=value' presentation in advance and, the serialization is not required when the message is sent.

Using GC-free interface

When you manipulate string field values of a message object by the general Get/Set interface, temporary string objects are created each time. It increases the GC pressure and, as a result, it can increase the latency of the code execution. Therefore, to minimize the GC pressure, one can use the following GC-free interface:

In order to compare field values, without a GC pressure, one can use CompareFieldValue(int, string) method. For example, the following code:

if (message.CompareFieldValue(Tag.ClOrdID, "ClOrdIdValue"))

does not allocate a managed memory at run-time, however, the following one:

if (message.Get(Tag.ClOrdID) == "ClOrdIdValue")

does, because it creates a temporary string object.

In order to compare the message type value, without a GC pressure, one can use CompareType(string) or CompareType(char) methods. For example, the following code:

if (message.CompareType("AE"))

does not allocate a managed memory at run-time, however, the following one:

if (message.Type == "AE")

does, because it creates a temporary string object.

Note

All string literals, in C# code, are converted to string objects implicitly. Such string objects are stored in the internal pool and they are created only once when a string literal appears the first time in the code. Therefore, you need to create it in advance to avoid a string allocation on the critical path.

In order to get/set field values, without a GC pressure, one can use Set(int, StringBuilder) and Get(int, StringBuilder) methods.

For example, the following code:

// stringValue object of StringBuilder type is created previously in some place.
msg1.Get(Tag.Text, stringValue);
msg2.Set(Tag.Text, stringValue);

does not allocate a managed memory at run-time, however, the following one:

string stringValue = msg1.Get(Tag.Text);
msg2.Set(Tag.Text, stringValue);

does, because it creates a temporary string object.

Note

The StringBuilder object should be created in advance, on the non-critical path. Also, it is important to create it with the required and sufficient capacity of the internal buffer to avoid further memory reallocation on the critical path.

See Also