This section describes settings that can improve the sustained message rate of the C++ FIX Engine. Their effects depend on the application, message sizes, traffic pattern, network path, and counterparty.
Start with a Release-build baseline that measures message rate, latency, CPU and memory usage, and network utilization. Change one setting at a time and repeat the measurements with representative message sizes, bursts, round-trip times, and counterparties before deploying the setting in production.
The Session class exposes several settings. This section covers the subset most relevant to high-throughput workloads.
See Selecting the Right Storage for the storage performance and recovery tradeoffs.
See Manipulating Thread Affinity for guidance on assigning heavily loaded session threads to CPU cores.
Many small packets incur more protocol and processing overhead than fewer larger packets. Message grouping lets an application trade some queuing latency for better network and receiver efficiency. Configure the default for all sessions with OnixS::FIX::EngineSettings::messageGrouping, or override it for an individual session with OnixS::FIX::Session::messageGrouping.
With values 0 and 1, the Engine first tries to send each FIX message from the caller's thread and queues it for the sending thread only when the inline send cannot complete. With a value greater than 1, the Engine immediately queues the serialized packet for transmission by the sending thread. The latter behavior can improve sustained throughput while adding queuing latency.
Message grouping and session batch sending use different aggregation mechanisms and should not be enabled together. See Using session send batch for details. Use the Throughput Sample as a starting point for workload-specific measurements.
TCP behavior depends on the operating-system network stack, the network path, and the peer. Verify the effect of each change on the complete deployment rather than relying on a fixed platform default.
Nagle's algorithm reduces protocol and processing overhead by combining small writes. For sustained high-throughput workloads dominated by small messages, enable it with OnixS::FIX::EngineSettings::tcpNoDelayOption(false). It can delay individual messages, so set OnixS::FIX::EngineSettings::tcpNoDelayOption(true) when measured latency requirements outweigh the throughput benefit.
The Bandwidth Delay Product (BDP) is the amount of data that must be in flight to fill a network path. It is the product of the available link bandwidth and round-trip time (RTT):
For a 100 Mbit/s path with a 50 ms RTT:
The corresponding line-rate throughput is 12.5 MB/s. A connection needs enough effective window and buffering to keep that volume of data in flight, but the BDP is a starting point for capacity analysis rather than an exact value to assign to every socket buffer.
The socket send buffer stores data awaiting transmission, while the socket receive buffer limits the capacity available to receive data. The advertised receive window also depends on TCP window scaling, and the sender is independently limited by its congestion window and network loss. Consequently, setting both socket buffers to the BDP does not guarantee line-rate throughput.
Modern operating systems normally tune TCP buffers and receive windows automatically. On Linux, explicitly setting SO_SNDBUF or SO_RCVBUF disables automatic tuning for that socket's corresponding buffer; see the Linux TCP documentation. Windows exposes receive-window autotuning through Get-NetTCPSetting. TCP window scaling is defined by RFC 7323.
Start with operating-system-managed sizing. Override it only when measurements show that buffer or receive-window capacity constrains the connection. The following example shows how to set the socket buffers with OnixS::FIX::EngineSettings::sendBufferSize (SO_SNDBUF) and OnixS::FIX::EngineSettings::receiveBufferSize (SO_RCVBUF):
