Understanding Send Latency

The overall latency of the OnixS::FIX::Session::send method includes all latencies of all operations inside this method. The following sequence of operations are performed when the OnixS::FIX::Session::send method is called:

• Invocation of the OnixS::FIX::ISessionListener::onOutboundSessionMsg or OnixS::FIX::ISessionListener::onOutboundApplicationMsg callback (for OnixS::FIX::Message objects only).
• Serialization of the outbound message to the raw buffer. The latency of this action depends on the message class you use. For the OnixS::FIX::Message class it is an expensive operation and it can take a few microseconds what depends on the message size. For the OnixS::FIX::FlatMessage class it is a very cheap operation because an object of this class represents the serialized buffer.
• Writing necessary data (message content, timestamp, sequence numbers, etc.) to the session storage. For the File-Based Session Storage, it is a very expensive operation and it can take tens of microseconds. For the Asynchronous File-Based Session Storage, it is far cheaper, because the actual writing is performed in a separate thread. And lastly, for the Memory-based Session Storage, it is a very cheap operation, because data is stored in the memory, not in the file system. Please note, that if the OnixS::FIX::Session::logBeforeSending option is false, then the logging is performed after sending, please see the corresponding article of the Low Latency Best Practices page for details.
• Invocation of the OnixS::FIX::ISessionListener::onMessageSending callback.
• Invocation of the system send function of the TCP socket. In practice, the latency of the system send function can be about 5-10 microseconds what depends on the size of send data, network workload, network software/hardware, etc. For example, you can decrease this latency by using Solarflare network cards with their own TCP stack implementation.
• If the system send function fails, then the serialized raw buffer is stored in the sending queue for the next process in the sending thread.

Based on the above, one can understand what the send latency includes. However, there is another important aspect. The latency of OnixS::FIX::Session::send method (as any other function) depends on the period with which you call it. When you call a function infrequently, in this case, the function path and associated data structures will not be in a processor cache and this can increase the function latency. However, if you call a function frequently then you can avoid cache misses and keep the function path fast. For this purpose, there is the OnixS::FIX::Session::warmUp method, please see the Low Latency Best Practices page for details.