Understanding Send Latency

The overall latency of the OnixS::CME::ConflatedTCP::Session::send method includes all latencies of all operations inside this method. The following sequence of operations are performed when the OnixS::CME::ConflatedTCP::Session::send method is called:

• Invocation of the OnixS::CME::ConflatedTCP::SessionListener::onOutboundSessionMsg or OnixS::CME::ConflatedTCP::SessionListener::onOutboundApplicationMsg callback.
• Writing necessary data (message content, timestamp, sequence numbers, etc.) to the session storage. For the File-Based Session Storage, it is a costly operation, and it can take tens of microseconds. For the Asynchronous File-Based Session Storage, it is far cheaper, because the actual writing is performed in a separate thread. And lastly, for the Memory-based Session Storage, it is a low-cost operation, because data is stored in the memory, not in the file system. Please note, that if the OnixS::CME::ConflatedTCP::Session::logBeforeSending option is false, then the logging is performed after sending.
• Invocation of the OnixS::CME::ConflatedTCP::SessionListener::onMessageSending callback.
• Invocation of the system send function. In practice, the latency of the system send function can be about 5-10 microseconds what depends on the size of send data, network workload, network software/hardware, etc. For example, you can significantly decrease this latency by using Solarflare network cards and the user-space "Onload" TCP stack implementation.
• If the system send function fails, then the unsent message is stored in the sending queue for the subsequent sending by the sending thread.

The latency of OnixS::CME::ConflatedTCP::Session::send method (as any other function) depends on the frequency with which you call it. When you call a function infrequently, the function path and associated data structures will not be in a processor cache, and this can increase the function latency. However, if you call a function frequently, then you can avoid cache misses and keep the function path fast. For this purpose, there is the OnixS::CME::ConflatedTCP::Session::warmUp method, please see Warmup.