Taking Timestamps of the Received Data

On Linux, the FIX Engine can probe the timestamp of the received data and provide it to the user along with the data. This can be useful to monitor the TCP/IP stack performance. The timestamp of the received data is delivered via the OnixS::FIX::ISessionListener::onReceivedBytes() callback:

class DataListener : public ISessionListener
{
public:

    void onInboundApplicationMsg(Message&, Session*) ONIXS_FIXENGINE_OVERRIDE { }

    void onReceivedBytes(const char *, size_t, const ReceivedDataTimestamp & timestamp, Session *) ONIXS_FIXENGINE_OVERRIDE
    {
        std::clog << "Received data timestamp: " << timestamp.toString() << std::endl;

        if(timestamp.source() != ReceivedDataTimestamp::None)
        {
            const TimeSpan latency = Timestamp::utc() - timestamp.timestamp();
            std::clog << "Received data latency: " << latency.toString(TimeSpanFormat::HHMMSSPsec) << std::endl;
        }
    }
};

Depending on the OnixS::FIX::Session configuration, hardware capabilities, and the operation system, OnixS::FIX::ReceivedDataTimestamp::Source can have the following values:

• None: no timestamp is taken;
• Software: a timestamp provided by the OS kernel / user-level network stack is taken;
• Hardware: a timestamp provided by the network card is taken;

If accessible, the network interface card timestamp is used (ReceivedDataTimestamp::Source::Hardware); otherwise, the timestamp supplied by the OS kernel or the user-level network stack is used (ReceivedDataTimestamp::Source::Software).

To configure the session to take the timestamp, the OnixS::FIX::EngineSettings::enableRxTimestamp() or OnixS::FIX::Session::enableRxTimestamp() must be set:

Session session("Sender", "Target", ProtocolVersion::FIX_42, &listener);
session.enableRxTimestamp(true);

Note

For acceptor sessions, the global OnixS::FIX::EngineSettings::enableRxTimestamp() setting can be only used because an acceptor connection is created before a corresponding acceptor session is determined.

To configure the session to take the Hardware timestamp, the corresponding configuration should be applied on the given network interface card. One can do it using the OnixS::Sockets::Nic::setHardwareTimestamp() helper method:

// Root privileges are required.
OnixS::Sockets::Nic::setHardwareTimestamp("ethX", true);

If the OnixS OnixS::FIX::TCPDirect::Stack is used, the timestamping must be enabled on the stack level:

    TCPDirect::Attributes attr;
    attr.set("rx_timestamping", 1);
    TCPDirect::Stack stack(attr);

If the Solarflare OpenOnload is used, The EF_RX_TIMESTAMPING environment variable must be set to 1:

EF_RX_TIMESTAMPING=1 onload ./onixs-ilink3-app

To compare hardware-based timestamps with the host time, it is necessary to synchronize the network interface card clock with the host clock, as described in IEEE 1588-2008.

For the Solarflare cards, the sfptpd utility can be used to perform this synchronization.

Note

According to our measurements, enabling this feature adds around 50-100 nanoseconds to the resulting latency.

This functionality is available on Linux only.

The TCP protocol is a stream protocol, so the received data timestamp may not correlate with the number of messages sent due to TCP fragmentation. Also, a few messages can be combined in one TCP segment. Therefore, the timestamp provided in the OnixS::FIX::ISessionListener::onReceivedBytes() represents the timestamp of the last data chunk.

When a counterparty sends FIX messages at a high rate, and the application cannot process received messages in time, the Linux core can drop the timestamp information due to an internal timestamp queue overflow. In this case, the OnixS::FIX::ReceivedDataTimestamp::Source will be set to ReceivedDataTimestamp::Source::None, and the timestamp value will contain the previous last timestamp value if it exists.

This article describes how to detect TCP congestion using described functional.