OnixS C++ CME Market Data Handler  5.7.0
API documentation
Benchmarking with Network/Kernel Layer

Preface

The default benchmarking procedure descrived in the Benchmarking Results topic measures application-level latency and does not take into consideration time the incoming data spends in the network/kernel layer. The given approach lets analyzing critical path inside the processing machinery (Handler). However, an application may not receive incoming data as soon as it arrives. Thus, measured latency does not reflect the other important aspects like time data spends in the network adapter or a kernel.

In order to improve the analysis, the support of hardware timestamps has been added to the SDK. Hardware timestamps are assigned to the incoming packets as soon as they arrive at the network adapter. If hardware timestamps are taken as the initial point of the latency measurement, that lets to obtain a timespan starting from the moment the data arrived to the network card and to the moment the results of data processing are delivered to the user.

The following tables depict the results of benchmarking using the regular (application-only) approach and compares them with the results obtained with the help of hardware timestamps. Also, the measurements were taken for the two major implementations of the Feed Engine machinery exposed by the SDK and encapsulated into OnixS::CME::MDH::SocketFeedEngine and OnixS::CME::MDH::SolarflareFeedEngine classes which use ordinary sockets and the Solarflare ef_vi SDK correspondently. Additionally, a socket-based feed engine was benchmarked in the OpenOnload environment to depict the actual benefits of using the OpenOnload for ordinary (socket-based) solutions.

Application-Level Only With 5 Milliseconds Delay Between Packets

Statistics Latency (μs)
SocketFeedEngine SolarflareFeedEngine
Minimal 0.993 0.856
Median 1.144 1.017
Mean 1.203 1.071
95% 1.493 1.383
99% 2.047 1.830
Maximal 14.344 14.680

With Network/Kernel-Layer And 5 Milliseconds Delay Between Packets

Statistics Latency (μs)
SocketFeedEngine SolarflareFeedEngine
Minimal 14.007 2.190
Median 52.087 2.664
Mean 48.853 2.709
95% 55.894 3.152
99% 60.966 3.798
Maximal 79.933 16.385

Application-Level Only Without Any Delay Between Packets

Statiscits Latency (μs)
SocketFeedEngine SocketFeedEngine+Openonload SolarflareFeedEngine
Minimal 0.507 0.533 0.600
Median 0.667 0.693 0.714
Mean 0.739 0.765 0.778
95% 1.156 1.113 1.067
99% 1.564 1.560 1.490
Maximal 14.019 14.617 14.756

With Network/Kernel-Layer And Without Any Delay Between Packets

Statiscits Latency (μs)
SocketFeedEngine SocketFeedEngine+Openonload SolarflareFeedEngine
Minimal 10.786 3.235 1.925
Median 12.610 4.148 2.353
Mean 12.676 4.267 2.377
95% 14.238 5.260 2.853
99% 15.562 9.407 3.310
Maximal 64.169 22.261 16.210

Conclusions and Important Notes

Using hardware timestamps brings kernel/hardware latency into overall measurements and thus allows performing analysis more precisely. For example, as depicted by the tables above, application-only latency is similar to the major implementations of the Feed Engine exposed by the SDK. However, using hardware timestamps shows the real benefits of using kernel-bypass solutions like the OpenOnload is. Moreover, it depicts in numbers the advantage of using specialized solutions like Solarflare ef_vi SDK while working with multicast data.