Detecting Packet Gaps

Due to the unreliable nature of multicast, packets transmitted by the MDP may come in an order different from the original or completely lost. The Handler uses an internal procedure to solve issues in the sequence of incoming packets. The Handler's behavior for the case is controlled by the two parameters OnixS::CME::ConflatedUDP::RealtimeFeedSettings::outOfOrderPacketMaxInterval and OnixS::CME::ConflatedUDP::RealtimeFeedSettings::lostPacketWaitTime.

Note: The noted parameters are a part of the parameters affecting the behavior of the incremental feeds. The set of parameters can be accessed using the following path: handler.settings().feeds().incrementalFeeds().; The described way the Handler detects gaps in incoming packets refers to incremental data only. In the case of instrument or snapshot recovery, the Handler uses different strategies for solving sequencing issues.

When the Handler receives an incoming packet with the sequence number greater than expected, it decides on whether to wait for expected packets or continue accepting incoming packets for further processing, thus ignoring the expected ones. Decision making is based on time measuring and comparing sequence numbers of incoming packets to the expected ones.

Suppose the Handler receives a packet with the sequence number greater than expected. In that case, it compares the packet's sequence number with the threshold, which represents the sum of the seqNumberOfLastAcceptedPacket and the outOfOrderPacketsMaxInterval values, where the outOfOrderPacketsMaxInterval is the value of the parameter mentioned above. The seqNumberOfLastAcceptedPacket is the sequence number of the last packet the Handler accepted for further processing. If the threshold is not exceeded, the Handler stores the incoming packet in the internal queue and waits until a new packet is received. Otherwise, the Handler decides on packet loss and raises the corresponding event.

The Handler also establishes the time limit it waits for missing packets. If the Handler receives missing data within the established time interval, it resumes normal data processing. However, if no data is received for the predefined time frame, the Handler considers missing data as lost and raises the gap event. The OnixS::CME::ConflatedUDP::RealtimeFeedSettings::lostPacketWaitTime parameter defines the size of the time interval the Handler waits for missing data.

The important aspect is that time limit and sequence number checking are two independent criteria. Suppose the missed data is not received within the predefined time interval. In that case, the Handler decides on data loss no matter whether sequence numbers of incoming packets exceed the noted threshold or not. And vice versa.

Gap Detection and Feed Arbitrage

Previously, the Handler used common queue to store incoming packets while waiting for missing ones. Therefore, if there was a significant time delay in data transmission between primary and secondary feeds, the threshold checking criterion could be satisfied quicker by incoming packets from a faster feed. As a result, the higher threshold had to be used to let the Handler recover missing packets from the slower feed. Also, the time limit had to be increased to cover the difference in data transmission between the faster and the slower feeds.

Needless to say, the given approach wasn't convenient enough because substantially depended on characteristics of a particular environment. The sequence number checking is performed for each feed independently starting from the 5.3 release. If arbitrage between incremental feeds is enabled, decision making does not trigger the gap event until sequence number threshold is exceeded by incoming packets from both feeds. The given behavior change improves gap detection in case of a significant time delay between real-time feeds.

Gap Interval

As previously noted, the Handler enqueues incoming packets while waiting for expected ones. When either of the gap detection criteria is get satisfied, the Handler raises the gap event. The important aspect is how the Handler determines the gap interval and what happens with enqueued packets.

If one of the gap detection criteria is satisfied, the Handler moves to raising the gap event. However, before the gap event is reported, the Handler determines the range of missing packets. For this purpose, it analyses enqueued packets and extracts the most recent packets representing continuous subsequence without gaps inside them. The extracted packets will be accepted for further processing. Other packets will be disregarded. Thus, the gap will represent an interval, the beginning of which is the last accepted (processed) packet plus one, and the end is the first packet of the extracted continuous sequence minus one:

1 [ seqNumberOfLastProcessedPacket + 1, seqNumberOfHeadOfContinuousSubsequence - 1 ]

For example, if the last processed (accepted) packet is #1000 and pending packets are #1002, #1005, #1007, and #1008, then the gap interval will represent [1001, 1006]. Pending packets #1002 and #1005 will be disregarded, whereas packets #1007 and #1008 will be accepted for further processing.