Market data processing requires the execution of different procedures like a large-scale recovery to obtain the up-to-date market state on late join or requesting lost Incremental data. While developing a trading solution, knowing the market data processing state is essential.
To report about state changes, the handler invokes the OnixS::CME::MDH::HandlerListener members once its instance is bound to the handler using the handler.settings().listeners().handler(&listener) path.
The following list summarises the handler events.
The phases below can overlap or follow different sequences depending on the handler settings and external conditions. They do not form a strict state machine.
| Phase | Indication | Data readiness | Application response |
|---|---|---|---|
| Stopped | Before the first OnixS::CME::MDH::Handler::start call, after OnixS::CME::MDH::Handler::stop returns, or after a failed startup has completed. | No market data is being processed. | Configure settings and listeners while the handler is stopped, outside callbacks; see Operations from callbacks. |
| Started | After OnixS::CME::MDH::HandlerListener::onStarted. | Processing has begun, but this event does not indicate that instruments or market data are ready. | Wait for the recovery and real-time processing events that apply to the configured session. |
| Instrument recovery | Between OnixS::CME::MDH::HandlerListener::onInstrumentRecovery and OnixS::CME::MDH::HandlerListener::onEndOfInstrumentRecovery. | Instrument definitions are being established. Successful completion reports restored instrument definitions; Failed or Incomplete completion can lead to another recovery attempt. Market readiness must not be inferred from this phase. | Inspect the recovery completion status. |
| Market recovery | Begins with OnixS::CME::MDH::HandlerListener::onMarketRecovery; completion is reported through OnixS::CME::MDH::HandlerListener::onEndOfMarketRecovery when reached. | Books are cleared on entry. Snapshots and queued updates restore the books. Succeeded can be reported while TCP recovery is active; a replay gap or processing failure can restart recovery without a completion callback. | Inspect the completion status and wait for OnixS::CME::MDH::HandlerListener::onRealtimeProcessing before treating the live Incremental data stream as current. |
| TCP recovery | Between OnixS::CME::MDH::HandlerListener::onTcpRecovery and OnixS::CME::MDH::HandlerListener::onEndOfTcpRecovery. | Real-time Incremental data processing is suspended, and incoming packets are queued while missing packets are recovered, so books must not be treated as advancing with the live stream. | Inspect the recovery completion status; the handler normally manages the recovery procedure itself. |
| Real-time processing | Between OnixS::CME::MDH::HandlerListener::onRealtimeProcessing and OnixS::CME::MDH::HandlerListener::onEndOfRealtimeProcessing. | Current Incremental data processing is active. | Apply the application's instrument, book, session, and risk-readiness policy. |
This table summarizes the behaviour described on this page and in the OnixS::CME::MDH::HandlerListener callback documentation. It does not determine whether data is universally safe to use for trading. For current notification exceptions under OnixS::CME::MDH::GapAndErrorHandlingOptions::ContinueProcessing, see ContinueProcessing notification limitations.
The configuration and settings for the data processing session define the handler's behaviour. Therefore, each market data processing configuration influences the sequence of events the handler raises. Also, the series of events depends on external conditions. For example, the handler may perform recovery procedures if data is lost. Therefore, the chain of events is not strictly defined for market data processing, although there is explicit predictability in events for each use case and external conditions.
The following sections describe the handler behaviour for particular use cases.
When the handler starts executing Instrument Definition recovery, it invokes the OnixS::CME::MDH::HandlerListener::onInstrumentRecovery callback. Once the Instrument Definition recovery is started, the handler listens to data on the Instrument feed and processes received packets. The handler detects the end of recovery based on the data processed. The handler calls the OnixS::CME::MDH::HandlerListener::onEndOfInstrumentRecovery method after successfully processing an Instrument Definition recovery loop.
The handler may invoke the OnixS::CME::MDH::HandlerListener::onInstrumentRecovery and the OnixS::CME::MDH::HandlerListener::onEndOfInstrumentRecovery members multiple times before it switches to executing other steps like market recovery or resumes real-time processing. This behaviour may happen due to errors detected by the handler while processing the Instrument feed data. If an error occurs, the handler will call the OnixS::CME::MDH::HandlerListener::onEndOfInstrumentRecovery with the OnixS::CME::MDH::RecoveryCompletionArgs::status equal to OnixS::CME::MDH::RecoveryCompletionStatus::Failed. The OnixS::CME::MDH::RecoveryCompletionArgs::details member includes the description of the issue that caused the recovery failure.
As noted in previous sections, users do not need to perform any additional steps. The handler automatically restarts an instrument recovery procedure. The instrument recovery is restarted as many times as necessary to get successful processing of a single recovery loop.
The handler may recover Instrument Definitions from a previously saved cache. Using an instrument definition cache usually helps to quickly pass the instrument definition recovery stage and move to further processing. From an events perspective, the handler raises the same pair of events identifying the beginning and the end of the instrument definition recovery.
If the instrument definition cache is corrupted or the handler cannot process it, the handler will end the cache-based recovery by invoking the OnixS::CME::MDH::HandlerListener::onEndOfInstrumentRecovery with the OnixS::CME::MDH::RecoveryCompletionArgs::status equal to OnixS::CME::MDH::RecoveryCompletionStatus::Failed.
If the cache is not yet available, the handler will end the cache-based recovery by invoking the OnixS::CME::MDH::HandlerListener::onEndOfInstrumentRecovery with the OnixS::CME::MDH::RecoveryCompletionArgs::status equal to OnixS::CME::MDH::RecoveryCompletionStatus::Incomplete.
In both cases (the cache is either corrupted or not yet available), the handler restarts the instrument definition recovery from the live network feed.
The following diagram represents a visual overview of the instrument definition recovery:
The handler starts market recovery when joining the market late or recovering from a processing issue. It invokes OnixS::CME::MDH::HandlerListener::onMarketRecovery before processing any Snapshot message, then receives snapshots from Market Recovery Feeds.
After receiving the snapshots and verifying their completeness, an uninterrupted recovery completes in this order:
The snapshot-completion log therefore does not indicate that replay has finished. If TCP recovery starts during replay, the handler can invoke OnixS::CME::MDH::HandlerListener::onEndOfMarketRecovery with OnixS::CME::MDH::RecoveryCompletionStatus::Succeeded while TCP recovery is still in progress. Real-time processing resumes after successful missing-packet recovery and replay. Use OnixS::CME::MDH::HandlerListener::onRealtimeProcessing to observe that transition.
If snapshot processing fails, the handler invokes OnixS::CME::MDH::HandlerListener::onEndOfMarketRecovery with OnixS::CME::MDH::RecoveryCompletionStatus::Failed. The OnixS::CME::MDH::RecoveryCompletionArgs::details member describes the failure. The handler automatically starts another recovery attempt. With OnixS::CME::MDH::GapAndErrorHandlingOptions::RecoverInstrumentsAndMarketState, it recovers instrument definitions before recovering the market state again.
Before replay, the handler checks whether snapshots are available for instruments whose incremental book updates are queued. If a required snapshot is missing, it invokes OnixS::CME::MDH::HandlerListener::onEndOfMarketRecovery with OnixS::CME::MDH::RecoveryCompletionStatus::Incomplete and restarts market recovery to avoid book maintenance errors. This CME MDP 3 condition is not treated as a processing error, so it does not trigger instrument recovery even when that is configured for gaps and errors.
A gap found while replaying queued Incremental data is reported as OnixS::CME::MDH::Issue::SequenceError. The handler does not invoke OnixS::CME::MDH::HandlerListener::onEndOfRealtimeProcessing again, since real-time processing is already suspended. It attempts TCP recovery and otherwise follows OnixS::CME::MDH::SessionSettings::gapAndErrorHandling:
When a replay gap restarts recovery without TCP recovery, the interrupted attempt does not invoke OnixS::CME::MDH::HandlerListener::onEndOfMarketRecovery. In the default configuration, the application can therefore observe another OnixS::CME::MDH::HandlerListener::onMarketRecovery after the gap warning without an intervening recovery-completion callback.
If the handler fails to process a queued packet during replay, it reports OnixS::CME::MDH::Issue::ProcessingFailure. Application callback exceptions are caught separately and do not trigger this path. With either recovery policy described above, the handler invokes OnixS::CME::MDH::HandlerListener::onEndOfRealtimeProcessing again even though real-time processing is already suspended, then starts recovery according to the policy. The interrupted attempt does not invoke OnixS::CME::MDH::HandlerListener::onEndOfMarketRecovery.
Replay under OnixS::CME::MDH::GapAndErrorHandlingOptions::ContinueProcessing has the following known limitations. These paths do not follow the uninterrupted completion sequence above:
Frequent packet loss on Incremental data can cause repeated recovery even when application callbacks are fast and there is no recovery-queue overflow. Under the default OnixS::CME::MDH::GapAndErrorHandlingOptions::RecoverMarketState policy, a gap encountered during replay starts another market recovery attempt if TCP recovery cannot fill it. Recurring gaps in data still needed after snapshots can prevent the handler from returning to real-time processing.
Loss may occur in the network or switches, at the NIC, or in operating system and socket buffers. Check for frequent OnixS::CME::MDH::Issue::SequenceError warnings and consult the Lost multicast packets troubleshooting guide. Consider using both A and B feeds (see Gap detection and feed arbitrage), reviewing Tuning network packet gap detection settings, and configuring TCP Recovery to fill eligible gaps without restarting market recovery.
To diagnose slow replay or repeated recovery, measure the interval after Finished recovering market from snapshots. Replay can end in a gap warning or processing failure and another recovery-start event without logging Realtime processing resumed. Compare these timings with application callback timings or a CPU profile. Check verbose logging costs, incoming packet/byte rates, process memory use and available operating system packet-drop counters. Lengthy replay can delay reception on the standard Socket Feed Engine; see Multi-threaded processing and Keeping callbacks responsive.
Also check for recovery-queue overflow warnings. Discarding queued packets can leave a gap during replay, whereas socket receive-buffer loss may produce a gap after real-time processing resumes. These are different causes of repeated recovery; see What do QueueOverflow warnings mean, and should I increase the queue limit?. Under the default recovery policy, repeated recovery-start events without a completion callback can indicate a replay gap or a processing failure during replay. A replay gap alone does not identify where packet loss occurred.
The market recovery supports the processing of recovery data from multiple sources. CME MDP 3 offers two independent sources of recovery data for market by order and market by price. The handler processes recovery loops from either one or all sources depending on the settings of the processing session (see the OnixS::CME::MDH::SessionSettings::marketRecovery parameter for more information). The selected MBP and MBO sources share one recovery sequence, reported through OnixS::CME::MDH::HandlerListener::onMarketRecovery and OnixS::CME::MDH::HandlerListener::onEndOfMarketRecovery, with the qualifications above for replay interruptions, TCP recovery and OnixS::CME::MDH::GapAndErrorHandlingOptions::ContinueProcessing.
When the handler either starts or resumes the processing of Incremental data after recovery, it notifies about the given event via the OnixS::CME::MDH::HandlerListener::onRealtimeProcessing callback. Real-time processing of Incremental data means that data is processed as soon as it is received without any delays (except the case when the handler keeps out-of-order data waiting a short time for missing data to arrive).
By invoking the OnixS::CME::MDH::HandlerListener::onRealtimeProcessing the handler informs that Incremental data is not processed under other conditions (e.g., during TCP Recovery).
If the handler faces an issue like a gap in Incremental data or book maintenance error, it suspends the real-time processing of the Incremental data and signals about this via the OnixS::CME::MDH::HandlerListener::onEndOfRealtimeProcessing. Once the callback is invoked, the handler performs the issue handling according to its settings. For example, if the handler detects a gap in Incremental data and the TCP Recovery service is bound to the handler, it tries to recover missing packets using TCP Recovery. At this stage, the handler raises events related to TCP recovery. Similarly, if the handler starts a large-scale market recovery, the events related to the market and, optionally, Instrument recovery are raised. When an issue handling is finished, the handler resumes real-time processing of Incremental data and notifies a user code about this using the OnixS::CME::MDH::HandlerListener::onRealtimeProcessing member.
The diagram below visualizes aspects of the real-time processing of [Incremental data]:
CME MDP 3 offers TCP Recovery Feeds to let applications recover missing data without spawning a large-scale recovery from Snapshots. The handler supports this functionality by using a bound instance of the OnixS::CME::MDH::TcpRecoveryService class. When a gap in Incremental data is detected, the handler suspends the real-time processing of Incremental data and tries to recover missing packets via the TCP Recovery service. At the beginning of each attempt, the handler invokes the OnixS::CME::MDH::HandlerListener::onTcpRecovery callback indicating the beginning of the TCP Recovery procedure. Once all missing packets are received and processed successfully, the handler ends the recovery procedure and notifies about that through the OnixS::CME::MDH::HandlerListener::onEndOfTcpRecovery callback.
The handler may perform multiple attempts to recover missing data through TCP Recovery. Each attempt invokes the OnixS::CME::MDH::HandlerListener::onTcpRecovery callback followed by the OnixS::CME::MDH::HandlerListener::onEndOfTcpRecovery callback. If an attempt fails, the handler provides the failure reason via the OnixS::CME::MDH::RecoveryCompletionArgs::details member of the OnixS::CME::MDH::RecoveryCompletionArgs parameter.
