Market data processing requires the execution of different procedures like a large-scale recovery to obtain the up-to-date market state on late join or requesting lost Incremental data. While developing a trading solution, knowing the market data processing state is essential.

To report about state changes, the handler invokes the OnixS::CME::MDH::HandlerListener members once its instance is bound to the handler using the handler.settings().listeners().handler(&listener) path.

The following table summarises the handler events.

Event	Description
OnixS::CME::MDH::HandlerListener::onStarted	The handler invokes this member when it starts the market data processing.
OnixS::CME::MDH::HandlerListener::onStopped	The handler invokes this member when it ends the market data processing.
OnixS::CME::MDH::HandlerListener::onInstrumentRecovery	The handler invokes this member when it starts an instrument recovery.
OnixS::CME::MDH::HandlerListener::onEndOfInstrumentRecovery	The handler invokes this member when it finishes the instrument recovery.
OnixS::CME::MDH::HandlerListener::onMarketRecovery	The handler invokes this member when it starts a market recovery.
OnixS::CME::MDH::HandlerListener::onEndOfMarketRecovery	The handler invokes this member when it finishes the market recovery.
OnixS::CME::MDH::HandlerListener::onTcpRecovery	The handler invokes this member when it recovers lost data via TCP Recovery Feeds.
OnixS::CME::MDH::HandlerListener::onEndOfTcpRecovery	The handler invokes this member when it finishes recovering lost data via TCP Recovery Feeds.
OnixS::CME::MDH::HandlerListener::onRealtimeProcessing	The handler invokes this member when real-time Incremental data processing is either started or resumed after recovery.
OnixS::CME::MDH::HandlerListener::onEndOfRealtimeProcessing	The handler invokes this member when it suspends the real-time processing of Incremental data due to a processing issue or data loss.

Events invocation sequence

The configuration and settings for the data processing session define the handler's behaviour. Therefore, each market data processing configuration influences the sequence of events the handler raises. Also, the series of events depends on external conditions. For example, the handler may perform recovery procedures if data is lost. Therefore, the chain of events is not strictly defined for market data processing, although there is explicit predictability in events for each use case and external conditions.

The following sections describe the handler behaviour for particular use cases.

Note: For each use case which may take place while processing market data, the handler has an established procedure which defines the exact behaviour for the use case. Therefore, it is not necessary to perform any additional steps to instruct the handler on what to do. The handler manages itself and notifies the users of stages of its behaviour.

Instrument Definition recovery

When the handler starts executing Instrument Definition recovery, it invokes the OnixS::CME::MDH::HandlerListener::onInstrumentRecovery callback. Once the Instrument Definition recovery is started, the handler listens to data on the Instrument feed and processes received packets. The handler detects the end of recovery based on the data processed. The handler calls the OnixS::CME::MDH::HandlerListener::onEndOfInstrumentRecovery method after successfully processing an Instrument Definition recovery loop.

The handler may invoke the OnixS::CME::MDH::HandlerListener::onInstrumentRecovery and the OnixS::CME::MDH::HandlerListener::onEndOfInstrumentRecovery members multiple times before it switches to executing other steps like market recovery or resumes real-time processing. This behaviour may happen due to errors detected by the handler while processing the Instrument feed data. If an error occurs, the handler will call the OnixS::CME::MDH::HandlerListener::onEndOfInstrumentRecovery with the OnixS::CME::MDH::RecoveryCompletionArgs::status equal to OnixS::CME::MDH::RecoveryCompletionStatus::Failed. The OnixS::CME::MDH::RecoveryCompletionArgs::details member includes the description of the issue that caused the recovery failure.

As noted in previous sections, users do not need to perform any additional steps. The handler automatically restarts an instrument recovery procedure. The instrument recovery is restarted as many times as necessary to get successful processing of a single recovery loop.

Recovering Instrument Definition from the cache

The handler may recover Instrument Definitions from a previously saved cache. Using an instrument definition cache usually helps to quickly pass the instrument definition recovery stage and move to further processing. From an events perspective, the handler raises the same pair of events identifying the beginning and the end of the instrument definition recovery.

If the instrument definition cache is corrupted or the handler cannot process it, the handler will end the cache-based recovery by invoking the OnixS::CME::MDH::HandlerListener::onEndOfInstrumentRecovery with the OnixS::CME::MDH::RecoveryCompletionArgs::status equal to OnixS::CME::MDH::RecoveryCompletionStatus::Failed.

If the cache is not yet available, the handler will end the cache-based recovery by invoking the OnixS::CME::MDH::HandlerListener::onEndOfInstrumentRecovery with the OnixS::CME::MDH::RecoveryCompletionArgs::status equal to OnixS::CME::MDH::RecoveryCompletionStatus::Incomplete.

In both cases (the cache is either corrupted or not yet available), the handler restarts the instrument definition recovery from the live network feed.

The following diagram represents a visual overview of the instrument definition recovery:

Instrument Definition Recovery And Events

Market State recovery

The handler can start the market recovery to obtain the current market state while joining the market or due to other reasons like processing failure.

When the handler starts executing market recovery, it invokes the OnixS::CME::MDH::HandlerListener::onMarketRecovery callback. The event is raised before any Snapshot message is processed. Once a market state recovery is started, the handler listens to data on Market Recovery Feeds and processes received data. The handler detects the end of recovery based on the data processed. It calls the OnixS::CME::MDH::HandlerListener::onEndOfMarketRecovery method after the end of the successful processing of the Snapshot message recovery loop.

The handler may invoke the OnixS::CME::MDH::HandlerListener::onMarketRecovery and the OnixS::CME::MDH::HandlerListener::onEndOfMarketRecovery methods multiple times before it resumes the real-time processing of Incremental data. The given behaviour may occur due to issues experienced by the handler while processing data, like gaps in cached Incremental data. If an error occurs, the handler calls the OnixS::CME::MDH::HandlerListener::onEndOfMarketRecovery method with the OnixS::CME::MDH::RecoveryCompletionArgs::status equal to OnixS::CME::MDH::RecoveryCompletionStatus::Failed. The OnixS::CME::MDH::RecoveryCompletionArgs::details member includes a detailed description of the issue that caused the recovery failure.

Like the Instrument Definitions recovery, users do not need to perform any additional steps in such a case. The handler automatically restarts the market recovery procedure. The market recovery is restarted as many times as necessary to get successful processing of the Snapshot message recovery loop.

The handler may switch to executing the Instrument Definitions recovery before repeating an attempt to recover a market state from snapshots in case of market recovery failure. The given behaviour happens if a processing session is configured to recover both instruments and the market state on data gap or processing failure (see the OnixS::CME::MDH::GapAndErrorHandlingOptions::RecoverInstrumentsAndMarketState for more information).

A market recovery can end with the OnixS::CME::MDH::RecoveryCompletionStatus::Incomplete status. When the handler ends the processing of recovery data from Market Recovery Feeds, it applies Incremental data queued during recovery. While processing queued Incremental data, The handler verifies the market recovery integrity. It may happen that the Market Recovery Feed did not send a Snapshot message for an instrument for which incremental book updates were queued during the recovery. In such a case, the handler restarts the market recovery to avoid book maintenance errors.

Note: The described CME MDP 3 behaviour is not considered an error. Therefore, if a processing session is configured to recover Instruments in case of data loss and processing errors, the handler does not recover instruments if the market recovery ends with the OnixS::CME::MDH::RecoveryCompletionStatus::Incomplete status.

Market By Order and Market By Price recovery

The market recovery supports the processing of recovery data from multiple sources. CME MDP 3 offers two independent sources of recovery data for market by order and market by price. The handler processes recovery loops from either one or all sources depending on the settings of the processing session (see the OnixS::CME::MDH::SesionSettings::marketRecovery parameter for more information). From the event perspective, there is no difference whether the handler performs recovery of the market by order or price, or both. All recovery-related steps, including the processing of queued Incremental data after processing snapshots, is done between invocations of the OnixS::CME::MDH::HandlerListener::onMarketRecovery and the OnixS::CME::MDH::HandlerListener::onEndOfMarketRecovery members.

Market Recovery And Events

Real-time processing of [Incremental data](\ref incremental-feeds)

When the handler either starts or resumes the processing of Incremental data after recovery, it notifies about the given event via the OnixS::CME::MDH::HandlerListener::onRealtimeProcessing callback. Real-time processing of Incremental data means that data is processed as soon as it is received without any delays (except the case when the handler keeps out-of-order data waiting a short time for missing data to arrive).

By invoking the OnixS::CME::MDH::HandlerListener::onRealtimeProcessing the handler informs that Incremental data is not processed under other conditions (e.g., during TCP Recovery).

If the handler faces an issue like a gap in Incremental data or book maintenance error, it suspends the real-time processing of the Incremental data and signals about this via the OnixS::CME::MDH::HandlerListener::onEndOfRealtimeProcessing. Once the callback is invoked, the handler performs the issue handling according to its settings. For example, if the handler detects a gap in Incremental data and the TCP Recovery service is bound to the handler, it tries to recover missing packets using TCP Recovery. At this stage, the handler raises events related to TCP recovery. Similarly, if the handler starts a large-scale market recovery, the events related to the market and, optionally, Instrument recovery are raised. When an issue handling is finished, the handler resumes real-time processing of Incremental data and notifies a user code about this using the OnixS::CME::MDH::HandlerListener::onRealtimeProcessing member.

The diagram below visualizes aspects of the real-time processing of Incremental data:

Real-time Processing of Incremental Data

Recovering missing Data from the TCP Recovery feed

CME MDP 3 offers TCP Recovery Feeds to let applications recover missing data without spawning a large-scale recovery from Snapshots. The handler supports this functionality by using a bound instance of the OnixS::CME::MDH::TcpRecoveryService class. When a gap in Incremental data is detected, the handler suspends the real-time processing of Incremental data and tries to recover missing packets via the TCP Recovery service. At the beginning of each attempt, the handler invokes the OnixS::CME::MDH::HandlerListener::onTcpRecovery callback indicating begin of the TCP Recovery procedure. Once all missing packets are received and processed successfully, the handler ends the recovery procedure and notifies about that through the OnixS::CME::MDH::HandlerListener::onEndOfTcpRecovery callback.

The handler may perform multiple attempts to recover missing data through TCP Recovery. Each time an attempt is made, the pair of the OnixS::CME::MDH::HandlerListener::onTcpRecovery and the OnixS::CME::MDH::HandlerListener::onTcpRecovery callbacks are invoked. If an attempt fails, the handler provides the failure reason via the OnixS::CME::MDH::RecoveryCompletionArgs::details member of the OnixS::CME::MDH::RecoveryCompletionArgs parameter.