Market data processing assumes performance or different procedures like a large-scale recovery to catch-up up-to-date market state on late join or requesting lost incremental data. While developing a trading solution, it's important to know in which state market data processing currently resides.

To meet needs, the SDK exposes the OnixS::CME::ConflatedUDP::HandlerListener listener. The Handler invokes members of the given listener once the instance is bound using the handler.settings().listeners().handler(&listener) path.

The following table summaries the Handler events.

Event	Description
OnixS::CME::ConflatedUDP::HandlerListener::onStarted	Invoked by the Handler when it starts market data processing due to calling the OnixS::CME::ConflatedUDP::Handler::start member.
OnixS::CME::ConflatedUDP::HandlerListener::onStopped	Invoked by the Handler when it ends market data processing due to calling the OnixS::CME::ConflatedUDP::Handler::start member.
OnixS::CME::ConflatedUDP::HandlerListener::onInstrumentRecovery	Invoked when the Handler spawns an instrument recovery.
OnixS::CME::ConflatedUDP::HandlerListener::onEndOfInstrumentRecovery	Invoked when the Handler accomplishes an instrument recovery. The Handler supplies the status of recovery (successful, incomplete, failed) as an additional parameter.
OnixS::CME::ConflatedUDP::HandlerListener::onMarketRecovery	Invoked when the Handler spawns a market recovery.
OnixS::CME::ConflatedUDP::HandlerListener::onEndOfMarketRecovery	Invoked when the Handler accomplishes a market recovery. The Handler supplies the status of recovery (successful, incomplete, failed) as an additional parameter.
OnixS::CME::ConflatedUDP::HandlerListener::onTcpRecovery	Invoked when the Handler recovers lost data via a TCP recovery feed.
OnixS::CME::ConflatedUDP::HandlerListener::onEndOfTcpRecovery	Invoked when the Handler accomplishes recovering lost data via a TCP feed. The Handler supplies the status of recovery (successful, incomplete, failed) as an additional parameter.
OnixS::CME::ConflatedUDP::HandlerListener::onRealtimeProcessing	Invoked by the Handler when real-time processing of incremental data is either started or resumed after recovery of any kind (TCP, snapshots). After invocation of the given callback, it is guaranteed that incremental data is processed as soon as it arrives and without any delays in processing.
OnixS::CME::ConflatedUDP::HandlerListener::onEndOfRealtimeProcessing	Invoked when the Handler suspends real-time processing of incremental data due to a processing issue or data loss.

The Event Invocation Sequence

The configuration and settings for the data processing session define the behavior of the Handler. Therefore, each market data processing pattern defines a primary sequence of events the Handler raises. Also, a sequence of events depends on the conditions the Handler occurs. For example, if data is lost, the Handler may perform recovery procedures according to its configuration. An issue may take place during recovery or data processing. Therefore, a chain of events is not strictly defined for market data processing, although there's explicit predictability in events for each case and conditions the Handler occurs in.

The following sections guide through the Handler behavior for major cases.

Note: For each case which may take place while processing market data, the Handler has an established procedure which defines exact behavior for the case. Therefore, it's not necessary to perform any additional steps to instruct the Handler what to do in the case. The Handler manages itself and notifies the users on stages of its behavior.

Instrument Definition Recovery and Events

When the Handler starts performing instrument definition recovery, it invokes the OnixS::CME::ConflatedUDP::HandlerListener::onInstrumentRecovery callback. The event is raised before any instrument definition is processed. Once instrument recovery is started, the Handler listens to data on instrument feed and processes received packets. The end of recovery is detected by the Handler based on data processed. The Handler calls the OnixS::CME::ConflatedUDP::HandlerListener::onEndOfInstrumentRecovery upon the end of successful processing of an instrument definition recovery loop.

The Handler may invoke the OnixS::CME::ConflatedUDP::HandlerListener::onInstrumentRecovery and the OnixS::CME::ConflatedUDP::HandlerListener::onEndOfInstrumentRecovery members multiple times before it switches performing other steps like market recovery or resumes real-time processing. The given behavior may happen due to issues experienced by the Handler while processing data from an instrument feed. If error occurs, the Handler will call the OnixS::CME::ConflatedUDP::HandlerListener::onEndOfInstrumentRecovery with the OnixS::CME::ConflatedUDP::RecoveryCompletionArgs::status equal to OnixS::CME::ConflatedUDP::RecoveryCompletionStatus::Failed. The OnixS::CME::ConflatedUDP::RecoveryCompletionArgs::details member includes a detailed description of the issue caused recovery failure.

As noted in previous sections, users do not need to perform any additional steps in such case. The Handler automatically restarts an instrument recovery procedure. The instrument recovery is restarted as many times as necessary to get successful processing of a single recovery loop.

Recovering Instrument Definition from Cache

The Handler may recover instrument definitions from a previously saved cache. Using an instrument definition cache usually helps to quickly pass an instrument definition recovery stage and move to further processing. From events perspective, the Handler raises the same pair of events identifying the beginning and the end of the instrument definition recovery.

If an instrument definition cache is corrupted or can't be successfully processed for any other reason, the Handler will end recovery from cache by invoking the OnixS::CME::ConflatedUDP::HandlerListener::onEndOfInstrumentRecovery with the OnixS::CME::ConflatedUDP::RecoveryCompletionArgs::status equal to the OnixS::CME::ConflatedUDP::RecoveryCompletionStatus::Failed.

If cache is not yet available, the Handler will end recovery from the cache by invoking the OnixS::CME::ConflatedUDP::HandlerListener::onEndOfInstrumentRecovery with the OnixS::CME::ConflatedUDP::RecoveryCompletionArgs::status equal to the OnixS::CME::ConflatedUDP::RecoveryCompletionStatus::Incomplete.

In both cases (if the cache is either corrupted or not yet available) the Handler restarts an instrument definition recovery from the live network feed.

The following diagram represents a visual overview of the instrument definition recovery:

Instrument Definition Recovery And Events

Market State Recovery and Events

Market recovery procedure may be spawned by the Handler to catch-up up-to-date market state while joining the market or due to other reasons like processing failure.

When the Handler starts performing market recovery, it invokes the OnixS::CME::ConflatedUDP::HandlerListener::onMarketRecovery callback. The event is raised before any snapshot is processed. Once a market state recovery is started, the Handler listens to data on snapshot feeds and processes received data. The end of recovery is detected by the Handler based on data processed. The Handler calls the OnixS::CME::ConflatedUDP::HandlerListener::onEndOfMarketRecovery upon the end of successful processing of the snapshot recovery loop.

The Handler may invoke the OnixS::CME::ConflatedUDP::HandlerListener::onMarketRecovery and the OnixS::CME::ConflatedUDP::HandlerListener::onEndOfMarketRecovery multiple times before it resumes real-time processing of incremental data. The given behavior may take place due to issues experienced by the Handler while processing data like gaps in cached incremental data. If an error occurs, the Handler calls the OnixS::CME::ConflatedUDP::HandlerListener::onEndOfMarketRecovery with the OnixS::CME::ConflatedUDP::RecoveryCompletionArgs::status equal to OnixS::CME::ConflatedUDP::RecoveryCompletionStatus::Failed. The OnixS::CME::ConflatedUDP::RecoveryCompletionArgs::details member includes a detailed description of the issue caused recovery failure.

Similar to the instrument definition recovery, users do not need to perform any additional steps in such a case. The Handler automatically restarts a market recovery procedure. The market recovery is restarted as many times as necessary to get successful processing of the snapshot recovery loop.

The Handler may switch performing an instrument recovery before repeating an attempt to recover a market state from snapshots in case of market recovery failure. The given behavior happens if a processing session is configured to recover both instruments and the market state on data gap or processing failure (see the OnixS::CME::ConflatedUDP::GapAndErrorHandlingOptions::RecoverInstrumentsAndMarketState for more information).

A market recovery can accomplish with the OnixS::CME::ConflatedUDP::RecoveryCompletionStatus::Incomplete status. When the Handler ends processing of recovery data from snapshots, it applies incremental data queued during recovery. While processing queued incremental data, The Handler verifies market recovery integrity. It may happen that MDP did not send a snapshot for an instrument which incremental book updates were queued during the recovery. In such a case, the Handler restarts the market recovery to avoid book maintenance errors.

Note: The described MDP behavior is not considered as an error. Therefore, if a processing session is configured to recover instruments in case of data loss and processing errors, the Handler does not recover instruments if the market recovery ends with OnixS::CME::ConflatedUDP::RecoveryCompletionStatus::Incomplete status.

Market By Order and Market By Price Recovery

A market recovery may assume processing of recovery data from multiple sources. With the invention of Market By Order functionality, CME MDP now offers two independent sources of recovery data for market by order and market by price. The Handler processes recovery loops from either one or all sources depending on the settings of a processing session (see the OnixS::CME::ConflatedUDP::SesionSettings::marketRecovery parameter for more information). From the event perspective, there's no difference whether the Handler performs recovery of the market by order or price or both. All recovery-related steps, including processing of queued incremental data after processing snapshots, is done between invocations of the OnixS::CME::ConflatedUDP::HandlerListener::onMarketRecovery and the OnixS::CME::ConflatedUDP::HandlerListener::onEndOfMarketRecovery members.

Market Recovery And Events

Real-time Processing of Incremental Data

When the Handler either starts or resumes processing of incremental data after recovery, it notifies about the given event via the OnixS::CME::ConflatedUDP::HandlerListener::onRealtimeProcessing callback. Real-time processing of incremental data means that data is processed as soon as it is received without any delays (except the case when the Handler keeps out-of-order data waiting for a short period of time for missing data to arrive).

By invoking the OnixS::CME::ConflatedUDP::HandlerListener::onRealtimeProcessing the Handler informs on the fact incremental data is not processed under other conditions like the market or TCP recovery.

If the Handler faces an issue like a gap in incremental data or book maintenance error, it suspends real-time processing of the incremental data and signals about this via the OnixS::CME::ConflatedUDP::HandlerListener::onEndOfRealtimeProcessing. Once the callback is invoked, the Handler performs an issue handling according to its settings. For example, if the Handler detected a gap in incremental data and the TCP recovery service is bound to the Handler, it tries to recover missing packets using the TCP recovery facility. At the given stage the Handler raises events which are related to TCP recovery. Similar, if the Handler spawns a large-scale market recovery, then the events related to the market and optionally instrument recovery are raised. When an issue handling is accomplished, the Handler resumes real-time processing of incremental data and notify a user code about this using the OnixS::CME::ConflatedUDP::HandlerListener::onRealtimeProcessing member.

The diagram below visualizes aspects of the real-time processing of incremental data:

Real-time Processing of Incremental Data

Recovering Missing Data from TCP Recovery Feed

CME MDP offers the TCP Recovery Feed to let applications recover missing data without spawning a large-scale recovery from snapshots. The Handler supports given functionality by using a bound instance of the OnixS::CME::ConflatedUDP::TcpRecoveryService class. When a gap in incremental data is detected, the Handler suspends real-time processing of incremental data and tries to recover missing packets via the TCP recovery service. At the beginning of each attempt the Handler invokes the OnixS::CME::ConflatedUDP::HandlerListener::onTcpRecovery callback indicating begin of the TCP recovery procedure. Once all missing packets are received and processed successfully, the Handler ends the recovery procedure and notifies about that through the OnixS::CME::ConflatedUDP::HandlerListener::onEndOfTcpRecovery callback.

The Handler may perform multiple attempts to recover missing data through the TCP Recovery. Each time an attempt is made, pair of the OnixS::CME::ConflatedUDP::HandlerListener::onTcpRecovery and the OnixS::CME::ConflatedUDP::HandlerListener::onTcpRecovery callbacks is invoked. If an attempt fails, the Handler uncovers a reason of the failure as the OnixS::CME::ConflatedUDP::RecoveryCompletionArgs::details member of the OnixS::CME::ConflatedUDP::RecoveryCompletionArgs parameter.