Onix Solutions Logo

OnixS Java CME Market Data Handler — Programmer's Guide

Version 3.19.0



Onix Solutions Java CME Market Data Handler is a Java library that provides an access to the CME Group MDP 3.0 market data via Simple Binary encoding (SBE) protocol. High-level Java API allows to build applications rapidly to get market data without much involving into raw protocol specifics.

CME Market Data Feed Handler implementations features include:

We recommend to read the "Market Data Platform (MDP) 3.0" document to get familiar with core aspects of CME Group Market Data System before reading the Guide. By reading this Guide, we also recommend to review a source code of the sample project which comes as a part of the library distributive package.

System Requirements

Getting Started

Using Handler usually includes the following:

Adjusting Settings

Market Data Channel

CME market data is distributed across the multiple logical units to reduce network and client system load. They are called channels. Handler requires the channel to be identified in order to process market data. The biz.onixs.cme.md.handler.CmeMdHandler constructor helps to instruct Handler market data of the channel which must be processed.

Market Data Configuration

Handler also requires the market data configuration file to be specified. Handler uses the stored data in this configuration to identify network connectivity attributes for a particular market data channel.
CME Group provides Market data configuration file via their public FTP servers. CME Group updates market data configuration on regular (weekly) basis. Read the CME documentation for more information about configuration updating schedule as well as how to access to public FTP servers.
The biz.onixs.cme.md.handler.CmeMdHandler#setChannelConfigurationFile method has value which has to take a path to the locally stored copy of such configuration file. The default value is "config.xml".
Single market data configuration file contains settings for all market data channels. However, CME Group uses different market data configurations for different environments (like Production and Certification environments). Multiple instances of the biz.onixs.cme.md.handler.CmeMdHandler class can be initialized to use either single or different market data configurations. In other words, it is possible to use multiple instances of Handler to connect to different environments.

SBE Coding Templates

All CME Group market data messages are SBE encoded to reduce network load. SBE coding supposes the use of SBE coding templates (sort of encoding / decoding configuration). Therefore, Handler requires such information to be specified for proper decoding of market data messages. The biz.onixs.cme.md.handler.CmeMdHandler#setTemplateFile method is to instruct Handler where the SBE coding templates file is located. The default value is "templates_FixBinary.xml".
Single SBE coding templates file is similar to the market data configuration. It can be used for different channels within single operating environment (like Production or Certification ones). CME Group maintains all SBE coding templates files for all operating environments by its own and updates them on regular (weekly) basis. All this data is available on public FTP servers of CME Group.


The following example demonstrates how to setup primary settings for Handler.
public class Sample {
    private void run() throws Exception {
        final String channel = "310";
        final CmeMdHandler handler = new CmeMdHandler(channel);

Complete Settings List

Name Description Default value
Path to the CME Market Data Feed Channel configuration XML file "config.xml"
Direct book default depth. 10
Path to the SBE templates file. "templates_FixBinary.xml"
Implied book default depth. 2
Incremental packets cache size. This cache are used for packet alignment. 5
Path to the license file. "OnixS.lic"
Local network interface that is used to receive UDP multicast packets. If set to null then all available local network interfaces are used. null
Local network interface that is used to receive UDP multicast packets for "A" feeds. If set to null then the LocalNetworkInterface value is used. null
Local network interface that is used to receive UDP multicast packets for "B" feeds. If set to null then the LocalNetworkInterface value is used. null
biz.onixs.cme.md.handler.CmeMdHandler# setMaximumNumberOfQueuedIncrementalRefreshMessages
biz.onixs.cme.md.handler.CmeMdHandler# getMaximumNumberOfQueuedIncrementalRefreshMessages
Maximum number of queued Market Data Incremental Refresh (X) messages. When the message gap is detected all subsequent Refresh(X) messages are queued until the Handler re-synchronizes with the market using Snapshot(W) messages. After that the queued Refresh(X) messages are re-played. This setting could be used to limit the memory usage when there is no Market Data - Snapshot/Full Refresh (W) messages. 10000
Amount of time in milliseconds a Handler will wait to receive data once a read operation is initiated. 36000
The flag to recover the Security Definition messages each time when the message sequence gap is detected. false
Report-no-data option value. false
Path where security definition cache will be stored. "SecurityDefinitionsCache"
UDP socket buffer size. 4MB
Set option to use the TcpReplay Channel for recovery of missed messages. This method recovers all missed messages. false
Password or passphrase to be used in the Logon (35=A) message from customer to CME Tcp Replay service. ONIXS
Number of attempts to receive missed messages via CME Tcp Replay service. 10
biz.onixs.cme.md.handler.CmeMdHandler# setTcpReplayReconnectIntervalInMilliseconds
biz.onixs.cme.md.handler.CmeMdHandler# getTcpReplayReconnectIntervalInMilliseconds
Interval between attempts to receive missed messages via CME Tcp Replay service (in milliseconds). 100
User ID or username to be used in the Logon (35=A) message from customer to CME Tcp Replay service. ONIXS
The flag to use or not incremental feed "A" option. true
The flag to use or not incremental feed "B" option. false
The flag to use or not instrument feed "A" option. true
The flag to use or not instrument feed "B" option. false
True, if one multicast socket should be created for A nd B feeds, if ports are same. false
The flag to use or not Market Recovery (UDP) Feed "A". true
The flag to use or not Market Recovery (UDP) Feed "B". false

Listening to Market Data

Events in Handler

Once Handler is started, it listens to a market data from the network, then processes it (decodes messages, validates message order, updates order books) and invokes client code for further processing.

Handler processes market data working asynchronously and uses concept of events and event listeners to notify client code about a particular occasion like reception of security definition or direct book update.

We highly recommend not to call handler methods which change handler state (like handler.stop()) from event listeners.

Listening to a particular Event

For each event like Error Occurred Handler provides the following interface like biz.onixs.cme.md.handler.event.ErrorListener. Client code must implement this interface to be able to handle events of a particular type. Handler also exposes a member like biz.onixs.cme.md.handler.CmeMdHandler#registerErrorListener which allows to associate an instance of the event handler with a particular instance of the biz.onixs.cme.md.handler.CmeMdHandler class.
Associating listener for a particular event with an instance of the biz.onixs.cme.md.handler.CmeMdHandler class must be performed while Handler is in stopped stated. Once the Handler is started, changing listener-event associations is not allowed, and may lead to unpredictable behavior as well as may cause unexpected errors.
Associating event listener with an instance of the biz.onixs.cme.md.handler.CmeMdHandler class is also called subscribing to an event.

Primary High-level Events

There are multiple events exposed by Handler. All events can be logically divided onto high- and low- level event sub-set. High-level events reflect various results of market data processing done by Handler like Security Definition Received, Market Book Updated, and Trading occurred. Low-level events are designed for more control over data processing and for more flexibility. They are described in different sections of this documentation.

The table below describes primary high-level events exposed by Handler. It also depicts correspondence between events, interfaces for listeners and the Handler's members to subscribe to an event.

Event Listener interface to be implemented Handler member to register listener Description
Security Definition Received biz.onixs.cme.md.handler.event.SecurityDefinitionListener biz.onixs.cme.md.handler.CmeMdHandler#registerSecurityDefinitionListener Fired if a definition of a new security is received. Security definitions normally are received at the beginning of the Handler's execution and before any other high-level event like Book Updated event.
Security Status Changed biz.onixs.cme.md.handler.event.SecurityStatusChangedListener biz.onixs.cme.md.handler.CmeMdHandler#registerSecurityStatusListener Fired if the remote system reports about changes in a status of a particular security. An example of such change is update of upper price or trading status.
Direct Book Updated biz.onixs.cme.md.handler.event.RegularBookUpdatedListener biz.onixs.cme.md.handler.CmeMdHandler#registerRegularBookUpdatedListener Occurs when direct book of a particular security is updated.
Direct Book Changed biz.onixs.cme.md.handler.event.RegularBookChangedListener biz.onixs.cme.md.handler.CmeMdHandler#registerRegularBookChangedListener Occurs when direct book of a particular security is changed.
Implied Book Updated biz.onixs.cme.md.handler.event.ImpliedBookUpdatedListener biz.onixs.cme.md.handler.CmeMdHandler#registerImpliedBookUpdatedListener Occurs when implied book of a particular security is updated.
Implied Book Changed biz.onixs.cme.md.handler.event.ImpliedBookChangedListener biz.onixs.cme.md.handler.CmeMdHandler#registerImpliedBookChangedListener Occurs when implied book of a particular security is changed.
Consolidated Book Updated biz.onixs.cme.md.handler.event.ConsolidatedBookUpdatedListener biz.onixs.cme.md.handler.CmeMdHandler#registerConsolidatedBookUpdatedListener Occurs when consolidated book of a particular security is updated.
Consolidated Book Changed biz.onixs.cme.md.handler.event.ConsolidatedBookChangedListener biz.onixs.cme.md.handler.CmeMdHandler#registerConsolidatedBookChangedListener Occurs when consolidated book of a particular security is changed.
Statistics Changed biz.onixs.cme.md.handler.event.StatisticsListener biz.onixs.cme.md.handler.CmeMdHandler#registerStatisticsListener Occurs when attributes like opening or closing price for a particular security are changed.
Trade Information Received biz.onixs.cme.md.handler.event.TradeListener biz.onixs.cme.md.handler.CmeMdHandler#registerTradeListener Occurs if trade information is received for a particular security. Trade attributes are collected into an instance of biz.onixs.cme.md.handler.event.TradeEventArgs class which is supplied as a parameter to the listener by Handler.
News Received biz.onixs.cme.md.handler.event.NewsReceivedListener biz.onixs.cme.md.handler.CmeMdHandler#registerNewsReceivedListener Fired when the system spreads arbitrary news to Handler. Headline and full text of the news are exposed via an instance of biz.onixs.cme.md.handler.event.NewsReceivedEventArgs class.


The following sample demonstrates how to listen to notifications about security definition reception.
public class MyListener implements SecurityDefinitionListener {
    private static final Logger LOG = LoggerFactory.getLogger(MyListener.class);

    public void onSecurityDefinition(SecurityDefinitionEventArgs args) {
        LOG.debug("onSecurityDefinition(): {}", args);

public class Sample {
    private void run() throws Exception {
        final CmeMdHandler handler = new CmeMdHandler ("310");
        // ...
        final MyListener listener = new MyListener();

Manipulating Books

Books in Handler

Handler builds and maintains books for each security whose definition was previously received in bounds of a particular market data channel.

All books, maintained by Handler, represent price-level books. Book abstraction is exposed by the biz.onixs.cme.md.handler.RegularOrderBook, biz.onixs.cme.md.handler.ImpliedOrderBook, biz.onixs.cme.md.handler.ConsolidatedOrderBook and biz.onixs.cme.md.handler.MarketByOrderBook classes.

There are several types of books that are available for each security: direct, implied and consolidated. Handler builds and maintains all type of books for all securities.

Book processing can be controlled using two properties:

If these properties were set to true, Handler will maintain books even if there are no corrseponding callbacks subscribed.

Once a book of any type and for any security is updated, Handler notifies client code using appropriate event:

Instance of a book for a particular security is passed through a parameter of event listener interface.

The depth of a book of a particular type is usually defined by security definition and is exposed by biz.onixs.cme.md.handler.MarketByPriceBookBase#getDepth member. Returned value defines maximum number of price levels that may exist in the book.

biz.onixs.cme.md.handler.BookBase#getBids and biz.onixs.cme.md.handler.BookBase#getAsks members of the biz.onixs.cme.md.handler.BookBase class can be used to access price value and quantity information for all levels available in book.

Behavioral Traits

Handler constructs books for each security when it receives Security Definition message for that security. Books are changed in time and their members are not thread-safe. Handler uses its own synchronization strategies while updating the books. Therefore, do not access price levels information outside of book-related event handling as well as do not store references to any book instance and data returned by its members like asks and bids collections. It happens because Handler may change internal book structures at the same time. If any data, which is exposed by the book objects, must be used outside of event handling, it should be copied before later using. To construct immutable copy of the book for using outside of callback, use the book clone member.

See Listening to Market Data topic for more information concerning how to subscribe for a book updates.

Maintaining Books by Natural Refresh

In the regular case, the Handler keeps checking incoming packets transmitted by MDP for gaps and uses recovery facilities like TCP Recovery and snapshot feeds to keep order books in the up-to-date state. When market data loss occurs, the Handler spawns market recovery using the market recovery feed. While recovering market, the real-time market data is cached, and thus, market data processing looks like suspended.

Natural refresh of order books supposes processing of real-time (incremental) market data without involving market recovery. As a result, the Handler processes the real-time market data as soon as it is received from the feeds. As a drawback, order books may not be fully recovered, and thus, some price levels may be absent.

Identifying Missed Price Levels

When recovery facilities are used, order books exposed by the Handler are always fully reconstructed and in up-to-date state. When books are naturally refreshed, a part of price level information may be absent.

When an order book is naturally refreshed, some price levels may be absent. Price level data types expose member biz.onixs.cme.md.handler.ImpliedPriceLevel.isExist to check whether actual data is available for particular bid or ask.

To start handler in NaturalRefresh mode, there is need to disable all recovery feeds. The following example demonstrates how to run the Handler in natural refresh mode.

public class Sample {
    private void run() throws Exception {
        final CmeMdHandler handler = new CmeMdHandler ("310");

Building and Maintaining Books by Yourself

Books Building Bricks

As it was mentioned in the previous topic, Handler builds and maintains books for each security by itself and notifies client code once book update takes place.

Sometimes there is a necessity to build and maintain books by own forces (for example, if there's no need in information of the lowest price-levels). For this purpose, Handler provides advanced events, which contain detailed information about the changes, needed to be applied to a book, to bring it into an actual state.

To get notified about detailed changes into a direct book for a particular security, implement biz.onixs.cme.md.handler.event.RegularBookChangedListener interface and register it in the Handler via biz.onixs.cme.md.handler.CmeMdHandler#registerRegularBookChangedListener member.

To get notified about detailed changes into an implied book for a particular security, implement biz.onixs.cme.md.handler.event.ImpliedBookChangedListener interface and register it in the Handler via biz.onixs.cme.md.handler.CmeMdHandler#registerImpliedBookChangedListener member.

Both events expose detailed information of elementary change into book's bids and asks.

There is no ability to listen to changes for consolidated books because these books are derived from direct and implied ones.

Book Change Machinery and Relation with 'Book Updated' Event

Book Updated and Book Changed events are not mutually exclusive ones. It is possible to subscribe to both types of events. Book changes represent more elementary (atomic) updates of the book. Therefore, Book Changed events may occur more frequently in compare to Book Updated event occasions. In fact, series of Book Changed events occur before single occasion of Book Updated event.

When an order book is delivered via Book Updated callbacks, it is always in a valid and up-to-date state. In contrast, a book change is an elementary action over a book and there're usually multiple changes inside single snapshot and/or incremental refresh. For this reason, the book may not be in a valid state between two changes. Only when all the changes are processed from the single network packet (a message like snapshot and incremental refresh), the book can be considered as valid. Book Updated callbacks are called exactly at the time when all changes are processed and the book appears to be valid and up-to-date.

To build an order book properly, it is also necessary to listen to Handler State changes. When Handler falls into biz.onixs.cme.md.handler.HandlerState#BOOKS_RESYNCHRONIZATION_STARTED, all books must be empty. Afterwards, changes that are reported via Book Changed callbacks, must be applied to build a correct order book. An important aspect is that Handler doesn't differ changes from snapshots and incremental refreshes. From the API perspective, all the changes, reported through callbacks, are of the same structure. To detect the source of a change (e.g. snapshot and/or incremental refresh) as well as to determine the bounds of changes transaction it is necessary to subscribe to Message Processing events (Raw Market Data Processing).

Handler exposes members in listening interfaces like biz.onixs.cme.md.handler.event.RegularBookChangedListener#onRegularBookChangesReset. These members can be used to reset all books instead of using Handler state listening.

Customizing Book Changes/Updates Notifications

Tracking the Best Bids and Asks Only

By default, Handler fires Book Changed and Book Updated events each time it receives incremental updates for a particular book from the remote system. In certain cases, there is a necessity to reduce amount of events, generated by Handler. In particular, it is quite often that only the top of a book is a point of intersect. For this reason, Handler allows to get notified only about changes and/or updates into the best bids and asks.

To take advantage of such facility, use overloads which accept instance of biz.onixs.cme.md.handler.BestBidAskTrackingOptions class and register listeners like biz.onixs.cme.md.handler.event.TopOfTheRegularBookUpdatedListener

biz.onixs.cme.md.handler.BestBidAskTrackingOptions#setBestBidAskTrackingParameters bit field specifies, of which available bid/ask attributes (e.g. price or quantity or both) should be tracked for changes. Handler will report about changes and/or updates into a book only if specified parameters of the best bid/ask were changed.

Other members of biz.onixs.cme.md.handler.BestBidAskTrackingOptions class allow to control sensitivity of Handler to the changes and updates.

In particular, biz.onixs.cme.md.handler.BestBidAskTrackingOptions#setPriceThreshold member allows to define sensitivity threshold for the best bid/ask price values exposed in percent. If price of the best bid or ask changes for more than specified value (in percent), Handler will raise appropriate event. In other case, it will not do it.

Similarly, biz.onixs.cme.md.handler.BestBidAskTrackingOptions#setQuantityThreshold member controls occasion of Top of The Book Updated event, in case the best bid/ask quantity value changes.


The following example demonstrates how to configure Handler to get notified only if price of best bid and ask changes for more than 10 percent or quantity changes for more than 50 percent.

public class MyListener implements TopOfTheRegularBookUpdatedListener {
    private static final Logger LOG = LoggerFactory.getLogger(MyListener.class);

    public void onTopOfTheRegularBookUpdated(TopOfTheOrderBookUpdatedEventArgs args) {
        LOG.debug("onTopOfTheRegularBookUpdated(): {}", args);

public class Sample {
    private void run() throws Exception {
        final CmeMdHandler handler = new CmeMdHandler ("310");
        handler.getBestBidAskTrackingOptions().setBestBidAskTrackingParameters(BestBidAskTrackingParameter.PRICE | BestBidAskTrackingParameter.QUANTITY);
        handler.getBestBidAskTrackingOptions().setPriceThreshold(new ScaledDecimal("10"));
        // ...
        final MyListener listener = new MyListener();

Filtering Securities

Defining Securities of Interest

By default, Handler processes market data for all securities that are available in a particular market data channel. For certain channels, it may be a huge amount of data which may cause Handler to allocate significant amount of system resources (like a memory). On the other side, it often happens that all securities are not the point of interest. For these reasons, Handler provides an ability to define a subset of securities for which it will monitor market data, maintain books and report about changes in statistics. No events will be fired by Handler for any security it they are not presented in the defined subset. This feature is called as securities filtering.

Operating over securities filter is simple. biz.onixs.cme.md.handler.CmeMdHandler#addSecurityIdFilter member must be used to put a security into the collection of securities for which Handler must process market data. There are no limits on quantity of securities which can be included into filtering. Client code may be put from single up to all securities that are available in the single market data channel.

If at least one security is added to the filter, Handler stops raising events for all securities that are available in the market data channel. From that moment Handler will fire events only for securities that are included into filtering.

To remove security from the filter, it's necessary to call biz.onixs.cme.md.handler.CmeMdHandler#removeSecurityIdFilter member. Handler will continue processing market data and raising events only for the securities that remain in the filter.

If collection of securities in the filter becomes empty, Handler starts processing market data and event notifying for all securities in the market data channel.

To ensure that no securities are available in the securities filter, use biz.onixs.cme.md.handler.CmeMdHandler#clearSecurityIdFilters member. It will clear the filter.

In the same way securities can be filtered by Security Description, Security Group and Symbol.

Filtering must be manipulated only when Handler is in biz.onixs.cme.md.handler.HandlerState#STOPPED, biz.onixs.cme.md.handler.HandlerState#STARTED or in biz.onixs.cme.md.handler.HandlerState#SECURITY_DEFINITIONS_RECOVERY_STARTED state. Once Handler changes its state to any other one, filters must not be modified.


The following code demonstrates how to instruct Handler to process data only for two particular securities:

public class Sample {
    private void run() throws Exception {
        final CmeMdHandler handler = new CmeMdHandler ("310");



        // ...



Error Handling

Error Handling Concept

Being presented as a Java library, Handler uses exceptions to report about error occurred while performing certain action. For example, Handler will raise regular exception to report about inability to find actual license for the product. However, as far as Handler processes market data asynchronously, it is not able to report about any further errors from the moment market data processing is started. For this reason, Handler exposes biz.onixs.cme.md.handler.event.ErrorListener interface and biz.onixs.cme.md.handler.CmeMdHandler#registerErrorListener member to be able to subscribe to and handle errors.

Once instance of biz.onixs.cme.md.handler.event.ErrorListener is assigned to Handler, it will invoke biz.onixs.cme.md.handler.event.ErrorListener#onError member each time whe the error occurs. biz.onixs.cme.md.handler.event.ErrorListener#onError member has several incoming parameters one of which defines human-readable explanation (description) of the error.

An important aspect of the Handler's behavior is that once error occurred Handler will automatically restart market data processing starting from security definitions.


The following sample demonstrates how to receive error notifications.

public class MyListener implements ErrorListener {
    private static final Logger LOG = LoggerFactory.getLogger(MyListener.class);

    public void onError(ErrorEventArgs args) {
        LOG.debug("onError(): {}", args);

public class Sample {
    private void run() throws Exception {
        final CmeMdHandler handler = new CmeMdHandler("310");
        // ...
        final MyListener listener = new MyListener();

Licensing the Handler

Specifying License Location

Handler needs a license for successful execution. If Handler is not able to find a valid license, it will throw an exception at the initialization stage.

biz.onixs.cme.md.handler.CmeMdHandler class exposes biz.onixs.cme.md.handler.CmeMdHandler#setLicenseFile member that allows to instruct Handler where to look for a valid license. By default, Handler looks for a license in the current directory for the application that it uses, home directory of current user and a class path. However, by using noted parameter, it's possible to specify another folder anywhere on a file system.

Handler looks for a valid license in the specified folder and selects the best one. If multiple licenses are available, it will select the most significant one.


The following example demonstrates how we can supply license to Handler:

public class Sample {
    private void run() throws Exception {

        final CmeMdHandler handler = new CmeMdHandler("310");

Understanding States

Handler States

Once Handler is started, it processes securities definitions that are retrieved either from multicast channel or previously recorded local cache file. Afterwards, Handler starts listening for a book-related market data. At first, it reconstructs all books (direct, implied or both) for all available securities. When reconstruction is completed , Handler listens for an incremental updates and processes them accordingly.

To check in which state Handler is currently, it's necessary to implement biz.onixs.cme.md.handler.event.HandlerStateChangedListener interface and associate an instance with the Handler using biz.onixs.cme.md.handler.CmeMdHandler#registerStateChangedListener member.

The table below describes all possible states for Handler:

State Description
biz.onixs.cme.md.handler.HandlerState#STOPPED Handler is stopped or was not executed yet.
biz.onixs.cme.md.handler.HandlerState#SECURITY_DEFINITIONS_RECOVERY_STARTED Handler started gathering security definitions either from multicast feed or previously stored cache file.
biz.onixs.cme.md.handler.HandlerState#SECURITY_DEFINITIONS_RECOVERY_FINISHED Handler accomplished processing of security definitions. In normal flow Handler moves to the BooksResynchronizationStarted right after it achieves SecurityDefinitionsRecoveryFinished state. However, in certain cases (for example, in case of error while processing security definitions) Handler may move back to the SecurityDefinitionsRecoveryStarted state.
biz.onixs.cme.md.handler.HandlerState#BOOKS_RESYNCHRONIZATION_STARTED Handler switches to this state when it joins the system for a market data and starts building the books for all securities whose definitions were obtained on previous stage. Moreover, Handler may move into this stage in case of message gap detection while it is maintaining books.
Once Handler moves into this state, books of all types for all securities are no longer valid. They are wiped by the Handler. From this moment Handler starts to rebuild all books from the scratch.
biz.onixs.cme.md.handler.HandlerState#BOOKS_RESYNCHRONIZATION_FINISHED Handler accomplishes process of books resynchronization. From this moment all books are valid. They are in the up-to-date state. Handler continues to listen to any changes in the books and notifies about updates through appropriate listeners.
biz.onixs.cme.md.handler.HandlerState#TCP_REPLAY_STARTED Because of multicast unreliability, message gap may happen while processing market data. If Handler is configured to use TCP Replay facility, it suspends regular processing, moves to this state and requests remote system to resend missed messages.
biz.onixs.cme.md.handler.HandlerState#TCP_REPLAY_FINISHED Handler switches into this state in case of success recover from message gap using TCP replay facility. Afterwards, normal market data processing is restored and Handler continues listening for books updates.

Receive Mode

Threre are two receive modes implemented in the Handler: SELECTOR and SPIN.

Listening to Warnings

Warnings Concept

Miscellaneous non-critical issues may occur while Handler is being executed. Handler will handle such issues by itself, thus no special handling is required for such cases. However, sometimes it's reasonable to be notified about such events. For this reason, handler exposes biz.onixs.cme.md.handler.event.WarningListener class and will invoke its biz.onixs.cme.md.handler.event.WarningListener#onWarning member each time a non-critical issue will take place.

Security Definitions Cache

Security Definitions Cache Concept

Once Handler is started it will listen to security definition channel and will receive all security definitions. Amount of security definitions can be large, so Handler startup time will be significantly increased. To prevent this, security definitions cache can be used.

If cache is enabled, Handler will try to load security definitions from cache instead of listening to the instrument channel during startup. If cache is absent or broken, Handler will start listening to the instrument channel. Then it will receive all security definitions, override cache and stop listenint to the instrument channel.

If cache is disabled, Handler will start listening to the instrument channel. Then it will receive all security definitions and stop listening to the instrument channel.

To enable security definitions we can use an overloaded biz.onixs.cme.md.handler.CmeMdHandler#start member with parameter cacheSecurityDefinitions that is set to true.

If Handler is restarted during the working week, a security definitions cache will be able to miss some security definitions (which were added in the middle of the week). In this case, the cache file must be manually deleted and Handler must be restarted.


The following sample shows the case when Security Definitions caching is enabled:

public class Sample {
    private void run() throws Exception {

        final CmeMdHandler handler = new CmeMdHandler("310");

Raw Market Data Processing

Processing Raw FIX Messages

High-level Handler's API allows to get rid of protocol specific and uses high-level entities directly like security definitions, trading information and books. Advanced events are exposed by Handler to allow the client code to build books by itself using data of atomic changes.

Handler exposes additional level of flexibility by providing ability to process FIX messages (raw market data) as they come from the remote system.

To listen for such kind of data (raw FIX messages), it's necessary to implement biz.onixs.cme.md.handler.event.MessageProcessingListener class interface and associate appropriate instance with the Handler using biz.onixs.cme.md.handler.CmeMdHandler.registerMessageProcessingListener member.

Handler notifies the client code when it begins and accomplishes processing of a message. biz.onixs.cme.md.handler.event.MessageProcessingListener#onMessageProcessingStarted member is called when Handler starts processing a message. biz.onixs.cme.md.handler.event.MessageProcessingListener#onMessageProcessingFinished is called when message processing is accomplished.

FIX messages which are passed to the listener's members by Handler are previously verified and checked for a proper order. If Handler notifies listener about starting of message processing, its guaranteed message is not out of sequence. However, messages, that are obtained from different network sources (different feeds) have uncorrelated sequence numbers.

Thread Specific

Handler may call members of callback interfaces from different threads but these calls was synchronized by Handler.


The following sample prints security description from instrument definition message:

public class MyListener implements MessageProcessingListener {
    private static final Logger LOG = LoggerFactory.getLogger(MyListener.class);

    public void onMessageProcessingStarted(MessageProcessingEventArgs args) {
            LOG.debug("Security ID: {}", args.getMessage().getInt(Tag.SecurityID));

    public void onMessageProcessingFinished(MessageProcessingEventArgs args) {

Group processing uses enumerator API:

public class MyListener implements MessageProcessingListener {
    public void onMessageProcessingStarted(MessageProcessingEventArgs messageProcessingEventArgs) {
        try {
            IMessage message = messageProcessingEventArgs.getMessage();
            IGroup group = message.getGroup(Tag.NoMDEntries);
            while(group.next()) {
                if (group.contains(Tag.SecurityID) && group.contains(Tag.RptSeq)) {
                    int securityId = group.getInt(Tag.SecurityID);
                    long sequenceNumber = group.getLong(Tag.RptSeq);
                    // do some stuff here
        } catch (FieldNotFoundException fnfe) {
    public void onMessageProcessingFinished(MessageProcessingEventArgs args) {}

Controlling Handler Logging

Controlling Logging in the Handler

By default, Handler logs all important aspects of its activity while processing market data. The SLF4J (Simple Logging Facade for Java) is used by Handler internally. The SLF4J detects and uses the concrete logging implementation configured by user. By default, the Logback logging implementation is recommended.

The Logback (or any other logging implementation) must be configured at the beginning of the application. The examples of Logback configuration can be found in the Handler samples. The details of Logback configuration can be found here.

By default logging is configured to log only errors and warnings. This is an example of logback.xml where detailed logging was switched on:

   <logger name="biz.onixs" level="INFO"/>
   <logger name="biz.onixs.cme.md.handler" level="INFO"/>
   <logger name="biz.onixs.cme.md.handler.OrderBookRepository" level="DEBUG"/>
   <logger name="biz.onixs.cme.md.handler.InstrumentRepository" level="DEBUG"/>
   <logger name="biz.onixs.cme.md.handler.sample" level="DEBUG"/>

Log Format and Log Replay

Log replay functionality depends on log record format. Handler uses the following logback pattern for log playing:


Record datetime can be specified in one of the following formats:

If record format will be changed, Handler can loose ability to play such logs.

Using TCP Replay Facility

Recovering from Packet Gaps

Handler receives market data through network multicast feeds. Due to the multicast nature, data may come in wrong order or may be completely lost. Handler does its best to fix all the network-related issues. However, packet gaps may still happen while processing market data.

CME market data system provides its clients with an ability to request and obtain lost packet using reliable TCP connection. This facility is called a TCP Replay. If the Handler is configured to use TCP Replay facility, in case of packet lost, it doesn't report error immediately and falls into books resynchronization. Instead, it suspends regular processing and requests remote system to resend missed packets. Once lost packets are successfully received from the system, Handler resumes regular processing without restarting its execution.

Enabling TCP Replay

To use TCP Replay facility, first of all, it's necessary to contact CME support for TCP Replay credentials. Once login and password information are obtained, they can be passed to Handler and TCP Replay feature activated.

The following example demonstrates how to enable to use TCP Replay feature in Handler:

public class Sample {
    private void run() throws Exception {

        final CmeMdHandler handler = new CmeMdHandler("310");

        // Take advantage of TCP Replay feature.


        // Fill connection credentials.


        // In case of absence of a response from the remote
        // system, Handler will try to reconnect three times
        // with half-of-minute interval between each attempt
        // before it reports about inability to establish
        // connection.


Replaying Log Files

Using Logs to Replay Market Data

In normal flow, Handler logs all important aspects of its execution onto the file system. Log files also include original market data that is processed by Handler. This information is usually saved for analysis of non-standard situations, which may occur during the using of Handler. However, it can be also used to reproduce normal Handler's behavior for a certain period of time.

Once Handler was executed with logging enabled, it is possible to use log files for further replay. First of all, logs must be backed up (copied to another location) or the Handler's configuration must be updated to use another directory for new logs. It happens because current implementation of Handler doesn't support replay from the same folder, in which new logs will be stored.

Handler will not replay log files if packet logging is not enabled. To enable packet logging set biz.onixs.cme.md.handler.Feed in logback.xml at INFO or DEBUG level:
        <logger name="biz.onixs.cme.md.handler" level="INFO"/>
        <logger name="biz.onixs.cme.md.handler.Feed" level="DEBUG"/>

Afterwards, instance of biz.onixs.cme.md.handler.ReplayOptions class must be constructed and initialized with the logs to replay.

When configuring the options is accomplished, replay can be started. For this purpose, biz.onixs.cme.md.handler.CmeMdHandler#start overload is to be called to start replaying. Replaying is performed by Handler asynchronously in the same way as regular execution flows. From the client code point of view, there's no difference whether Handler processes market from the network or from log files.

Handler fires appropriate events as soon as market data is received from the network and processed. However, since CME environment spreads market data with different frequency which depends on live market activity, there're delays of different duration between fired events. When log file is replayed, Handler does not do any pauses and fires corresponding events as soon as data is retrieved from the log file.

To stop Handler to replay log files, biz.onixs.cme.md.handler.CmeMdHandler#stop member can be used.

When replay is accomplished Handler will call biz.onixs.cme.md.handler.event.ReplayListener#onReplayFinished member of the biz.onixs.cme.md.handler.event.ReplayListener class. To get notified about this and other replay-related events it's necessary to implement biz.onixs.cme.md.handler.event.ReplayListener interface and supply an instance of the derived class to the Handler using biz.onixs.cme.md.handler.CmeMdHandler#registerLogReplayListener member.

When market data is being replayed, message receiving time will return time of corresponding log file entry. That is the value that returned the represent original time, when market data message was obtained from the network.
Currently, log replay must be run with the same set of Handler's settings that were originally used.


The following example demonstrates how to replay log files backed up into 'replay' folder:

public class MyListener implements ReplayListener {
    private static final Logger LOG = LoggerFactory.getLogger(MyListener.class);

    public void onReplayError(ErrorEventArgs args) {
        LOG.debug("onReplayError(): {}", args);

    public void onReplayFinished() {

public class Sample {
    private void run() throws Exception {

        final CmeMdHandler handler = new CmeMdHandler("310");

        handler.registerLogReplayListener(new MyListener());

        handler.start(new ReplayOptions("path_to_log"));

        // ...


Custom Log Provider

Handler has an ability to use custom log provider to provide log data from sources different from log files (databases, queues, etc). Handler package has custom-log-replay sample which uses a two handler instances - one for storing received packets in the in-memory queue and second for playing that queue. Packets to store are obtained via biz.onixs.cme.md.handler.PacketProcessingListener#onPacketReceived callback from first Handler, and provided to second Handler implementing biz.onixs.cme.md.handler.LogProvider interface.

Please note that both Handler instances should use same parameters like templates, byte order, startup strategy, etc.

The following example demonstrates how to implement custom log provider:

public class CustomLogProvider implements LogProvider, PacketProcessingListener {

    private class PacketData {
        private final byte[] data;
        private final ConnectionType connectionType;

        public PacketData(byte[] data, ConnectionType connectionType) {
            this.data = data;
            this.connectionType = connectionType;
    private CmeMdHandler source;

    private LinkedBlockingQueue<PacketData> packets = new LinkedBlockingQueue<>();

    private boolean stopRequested;

    public CustomLogProvider(CmeMdHandler source) {
        this.source = source;

    public ReceivedPacket provide() {
        try {
            PacketData packetData = null;
            while (!stopRequested && packetData == null)
                packetData =  packets.poll(100, TimeUnit.MILLISECONDS);
            if (packetData == null)
                return null;
            return new ReceivedPacket(ByteBuffer.wrap(packetData.data).order(source.getMessageSchema().isLittleEndian()?ByteOrder.LITTLE_ENDIAN:ByteOrder.BIG_ENDIAN), FeedType.UNKNOWN, packetData.connectionType);
        } catch (IOException | InterruptedException ignored) {
        return null;

    public void close() {

    public void onPacketProcessingStarted(PacketProcessingEventArgs args) {

    public void onPacketProcessingFinished(PacketProcessingEventArgs args) {

    public void onPacketReceived(PacketProcessingEventArgs args) {
        try {
            packets.put(new PacketData(args.getPacket().getRaw(), args.getPacket().getConnectionType()));
        } catch (InterruptedException ignored) {

    public void stop() {
        stopRequested = true;

The following example demonstrates how to register custom log provider with Handler:

        ReplayOptions replayOptions = new ReplayOptions();
        CustomLogProvider provider = new CustomLogProvider(sourceHandler);

Low Latency Best Practices

There are several tips to achieve minimum latency.

Configure process priority and affinity

To control java vm process priority and affinity under Windows you can use command start. To setup high priority add key /HIGH or /REALTIME. To setup affinity add key /AFFINITY hex_mask.

Example (realtime priority and 3rd CPU affinity mask):

start /REALTIME /AFFINITY 0x8 java -jar app_file.jar

JVM settings

Reducing Gap Count

There are a lot of gap sources: network-related issues, high CPU load, the heavy code in handler listeners, etc. Below are some hints which allow detecting the source of gaps:

CME Smart Stream on GCP

With CME Smart Stream on Google Cloud Platform (GCP), clients can access real-time CME Group market data feeds through native Google Cloud Platform services.

Setup GCP account

The Handler uses a service account as Goggle recommended way of authentication. To set up a service account, please make the following steps:

Setup Handler

There are several configuration options to use GCP with Handler:

Parameter Description Values
UseGcp True to use Google Cloud Platform, false to use multicast.
Default value: false.
GcpKeyPath Defines the path to the downloaded service account JSON file.
GcpProjectId Defines the Customer Google Cloud Platform project id for which a service account was created.
GcpCmeProjectId Defines the CME Google Cloud Platform project id. cmegroup-marketdata-newrel - CERT and NEW RELEASE
cmegroup-marketdata - PRODUCTION
GcpEnvironmentType Defines the CME environment type. GcpEnvironmentType.PRODUCTION - Production.
GcpEnvironmentType.NEW_RELEASE - New Release.
GcpEnvironmentType.CERTIFICATION - Certification.
GcpContentOwner Defines the CME content owner. Only CME MDP 3.0 Globex Data Channels were supported. GcpContentOwner.CME_GROUP_OWNED_MARKET_DATA - CME Group Owned Market Data.
GcpContentOwner.MINNEAPOLIS_GRAIN_EXCHANGE - Minneapolis Grain Exchange.
GcpContentOwner.DUBAI_MERCANTILE_EXCHANGE - Dubai Mercantile Exchange.
GcpParallelPullCount Defines the Google Cloud Platform parallel pull count.
GcpDeleteSubscriptions Defines if Handler should delete Google Cloud Platform subscription after stopping the corresponding feed.
Default value: true.
True - Handler will create a subscription or use existing one on feed start, and delete on feed stop.
False - Handler will create a subscription or use existing one on feed start, and leave subscription on feed stop.


The following example demonstrates how to use Google Cloud Platform as CME MDP 3.0 data provider:

public class Sample {
    private void run() throws Exception {

        final CmeMdHandler handler = new CmeMdHandler("310");




        // ...


See also

CME Smart Stream on GCP
Google Pub/Sub documentation

Connectivity Troubleshooting

Issues Related with Receiving Multicast Data

In most cases, absence of multicast data is caused by network-related configuration issues that are described below.

Firewall Blocks Program and GettingStarted Sample

Updating firewall rules or simply turning it off usually solves the issue.

Disabling firewall explicitly may be dangerous. Consult your network and system administrator before turning firewall off.

Linux-like Operating System Turns on Reverse Path Filtering on Network Interfaces

Disabling reverse path filtering by updating system control parameters usually resolves the issue.

System control parameters (/etc/sysctl.conf) affecting reverse path filtering:

Parameter Description
net.ipv4.conf.default.rp_filter Defines default policy.
net.ipv4.conf.<NetworkInterface>.rp_filter Defines policy for a <NetworkInterface> network interface.

To determine current status of a reverse path filtering, we must run the command below:

sysctl -a | grep rp_filter

To disable a reverse path filtering from console, we must run the following command:

echo 0 > /proc/sys/net/ipv4/conf/<NetworkInterface>/rp_filter

Explicit Network Interface Must Be Specified

The sample program configures Handler to listen to market data on all network interfaces on Windows and Linux. If network configuration supposes the specifying exact network interface (like 'eth0' on Linux) for market data listening, then biz.onixs.cme.md.handler.CmeMdHandler#setLocalNetworkInterface member value must be updated accordingly.


After switching the Operating System even the sample program is no longer able to receive market data. However, I am able to ping the CME router and iLink server. Any insight or suggestions on what may be going on here?

See Connectivity Troubleshooting.

If Handler joins while the market is busy, warnings biz.onixs.cme.sbe.Packet Cache Overflow are reported. Is there any possibility to raise the configured limit, if so how it can be done?

When Handler performs book resynchronization, it caches all the data that are received on incremental feeds. When the market is busy, the number of cached messages may exceed the configured limit defined by biz.onixs.cme.md.handler.CmeMdHandler#setMaximumNumberOfQueuedIncrementalRefreshMessages parameter value. So, to avoid PacketCacheOverflow warnings, it is necessary to increase value of the noted parameter which equals to 10000 by default.

What's the difference between "Book Changed" and "Book Updated" events?

If you subscribe to Book Updated, you will always have a consistent book. Book Changed callbacks are designed for those who want to build and maintain books by themselves. Book Changed is an elementary action over the order book and usually there are multiple changes inside single snapshot and/or incremental refresh. For this reason, the book may not be valid between two changes. If only all the changes are processed from the single network packet (message like snapshot and incremental refresh), the book is considered as valid. Book Updated callbacks are called exactly at the a time when all changes are processed and book appears to be valid and up-to-date.

While using the onConsolidatedBookUpdated callback, we seem to see crosses. Can you please assist?

For the implied eligible products, one may see the cross book momentarily and the book becomes normal after the next transaction at the engine. So it is possible to see the crossed book in the biz.onixs.cme.md.handler.event.ConsolidatedBookUpdatedListener#onConsolidatedBookUpdated callback even when the market is in the opened state.

© Onix Solutions