Onix Solutions Logo

OnixS Java CME Market Data Handler — Programming Guide

Version 3.14.1

TOC

Introduction

Onix Solutions Java CME Market Data Handler is a Java library that provides an access to the CME Group MDP 3.0 market data via Simple Binary encoding (SBE) protocol. High-level Java API allows to build applications rapidly to get market data without much involving into raw protocol specifics.

CME Market Data Feed Handler implementations features include:

Note
We recommend to read the "Market Data Platform (MDP) 3.0" document to get familiar with core aspects of CME Group Market Data System before reading the Guide. By reading this Guide, we also recommend to review a source code of the sample project which comes as a part of the library distributive package.

System Requirements

Getting Started

Using Handler usually includes the following:

Migration Guide

This topic will contain major changes which is important for migration from CME FIX/FAST Market Data Handler.

CME Changes:

Handler Changes:

Adjusting Settings

Market Data Channel

CME market data is distributed across the multiple logical units to reduce network and client system load. They are called channels. Handler requires the channel to be identified in order to process market data. The biz.onixs.cme.md.handler.CmeMdHandler constructor helps to instruct Handler market data of the channel which must be processed.

Market Data Configuration

Handler also requires the market data configuration file to be specified. Handler uses the stored data in this configuration to identify network connectivity attributes for a particular market data channel.
Note
CME Group provides Market data configuration file via their public FTP servers. CME Group updates market data configuration on regular (weekly) basis. Read the CME documentation for more information about configuration updating schedule as well as how to access to public FTP servers.
The biz.onixs.cme.md.handler.CmeMdHandler#setChannelConfigurationFile method has value which has to take a path to the locally stored copy of such configuration file. The default value is "config.xml".
Note
Single market data configuration file contains settings for all market data channels. However, CME Group uses different market data configurations for different environments (like Production and Certification environments). Multiple instances of the biz.onixs.cme.md.handler.CmeMdHandler class can be initialized to use either single or different market data configurations. In other words, it is possible to use multiple instances of Handler to connect to different environments.

SBE Coding Templates

All CME Group market data messages are SBE encoded to reduce network load. SBE coding supposes the use of SBE coding templates (sort of encoding / decoding configuration). Therefore, Handler requires such information to be specified for proper decoding of market data messages. The biz.onixs.cme.md.handler.CmeMdHandler#setTemplateFile method is to instruct Handler where the SBE coding templates file is located. The default value is "templates_FixBinary.xml".
Note
Single SBE coding templates file is similar to the market data configuration. It can be used for different channels within single operating environment (like Production or Certification ones). CME Group maintains all SBE coding templates files for all operating environments by its own and updates them on regular (weekly) basis. All this data is available on public FTP servers of CME Group.

Example

The following example demonstrates how to setup primary settings for Handler.
public class Sample {
    private void run() throws Exception {
        final CmeMdHandler handler = new CmeMdHandler("310");
        handler.setChannelConfigurationFile("my_config.xml");
        handler.setTemplateFile("my_templates.xml");
    }
}

Complete Settings List

Name Description Default value
biz.onixs.cme.md.handler.CmeMdHandler#setChannelConfigurationFile
biz.onixs.cme.md.handler.CmeMdHandler#getChannelConfigurationFile
Path to the CME Market Data Feed Channel configuration XML file "config.xml"
biz.onixs.cme.md.handler.CmeMdHandler#setDirectBookDefaultDepth
biz.onixs.cme.md.handler.CmeMdHandler#getDirectBookDefaultDepth
Direct book default depth. 10
biz.onixs.cme.md.handler.CmeMdHandler#setTemplateFile
biz.onixs.cme.md.handler.CmeMdHandler#getTemplateFile
Path to the SBE templates file. "templates_FixBinary.xml"
biz.onixs.cme.md.handler.CmeMdHandler#setImpliedBookDefaultDepth
biz.onixs.cme.md.handler.CmeMdHandler#getImpliedBookDefaultDepth
Implied book default depth. 2
biz.onixs.cme.md.handler.CmeMdHandler#setIncrementalCacheSize
biz.onixs.cme.md.handler.CmeMdHandler#getIncrementalCacheSize
Incremental packets cache size. This cache are used for packet alignment. 5
biz.onixs.cme.md.handler.CmeMdHandler#setLicenseFile
biz.onixs.cme.md.handler.CmeMdHandler#getLicenseFile
Path to the license file. "OnixS.lic"
biz.onixs.cme.md.handler.CmeMdHandler#setLocalNetworkInterface
biz.onixs.cme.md.handler.CmeMdHandler#getLocalNetworkInterface
Local network interface that is used to receive UDP multicast packets. If set to null then all available local network interfaces are used. null
biz.onixs.cme.md.handler.CmeMdHandler#setLocalNetworkInterfaceA
biz.onixs.cme.md.handler.CmeMdHandler#getLocalNetworkInterfaceA
Local network interface that is used to receive UDP multicast packets for "A" feeds. If set to null then the LocalNetworkInterface value is used. null
biz.onixs.cme.md.handler.CmeMdHandler#setLocalNetworkInterfaceB
biz.onixs.cme.md.handler.CmeMdHandler#getLocalNetworkInterfaceB
Local network interface that is used to receive UDP multicast packets for "B" feeds. If set to null then the LocalNetworkInterface value is used. null
biz.onixs.cme.md.handler.CmeMdHandler# setMaximumNumberOfQueuedIncrementalRefreshMessages
biz.onixs.cme.md.handler.CmeMdHandler# getMaximumNumberOfQueuedIncrementalRefreshMessages
Maximum number of queued Market Data Incremental Refresh (X) messages. When the message gap is detected all subsequent Refresh(X) messages are queued until the Handler re-synchronizes with the market using Snapshot(W) messages. After that the queued Refresh(X) messages are re-played. This setting could be used to limit the memory usage when there is no Market Data - Snapshot/Full Refresh (W) messages. 10000
biz.onixs.cme.md.handler.CmeMdHandler#setReceiveTimeoutInMilliseconds
biz.onixs.cme.md.handler.CmeMdHandler#getReceiveTimeoutInMilliseconds
Amount of time in milliseconds a Handler will wait to receive data once a read operation is initiated. 36000
biz.onixs.cme.md.handler.CmeMdHandler#setRecoverSecurityDefinitionsOnGap
biz.onixs.cme.md.handler.CmeMdHandler#getRecoverSecurityDefinitionsOnGap
The flag to recover the Security Definition messages each time when the message sequence gap is detected. false
biz.onixs.cme.md.handler.CmeMdHandler#setReportNoDataWarning
biz.onixs.cme.md.handler.CmeMdHandler#getReportNoDataWarning
Report-no-data option value. false
biz.onixs.cme.md.handler.CmeMdHandler#setSecurityDefinitionsCacheRoot
biz.onixs.cme.md.handler.CmeMdHandler#getSecurityDefinitionsCacheRoot
Path where security definition cache will be stored. "SecurityDefinitionsCache"
biz.onixs.cme.md.handler.CmeMdHandler#setSocketBufferSize
biz.onixs.cme.md.handler.CmeMdHandler#getSocketBufferSize
UDP socket buffer size. 4MB
biz.onixs.cme.md.handler.CmeMdHandler#setTcpReplay
biz.onixs.cme.md.handler.CmeMdHandler#getTcpReplay
Set option to use the TcpReplay Channel for recovery of missed messages. This method recovers all missed messages. false
biz.onixs.cme.md.handler.CmeMdHandler#setTcpReplayPassword
biz.onixs.cme.md.handler.CmeMdHandler#getTcpReplayPassword
Password or passphrase to be used in the Logon (35=A) message from customer to CME Tcp Replay service. ONIXS
biz.onixs.cme.md.handler.CmeMdHandler#setTcpReplayReconnectAttempts
biz.onixs.cme.md.handler.CmeMdHandler#getTcpReplayReconnectAttempts
Number of attempts to receive missed messages via CME Tcp Replay service. 10
biz.onixs.cme.md.handler.CmeMdHandler# setTcpReplayReconnectIntervalInMilliseconds
biz.onixs.cme.md.handler.CmeMdHandler# getTcpReplayReconnectIntervalInMilliseconds
Interval between attempts to receive missed messages via CME Tcp Replay service (in milliseconds). 100
biz.onixs.cme.md.handler.CmeMdHandler#setTcpReplayUsername
biz.onixs.cme.md.handler.CmeMdHandler#getTcpReplayUsername
User ID or username to be used in the Logon (35=A) message from customer to CME Tcp Replay service. ONIXS
biz.onixs.cme.md.handler.CmeMdHandler#setUseIncrementalFeedA
biz.onixs.cme.md.handler.CmeMdHandler#getUseIncrementalFeedA
The flag to use or not incremental feed "A" option. true
biz.onixs.cme.md.handler.CmeMdHandler#setUseIncrementalFeedB
biz.onixs.cme.md.handler.CmeMdHandler#getUseIncrementalFeedB
The flag to use or not incremental feed "B" option. false
biz.onixs.cme.md.handler.CmeMdHandler#setUseInstrumentReplayFeedA
biz.onixs.cme.md.handler.CmeMdHandler#getUseInstrumentReplayFeedA
The flag to use or not instrument feed "A" option. true
biz.onixs.cme.md.handler.CmeMdHandler#setUseInstrumentReplayFeedB
biz.onixs.cme.md.handler.CmeMdHandler#getUseInstrumentReplayFeedB
The flag to use or not instrument feed "B" option. false
biz.onixs.cme.md.handler.CmeMdHandler#setUseOneReceiverOnSamePort
biz.onixs.cme.md.handler.CmeMdHandler#getUseOneReceiverOnSamePort
True, if one multicast socket should be created for A nd B feeds, if ports are same. false
biz.onixs.cme.md.handler.CmeMdHandler#setUseSnapshotFeedA
biz.onixs.cme.md.handler.CmeMdHandler#getUseSnapshotFeedA
The flag to use or not Market Recovery (UDP) Feed "A". true
biz.onixs.cme.md.handler.CmeMdHandler#setUseSnapshotFeedB
biz.onixs.cme.md.handler.CmeMdHandler#getUseSnapshotFeedB
The flag to use or not Market Recovery (UDP) Feed "B". false

Listening to Market Data

Events in Handler

Once Handler is started, it listens to a market data from the network, then processes it (decodes messages, validates message order, updates order books) and invokes client code for further processing.

Handler processes market data working asynchronously and uses concept of events and event listeners to notify client code about a particular occasion like reception of security definition or direct book update.

Note
We recommend not to call handler methods which change handler state (like handler.stop()) from event listeners.

Listening to a particular Event

For each event like Error Occurred Handler provides the following interface like biz.onixs.cme.md.handler.event.ErrorListener. Client code must implement this interface to be able to handle events of a particular type. Handler also exposes a member like biz.onixs.cme.md.handler.CmeMdHandler#registerErrorListener which allows to associate an instance of the event handler with a particular instance of the biz.onixs.cme.md.handler.CmeMdHandler class.
Note
Associating listener for a particular event with an instance of the biz.onixs.cme.md.handler.CmeMdHandler class must be performed while Handler is in stopped stated. Once the Handler is started, changing listener-event associations is not allowed, and may lead to unpredictable behavior as well as may cause unexpected errors.
Associating event listener with an instance of the biz.onixs.cme.md.handler.CmeMdHandler class is also called subscribing to an event.

Primary High-level Events

There are multiple events exposed by Handler. All events can be logically divided onto high- and low- level event sub-set. High-level events reflect various results of market data processing done by Handler like Security Definition Received, Market Book Updated, and Trading occurred. Low-level events are designed for more control over data processing and for more flexibility. They are described in different sections of this documentation.

The table below describes primary high-level events exposed by Handler. It also depicts correspondence between events, interfaces for listeners and thes Handler´┐Żs members to subscribe to an event.

Event Listener interface to be implemented Handler member to register listener Description
Security Definition Received biz.onixs.cme.md.handler.event.SecurityDefinitionListener biz.onixs.cme.md.handler.CmeMdHandler#registerSecurityDefinitionListener Fired if a definition of a new security is received. Security definitions normally are received at the beginning of the Handler's execution and before any other high-level event like Book Updated event.
Security Status Changed biz.onixs.cme.md.handler.event.SecurityStatusChangedListener biz.onixs.cme.md.handler.CmeMdHandler#registerSecurityStatusListener Fired if the remote system reports about changes in a status of a particular security. An example of such change is update of upper price or trading status.
Direct Book Updated biz.onixs.cme.md.handler.event.RegularBookUpdatedListener biz.onixs.cme.md.handler.CmeMdHandler#registerRegularBookUpdatedListener Occurs when direct book of a particular security is updated.
Direct Book Changed biz.onixs.cme.md.handler.event.RegularBookChangedListener biz.onixs.cme.md.handler.CmeMdHandler#registerRegularBookChangedListener Occurs when direct book of a particular security is changed.
Implied Book Updated biz.onixs.cme.md.handler.event.ImpliedBookUpdatedListener biz.onixs.cme.md.handler.CmeMdHandler#registerImpliedBookUpdatedListener Occurs when implied book of a particular security is updated.
Implied Book Changed biz.onixs.cme.md.handler.event.ImpliedBookChangedListener biz.onixs.cme.md.handler.CmeMdHandler#registerImpliedBookChangedListener Occurs when implied book of a particular security is changed.
Consolidated Book Updated biz.onixs.cme.md.handler.event.ConsolidatedBookUpdatedListener biz.onixs.cme.md.handler.CmeMdHandler#registerConsolidatedBookUpdatedListener Occurs when consolidated book of a particular security is updated.
Consolidated Book Changed biz.onixs.cme.md.handler.event.ConsolidatedBookChangedListener biz.onixs.cme.md.handler.CmeMdHandler#registerConsolidatedBookChangedListener Occurs when consolidated book of a particular security is changed.
Statistics Changed biz.onixs.cme.md.handler.event.StatisticsListener biz.onixs.cme.md.handler.CmeMdHandler#registerStatisticsListener Occurs when attributes like opening or closing price for a particular security are changed.
Trade Information Received biz.onixs.cme.md.handler.event.TradeListener biz.onixs.cme.md.handler.CmeMdHandler#registerTradeListener Occurs if trade information is received for a particular security. Trade attributes are collected into an instance of biz.onixs.cme.md.handler.event.TradeEventArgs class which is supplied as a parameter to the listener by Handler.
News Received biz.onixs.cme.md.handler.event.NewsReceivedListener biz.onixs.cme.md.handler.CmeMdHandler#registerNewsReceivedListener Fired when the system spreads arbitrary news to Handler. Headline and full text of the news are exposed via an instance of biz.onixs.cme.md.handler.event.NewsReceivedEventArgs class.

Example

The following sample demonstrates how to listen to notifications about security definition reception.
public class MyListener implements SecurityDefinitionListener {
    private static final Logger LOG = LoggerFactory.getLogger(MyListener.class);

    public void onSecurityDefinition(SecurityDefinitionEventArgs args) {
        LOG.debug("onSecurityDefinition(): {}", args);
    }
}

public class Sample {
    private void run() throws Exception {
        final CmeMdHandler handler = new CmeMdHandler ("310");
        // ...
        final MyListener listener = new MyListener();
        handler.registerSecurityDefinitionListener(listener);
    }
}

Manipulating Books

Books in Handler

Handler builds and maintains books for each security whose definition was previously received in bounds of a particular market data channel.

All books, maintained by Handler, represent price-level books. Book abstraction is exposed by the biz.onixs.cme.md.handler.IOrderBook interface.

There are several types of books that are available for each security: direct, implied and consolidated. Handler builds and maintains all type of books for all securities.

Once a book of any type and for any security is updated, Handler notifies client code using appropriate event:

Instance of a book for a particular security is passed through a parameter of event listener interface. Although Handler differs a book update event for each type of a book, biz.onixs.cme.md.handler.IOrderBook#getType member can be used to identify whether given instance of the book is direct, implied or consolidated.

The depth of a book of a particular type is usually defined by security definition and is exposed by biz.onixs.cme.md.handler.IOrderBook#getDepth member. Returned value defines maximum number of price levels that may exist in the book.

biz.onixs.cme.md.handler.IOrderBook#getBids and biz.onixs.cme.md.handler.IOrderBook#getAsks members of the biz.onixs.cme.md.handler.IOrderBook class can be used to access price value and quantity information for all levels available in book.

Behavioral Traits

Handler constructs books for each security when it receives Security Definition message for that security. Books are changed in time and their members are not thread-safe. Handler uses its own synchronization strategies while updating the books. Therefore, do not access price levels information outside of book-related event handling as well as do not store references to any book instance and data returned by its members like asks and bids collections. It happens because Handler may change internal book structures at the same time. If any data, which is exposed by the biz.onixs.cme.md.handler.IOrderBook class instance, must be used outside of event handling, it should be copied before later using. To construct immutable copy of the book for using outside of callback, use the biz.onixs.cme.md.handler.IOrderBook#clone member.

See Listening to Market Data topic for more information concerning how to subscribe for a book updates.

Building and Maintaining Books by Yourself

Books Building Bricks

As it was mentioned in the previous topic, Handler builds and maintains books for each security by itself and notifies client code once book update takes place.

Sometimes there is a necessity to build and maintain books by own forces (for example, if there's no need in information of the lowest price-levels). For this purpose, Handler provides advanced events, which contain detailed information about the changes, needed to be applied to a book, to bring it into an actual state.

To get notified about detailed changes into a direct book for a particular security, implement biz.onixs.cme.md.handler.event.RegularBookChangedListener interface and register it in the Handler via biz.onixs.cme.md.handler.CmeMdHandler#registerRegularBookChangedListener member.

To get notified about detailed changes into an implied book for a particular security, implement biz.onixs.cme.md.handler.event.ImpliedBookChangedListener interface and register it in the Handler via biz.onixs.cme.md.handler.CmeMdHandler#registerImpliedBookChangedListener member.

Both events expose detailed information of elementary change into book's bids and asks.

Note
There is no ability to listen to changes for consolidated books because these books are derived from direct and implied ones.

Book Change Machinery and Relation with 'Book Updated' Event

Book Updated and Book Changed events are not mutually exclusive ones. It is possible to subscribe to both types of events. Book changes represent more elementary (atomic) updates of the book. Therefore, Book Changed events may occur more frequently in compare to Book Updated event occasions. In fact, series of Book Changed events occur before single occasion of Book Updated event.

When an order book is delivered via Book Updated callbacks, it is always in a valid and up-to-date state. In contrast, a book change is an elementary action over a book and there're usually multiple changes inside single snapshot and/or incremental refresh. For this reason, the book may not be in a valid state between two changes. Only when all the changes are processed from the single network packet (a message like snapshot and incremental refresh), the book can be considered as valid. Book Updated callbacks are called exactly at the time when all changes are processed and the book appears to be valid and up-to-date.

To build an order book properly, it is also necessary to listen to Handler State changes. When Handler falls into biz.onixs.cme.md.handler.HandlerState#BOOKS_RESYNCHRONIZATION_STARTED, all books must be empty. Afterwards, changes that are reported via Book Changed callbacks, must be applied to build a correct order book. An important aspect is that Handler doesn't differ changes from snapshots and incremental refreshes. From the API perspective, all the changes, reported through callbacks, are of the same structure. To detect the source of a change (e.g. snapshot and/or incremental refresh) as well as to determine the bounds of changes transaction it is necessary to subscribe to Message Processing events (Raw Market Data Processing).

Note
Handler exposes members in listening interfaces like biz.onixs.cme.md.handler.event.RegularBookChangedListener#onRegularBookChangesReset. These members can be used to reset all books instead of using Handler state listening.

Customizing Book Changes/Updates Notifications

Tracking the Best Bids and Asks Only

By default, Handler fires Book Changed and Book Updated events each time it receives incremental updates for a particular book from the remote system. In certain cases, there is a necessity to reduce amount of events, generated by Handler. In particular, it is quite often that only the top of a book is a point of intersect. For this reason, Handler allows to get notified only about changes and/or updates into the best bids and asks.

To take advantage of such facility, use overloads which accept instance of biz.onixs.cme.md.handler.BestBidAskTrackingOptions class and register listeners like biz.onixs.cme.md.handler.event.TopOfTheRegularBookUpdatedListener

biz.onixs.cme.md.handler.BestBidAskTrackingOptions#setBestBidAskTrackingParameters bit field specifies, of which available bid/ask attributes (e.g. price or quantity or both) should be tracked for changes. Handler will report about changes and/or updates into a book only if specified parameters of the best bid/ask were changed.

Other members of biz.onixs.cme.md.handler.BestBidAskTrackingOptions class allow to control sensitivity of Handler to the changes and updates.

In particular, biz.onixs.cme.md.handler.BestBidAskTrackingOptions#setPriceThreshold member allows to define sensitivity threshold for the best bid/ask price values exposed in percent. If price of the best bid or ask changes for more than specified value (in percent), Handler will raise appropriate event. In other case, it will not do it.

Similarly, biz.onixs.cme.md.handler.BestBidAskTrackingOptions#setQuantityThreshold member controls occasion of Top of The Book Updated event, in case the best bid/ask quantity value changes.

Example

The following example demonstrates how to configure Handler to get notified only if price of best bid and ask changes for more than 10 percent or quantity changes for more than 50 percent.

public class MyListener implements TopOfTheRegularBookUpdatedListener {
    private static final Logger LOG = LoggerFactory.getLogger(MyListener.class);

    public void onTopOfTheRegularBookUpdated(TopOfTheOrderBookUpdatedEventArgs args) {
        LOG.debug("onTopOfTheRegularBookUpdated(): {}", args);
    }
}

public class Sample {
    private void run() throws Exception {
        final CmeMdHandler handler = new CmeMdHandler ("310");
        handler.getBestBidAskTrackingOptions().setBestBidAskTrackingParameters(BestBidAskTrackingParameter.PRICE | BestBidAskTrackingParameter.QUANTITY);
        handler.getBestBidAskTrackingOptions().setPriceThreshold(new ScaledDecimal("10"));
        handler.getBestBidAskTrackingOptions().setQuantityThreshold(50);
        // ...
        final MyListener listener = new MyListener();
        handler.registerTopOfTheRegularBookUpdatedListener(listener);
    }
}

Filtering Securities

Defining Securities of Interest

By default, Handler processes market data for all securities that are available in a particular market data channel. For certain channels, it may be a huge amount of data which may cause Handler to allocate significant amount of system resources (like a memory). On the other side, it often happens that all securities are not the point of interest. For these reasons, Handler provides an ability to define a subset of securities for which it will monitor market data, maintain books and report about changes in statistics. No events will be fired by Handler for any security it they are not presented in the defined subset. This feature is called as securities filtering.

Operating over securities filter is simple. biz.onixs.cme.md.handler.CmeMdHandler#addSecurityIdFilter member must be used to put a security into the collection of securities for which Handler must process market data. There are no limits on quantity of securities which can be included into filtering. Client code may be put from single up to all securities that are available in the single market data channel.

If at least one security is added to the filter, Handler stops raising events for all securities that are available in the market data channel. From that moment Handler will fire events only for securities that are included into filtering.

To remove security from the filter, it's necessary to call biz.onixs.cme.md.handler.CmeMdHandler#removeSecurityIdFilter member. Handler will continue processing market data and raising events only for the securities that remain in the filter.

If collection of securities in the filter becomes empty, Handler starts processing market data and event notifying for all securities in the market data channel.

To ensure that no securities are available in the securities filter, use biz.onixs.cme.md.handler.CmeMdHandler#clearSecurityIdFilters member. It will clear the filter.

In the same way securities can be filtered by Security Description, Security Group and Symbol.

Note
Filtering must be manipulated only when Handler is in biz.onixs.cme.md.handler.HandlerState#STOPPED, biz.onixs.cme.md.handler.HandlerState#STARTED or in biz.onixs.cme.md.handler.HandlerState#SECURITY_DEFINITIONS_RECOVERY_STARTED state. Once Handler changes its state to any other one, filters must not be modified.

Example

The following code demonstrates how to instruct Handler to process data only for two particular securities:

public class Sample {
    private void run() throws Exception {
        final CmeMdHandler handler = new CmeMdHandler ("310");

        handler.addSecurityIdFilter(05930);
        handler.addSecurityIdFilter(05931);

        handler.start();

        // ...

        handler.stop();

        handler.clearSecurityIdFilters();
    }
}

Error Handling

Error Handling Concept

Being presented as a Java library, Handler uses exceptions to report about error occurred while performing certain action. For example, Handler will raise regular exception to report about inability to find actual license for the product. However, as far as Handler processes market data asynchronously, it is not able to report about any further errors from the moment market data processing is started. For this reason, Handler exposes biz.onixs.cme.md.handler.event.ErrorListener interface and biz.onixs.cme.md.handler.CmeMdHandler#registerErrorListener member to be able to subscribe to and handle errors.

Once instance of biz.onixs.cme.md.handler.event.ErrorListener is assigned to Handler, it will invoke biz.onixs.cme.md.handler.event.ErrorListener#onError member each time whe the error occurs. biz.onixs.cme.md.handler.event.ErrorListener#onError member has several incoming parameters one of which defines human-readable explanation (description) of the error.

An important aspect of the Handler's behavior is that once error occurred Handler will automatically restart market data processing starting from security definitions.

Example

The following sample demonstrates how to receive error notifications.

public class MyListener implements ErrorListener {
    private static final Logger LOG = LoggerFactory.getLogger(MyListener.class);

    public void onError(ErrorEventArgs args) {
        LOG.debug("onError(): {}", args);
    }
}

public class Sample {
    private void run() throws Exception {
        final CmeMdHandler handler = new CmeMdHandler("310");
        // ...
        final MyListener listener = new MyListener();
        handler.registerErrorListener(listener);
    }
}

Licensing the Handler

Specifying License Location

Handler needs a license for successful execution. If Handler is not able to find a valid license, it will throw an exception at the initialization stage.

biz.onixs.cme.md.handler.CmeMdHandler class exposes biz.onixs.cme.md.handler.CmeMdHandler#setLicenseFile member that allows to instruct Handler where to look for a valid license. By default, Handler looks for a license in the current directory for the application that it uses, home directory of current user and a class path. However, by using noted parameter, it's possible to specify another folder anywhere on a file system.

Note
Handler looks for a valid license in the specified folder and selects the best one. If multiple licenses are available, it will select the most significant one.

Example

The following example demonstrates how we can supply license to Handler:


public class Sample {
    private void run() throws Exception {
        CmeMdHandler.setLicenseFile("license.lic");

        final CmeMdHandler handler = new CmeMdHandler("310");
    }
}

Understanding States

Handler States

Once Handler is started, it processes securities definitions that are retrieved either from multicast channel or previously recorded local cache file. Afterwards, Handler starts listening for a book-related market data. At first, it reconstructs all books (direct, implied or both) for all available securities. When reconstruction is completed , Handler listens for an incremental updates and processes them accordingly.

To check in which state Handler is currently, it's necessary to implement biz.onixs.cme.md.handler.event.HandlerStateChangedListener interface and associate an instance with the Handler using biz.onixs.cme.md.handler.CmeMdHandler#registerStateChangedListener member.

The table below describes all possible states for Handler:

State Description
biz.onixs.cme.md.handler.HandlerState#STOPPED Handler is stopped or was not executed yet.
biz.onixs.cme.md.handler.HandlerState#SECURITY_DEFINITIONS_RECOVERY_STARTED Handler started gathering security definitions either from multicast feed or previously stored cache file.
biz.onixs.cme.md.handler.HandlerState#SECURITY_DEFINITIONS_RECOVERY_FINISHED Handler accomplished processing of security definitions. In normal flow Handler moves to the BooksResynchronizationStarted right after it achieves SecurityDefinitionsRecoveryFinished state. However, in certain cases (for example, in case of error while processing security definitions) Handler may move back to the SecurityDefinitionsRecoveryStarted state.
biz.onixs.cme.md.handler.HandlerState#BOOKS_RESYNCHRONIZATION_STARTED Handler switches to this state when it joins the system for a market data and starts building the books for all securities whose definitions were obtained on previous stage. Moreover, Handler may move into this stage in case of message gap detection while it is maintaining books.
Note
Once Handler moves into this state, books of all types for all securities are no longer valid. They are wiped by the Handler. From this moment Handler starts to rebuild all books from the scratch.
biz.onixs.cme.md.handler.HandlerState#BOOKS_RESYNCHRONIZATION_FINISHED Handler accomplishes process of books resynchronization. From this moment all books are valid. They are in the up-to-date state. Handler continues to listen to any changes in the books and notifies about updates through appropriate listeners.
biz.onixs.cme.md.handler.HandlerState#TCP_REPLAY_STARTED Because of multicast unreliability, message gap may happen while processing market data. If Handler is configured to use TCP Replay facility, it suspends regular processing, moves to this state and requests remote system to resend missed messages.
biz.onixs.cme.md.handler.HandlerState#TCP_REPLAY_FINISHED Handler switches into this state in case of success recover from message gap using TCP replay facility. Afterwards, normal market data processing is restored and Handler continues listening for books updates.

Listening to Warnings

Warnings Concept

Miscellaneous non-critical issues may occur while Handler is being executed. Handler will handle such issues by itself, thus no special handling is required for such cases. However, sometimes it's reasonable to be notified about such events. For this reason, handler exposes biz.onixs.cme.md.handler.event.WarningListener class and will invoke its biz.onixs.cme.md.handler.event.WarningListener#onWarning member each time a non-critical issue will take place.

Security Definitions Cache

Security Definitions Cache Concept

Once Handler is started it will listen to security definition channel and will receive all security definitions. Amount of security definitions can be large, so Handler startup time will be significantly increased. To prevent this, security definitions cache can be used.

If cache is enabled, Handler will try to load security definitions from cache instead of listening to the instrument channel during startup. If cache is absent or broken, Handler will start listening to the instrument channel. Then it will receive all security definitions, override cache and stop listenint to the instrument channel.

If cache is disabled, Handler will start listening to the instrument channel. Then it will receive all security definitions and stop listening to the instrument channel.

To enable security definitions we can use an overloaded biz.onixs.cme.md.handler.CmeMdHandler#start member with parameter cacheSecurityDefinitions that is set to true.

Note
If Handler is restarted during the working week, a security definitions cache will be able to miss some security definitions (which were added in the middle of the week). In this case, the cache file must be manually deleted and Handler must be restarted.

Example

The following sample shows the case when Security Definitions caching is enabled:


public class Sample {
    private void run() throws Exception {

        final CmeMdHandler handler = new CmeMdHandler("310");
        handler.start(true);
    }
}

Raw Market Data Processing

Processing Raw FIX Messages

High-level Handler's API allows to get rid of protocol specific and uses high-level entities directly like security definitions, trading information and books. Advanced events are exposed by Handler to allow the client code to build books by itself using data of atomic changes.

Handler exposes additional level of flexibility by providing ability to process FIX messages (raw market data) as they come from the remote system.

To listen for such kind of data (raw FIX messages), it's necessary to implement biz.onixs.cme.md.handler.event.MessageProcessingListener class interface and associate appropriate instance with the Handler using biz.onixs.cme.md.handler.CmeMdHandler.registerMessageProcessingListener member.

Handler notifies the client code when it begins and accomplishes processing of a message. biz.onixs.cme.md.handler.event.MessageProcessingListener#onMessageProcessingStarted member is called when Handler starts processing a message. biz.onixs.cme.md.handler.event.MessageProcessingListener#onMessageProcessingFinished is called when message processing is accomplished.

Note
FIX messages which are passed to the listener's members by Handler are previously verified and checked for a proper order. If Handler notifies listener about starting of message processing, its guaranteed message is not out of sequence. However, messages, that are obtained from different network sources (different feeds) have uncorrelated sequence numbers.

Thread Specific

Handler may call biz.onixs.cme.md.handler.event.MessageProcessingListener members simultaneously from different threads. Invocation of any member of biz.onixs.cme.md.handler.event.MessageProcessingListener class is not synchronized by Handler.

Example

The following sample prints security description from instrument definition message:


public class MyListener implements MessageProcessingListener {
    private static final Logger LOG = LoggerFactory.getLogger(MyListener.class);

    public void onMessageProcessingStarted(MessageProcessingEventArgs args) {
        if("d".equals(args.getMessage().getSemanticType()))
            LOG.debug("Security ID: {}", args.getMessage().getInt(Tag.SecurityID));
    }

    public void onMessageProcessingFinished(MessageProcessingEventArgs args) {
    }
}

Group processing uses enumerator API:

public class MyListener implements MessageProcessingListener {
    @Override
    public void onMessageProcessingStarted(MessageProcessingEventArgs messageProcessingEventArgs) {
        try {
            IMessage message = messageProcessingEventArgs.getMessage();
            IGroup group = message.getGroup(Tag.NoMDEntries);
            while(group.next()) {
                if (group.contains(Tag.SecurityID) && group.contains(Tag.RptSeq)) {
                    int securityId = fs.getInt(Tag.SecurityID);
                    long sequenceNumber = fs.getLong(Tag.RptSeq);
                    // do some stuff here
                }
            }
        } catch (FieldNotFoundException fnfe) {
            fnfe.printStackTrace();
        }
    }
    @Override
    public void onMessageProcessingFinished(MessageProcessingEventArgs args) {}
}

Controlling Handler Logging

Controlling Logging in the Handler

By default, Handler logs all important aspects of its activity while processing market data. The SLF4J (Simple Logging Facade for Java) is used by Handler internally. The SLF4J detects and uses the concrete logging implementation configured by user. By default, the Logback logging implementation is recommended.

The Logback (or any other logging implementation) must be configured at the beginning of the application. The examples of Logback configuration can be found in the Handler samples. The details of Logback configuration can be found here.

By default logging is configured to log only errors and warnings. This is an example of logback.xml where detailed logging was switched on:
<logger name="biz.onixs" level="INFO"/>
<logger name="biz.onixs.cme.md.handler" level="INFO"/>
<logger name="biz.onixs.cme.md.handler.OrderBookRepository" level="DEBUG"/>
<logger name="biz.onixs.cme.md.handler.InstrumentRepository" level="DEBUG"/>
<logger name="biz.onixs.cme.md.handler.sample" level="DEBUG"/>

Using TCP Replay Facility

Recovering from Message Gaps

Handler receives market data through network multicast feeds. Due to the multicast nature, data may come in wrong order or may be completely lost. Handler does its best to fix all the network-related issues. However, message gaps may still happen while processing market data.

CME market data system provides its clients with an ability to request and obtain lost messages using reliable TCP connection. This facility is called a TCP Replay. If the Handler is configured to use TCP Replay facility, in case of message lost, it doesn't report error immediately and falls into books resynchronization. Instead, it suspends regular processing and requests remote system to resend missed messages. Once lost messages are successfully received from the system, Handler resumes regular processing without restarting its execution.

Enabling TCP Replay

To use TCP Replay facility, first of all, it's necessary to contact CME support for TCP Replay credentials. Once login and password information are obtained, they can be passed to Handler and TCP Replay feature activated.

The following example demonstrates how to enable to use TCP Replay feature in Handler:


public class Sample {
    private void run() throws Exception {

        final CmeMdHandler handler = new CmeMdHandler("310");

        // Take advantage of TCP Replay feature.

        handler.setTcpReplay(true);

        // Fill connection credentials.

        handler.setTcpReplayUsername("MyLogin");
        handler.setTcpReplayPassword("MyPassword");

        // In case of absence of a response from the remote
        // system, Handler will try to reconnect three times
        // with half-of-minute interval between each attempt
        // before it reports about inability to establish
        // connection.

        handler.setTcpReplayReconnectAttempts(3);
        handler.setTcpReplayReconnectIntervalInMilliseconds(30000);
    }
}

Replaying Log Files

Using Logs to Replay Market Data

In normal flow, Handler logs all important aspects of its execution onto the file system. Log files also include original market data that is processed by Handler. This information is usually saved for analysis of non-standard situations, which may occur during the using of Handler. However, it can be also used to reproduce normal Handler's behavior for a certain period of time.

Once Handler was executed with logging enabled, it is possible to use log files for further replay. First of all, logs must be backed up (copied to another location) or the Handler's configuration must be updated to use another directory for new logs. It happens because current implementation of Handler doesn't support replay from the same folder, in which new logs will be stored.

Note
Handler will not replay log files if packet logging are not enabled. To enable packet logging set biz.onixs.cme.md.handler.Feed in logback.xml at INFO or DEBUG level:
<logger name="biz.onixs.cme.md.handler" level="INFO"/>
or
<logger name="biz.onixs.cme.md.handler.Feed" level="DEBUG"/>

Afterwards, instance of biz.onixs.cme.md.handler.ReplayOptions class must be constructed and initialized with the logs to replay.

When configuring the options is accomplished, replay can be started. For this purpose, biz.onixs.cme.md.handler.CmeMdHandler#start overload is to be called to start replaying. Replaying is performed by Handler asynchronously in the same way as regular execution flows. From the client code point of view, there's no difference whether Handler processes market from the network or from log files.

Note
Handler fires appropriate events as soon as market data is received from the network and processed. However, since CME environment spreads market data with different frequency which depends on live market activity, there're delays of different duration between fired events. When log file is replayed, Handler does not do any pauses and fires corresponding events as soon as data is retrieved from the log file.

To stop Handler to replay log files, biz.onixs.cme.md.handler.CmeMdHandler#stop member can be used.

When replay is accomplished Handler will call biz.onixs.cme.md.handler.event.ReplayListener#onReplayFinished member of the biz.onixs.cme.md.handler.event.ReplayListener class. To get notified about this and other replay-related events it's necessary to implement biz.onixs.cme.md.handler.event.ReplayListener interface and supply an instance of the derived class to the Handler using biz.onixs.cme.md.handler.CmeMdHandler#registerLogReplayListener member.

Note
When market data is being replayed, message receiving time will return time of corresponding log file entry. That is the value that returned the represent original time, when market data message was obtained from the network.
Currently, log replay must be run with the same set of Handler's settings that were originally used.

Example

The following example demonstrates how to replay log files backed up into 'replay' folder:


public class MyListener implements ReplayListener {
    private static final Logger LOG = LoggerFactory.getLogger(MyListener.class);

    public void onReplayError(ErrorEventArgs args) {
        LOG.debug("onReplayError(): {}", args);
    }

    public void onReplayFinished() {
        LOG.debug("onReplayFinished()");
    }
}

public class Sample {
    private void run() throws Exception {

        final CmeMdHandler handler = new CmeMdHandler("310");

        handler.registerLogReplayListener(new MyListener());

        handler.start(new ReplayOptions("path_to_log"));

        // ...

        handler.stop();
    }
}

Low Latency Best Practices

There are several tips to achieve minimum latency.

Use Direct Threading Model.

Direct threading model uses single thread to receive and process data (old Buffered mode uses two threads: one to receive data and put it into the queue, and other to take data from the queue and process it).

How to enable Direct threading model

handler.setPacketsPullingStrategy(PacketsPullingStrategy.DIRECT);

Configure process priority and affinity.

To control java vm process priority and affinity under Windows you can use command start. To setup high priority add key /HIGH or /REALTIME. To setup affinity add key /AFFINITY hex_mask

Example (realtime priority and 3rd CPU affinity mask)

start /REALTIME /AFFINITY 0x8 java -jar app_file.jar

JVM settings.

Connectivity Troubleshooting

Issues Related with Receiving Multicast Data

In most cases, absence of multicast data is caused by network-related configuration issues that are described below.

Firewall Blocks Program and GettingStarted Sample

Updating firewall rules or simply turning it off usually solves the issue.

Note
Disabling firewall explicitly may be dangerous. Consult your network and system administrator before turning firewall off.

Linux-like Operating System Turns on Reverse Path Filtering on Network Interfaces

Disabling reverse path filtering by updating system control parameters usually resolves the issue.

System control parameters (/etc/sysctl.conf) affecting reverse path filtering:

Parameter Description
net.ipv4.conf.default.rp_filter Defines default policy.
net.ipv4.conf.<NetworkInterface>.rp_filter Defines policy for a <NetworkInterface> network interface.

To determine current status of a reverse path filtering, we must run the command below:

sysctl -a | grep rp_filter

To disable a reverse path filtering from console, we must run the following command:

echo 0 > /proc/sys/net/ipv4/conf/<NetworkInterface>/rp_filter

Explicit Network Interface Must Be Specified

The sample program configures Handler to listen to market data on all network interfaces on Windows and Linux. If network configuration supposes the specifying exact network interface (like 'eth0' on Linux) for market data listening, then biz.onixs.cme.md.handler.CmeMdHandler#setLocalNetworkInterface member value must be updated accordingly.

FAQ

After switching the Operating System even the sample program is no longer able to receive market data. However, I am able to ping the CME router and iLink server. Any insight or suggestions on what may be going on here?

See Connectivity Troubleshooting.

If Handler joins while the market is busy, warnings biz.onixs.cme.sbe.Packet Cache Overflow are reported. Is there any possibility to raise the configured limit, if so how it can be done?

When Handler performs book resynchronization, it caches all the data that are received on incremental feeds. When the market is busy, the number of cached messages may exceed the configured limit defined by biz.onixs.cme.md.handler.CmeMdHandler#setMaximumNumberOfQueuedIncrementalRefreshMessages parameter value. So, to avoid 'PacketCacheOverflow' warnings, it is necessary to increase value of the noted parameter which equals to 10000 by default.

What's the difference between "Book Changed" and "Book Updated" events?

If you subscribe to Book Updated, you will always have a consistent book. Book Changed callbacks are designed for those who want to build and maintain books by themselves. Book Changed is an elementary action over the order book and usually there are multiple changes inside single snapshot and/or incremental refresh. For this reason, the book may not be valid between two changes. If only all the changes are processed from the single network packet (message like snapshot and incremental refresh), the book is considered as valid. Book Updated callbacks are called exactly at the a time when all changes are processed and book appears to be valid and up-to-date.

While using the onConsolidatedBookUpdated callback, we seem to see crosses. Can you please assist?

For the implied eligible products, one may see the cross book momentarily and the book becomes normal after the next transaction at the engine. So it is possible to see the crossed book in the biz.onixs.cme.md.handler.event.ConsolidatedBookUpdatedListener#onConsolidatedBookUpdated callback even when the market is in the opened state.


© Onix Solutions