OnixS Java CME FIX/FAST Market Data Handler — Programmer's Guide
Version 1.16.0
TOC
- Introduction
- System Requirements
- Getting Started
- Advanced Programming
- Connectivity Troubleshooting
- FAQ
Introduction
Onix Solutions Java CME FIX/FAST Market Data Handler is a Java library that provides an access to the CME Group market data using FIX/FAST Protocol. High-level Java API allows to build applications rapidly to get market data without much involving into raw protocol specifics.CME Market Data Feed Handler implementations features include:
- Easy-to-use API.
- Multithreaded architecture.
- Low latency, high throughput.
- Synchronization of client systems to the latest market state.
- Security Definition and Security Status Delivery to the client system.
- Security Definitions caching for faster start-up and recovery.
- Incremental order book management.
- Real-time Trade and News notifications.
- Recovery from network-related issues using "TCP Replay" and "Market Recovery" facilities.
- Arbitration between primary (A) and secondary (B) feeds.
- Per-security filtering.
- Ability to handle multihomed hosts.
- Flexible logging.
- Log replay facility.
- Note
- We recommend to read the "Streamlined Market Data" document to get familiar with core aspects of CME Group Market Data System before reading the Guide. By reading this Guide, we also recommend to review a source code of the sample project which comes as a part of the library distributive package.
System Requirements
- JDK 1.7 (or higher).
Getting Started
Using Handler usually includes the following:
- Creates an instance of
biz.onixs.cme.md.handler.CmeMdHandler
class. - Adjusts Handler settings like connectivity configuration and FAST coding templates.
- Registers listeners for miscellaneous events like security definition reception or order book update.
- Registers listeners for errors and warnings to be notified about failures occurred while the Handler processes market data.
- Starts market data processing by invoking
biz.onixs.cme.md.handler.CmeMdHandler#start
member. - Processes data of the events for which listeners were previously registered.
- Stops market data processing using the
biz.onixs.cme.md.handler.CmeMdHandler#stop
member.
Adjusting Settings
Market Data Channel
CME market data is distributed across the multiple logical units to reduce network and client system load. They are called channels. Handler requires the channel to be identified in order to process market data. Thebiz.onixs.cme.md.handler.CmeMdHandler
constructor helps to instruct Handler market data of the channel which must be processed.
Market Data Configuration
Handler also requires the market data configuration file to be specified. Handler uses the stored data in this configuration to identify network connectivity attributes for a particular market data channel.- Note
- CME Group provides Market data configuration file via their public FTP servers. CME Group updates market data configuration on regular (weekly) basis. Read the CME Group documentation for more information about configuration updating schedule as well as how to access to public FTP servers.
biz.onixs.cme.md.handler.CmeMdHandler#setChannelConfigurationFile
method has value which has to take a path to the locally stored copy of such configuration file. The default value is "config.xml".
- Note
- Single market data configuration file contains settings for all market data channels. However, CME Group uses different market data configurations for different environments (like Production and Certification environments). Multiple instances of the
biz.onixs.cme.md.handler.CmeMdHandler
class can be initialized to use either single or different market data configurations. In other words, it is possible to use multiple instances of Handler to connect to different environments.
FAST Coding Templates
All CME Group market data messages are FAST encoded to reduce network load. FAST coding supposes the use of FAST coding templates (sort of encoding / decoding configuration). Therefore, Handler requires such information to be specified for proper decoding of market data messages. Thebiz.onixs.cme.md.handler.CmeMdHandler#setFastTemplateFile
method is to instruct Handler where the FAST coding templates file is located.
The default value is
"templates.xml".
- Note
- Single FAST coding templates file is similar to the market data configuration. It can be used for different channels within single operating environment (like Production or Certification ones). CME Group maintains all FAST coding templates files for all operating environments by its own and updates them on regular (weekly) basis. All this data is available on public FTP servers of CME Group.
Example
The following example demonstrates how to setup primary settings for Handler.public class Sample { private void run() throws Exception { final CmeMdHandler handler = new CmeMdHandler("30"); handler.setChannelConfigurationFile("my_config.xml"); handler.setFastTemplateFile("my_templates.xml"); } }
Complete Settings List
Name | Description | Default value |
---|---|---|
biz.onixs.cme.md.handler.CmeMdHandler#setChannelConfigurationFile biz.onixs.cme.md.handler.CmeMdHandler#getChannelConfigurationFile |
Path to the CME Market Data Feed Channel configuration XML file | "config.xml" |
biz.onixs.cme.md.handler.CmeMdHandler#setDirectBookDefaultDepth biz.onixs.cme.md.handler.CmeMdHandler#getDirectBookDefaultDepth |
Direct book default depth. | 10 |
biz.onixs.cme.md.handler.CmeMdHandler#setFastTemplateFile biz.onixs.cme.md.handler.CmeMdHandler#getFastTemplateFile |
Path to the CME FAST templates file. | "templates.xml" |
biz.onixs.cme.md.handler.CmeMdHandler#setFixDialectDescriptionFile biz.onixs.cme.md.handler.CmeMdHandler#getFixDialectDescriptionFile |
Path the CME FIX dialect description file. | "CmeFastFixDialect.xml" |
biz.onixs.cme.md.handler.CmeMdHandler#setImpliedBookDefaultDepth biz.onixs.cme.md.handler.CmeMdHandler#getImpliedBookDefaultDepth |
Implied book default depth. | 2 |
biz.onixs.cme.md.handler.CmeMdHandler#setIncrementalCacheSize biz.onixs.cme.md.handler.CmeMdHandler#getIncrementalCacheSize |
Incremental packets cache size. This cache are used for packet alignment. | 5 |
biz.onixs.cme.md.handler.CmeMdHandler#setLicenseFile biz.onixs.cme.md.handler.CmeMdHandler#getLicenseFile |
Path to the license file. | "OnixS.lic" |
biz.onixs.cme.md.handler.CmeMdHandler#setLocalNetworkInterface biz.onixs.cme.md.handler.CmeMdHandler#getLocalNetworkInterface |
Local network interface that is used to receive UDP multicast packets. If set to null then all available local network interfaces are used. | null |
biz.onixs.cme.md.handler.CmeMdHandler#setLocalNetworkInterfaceA biz.onixs.cme.md.handler.CmeMdHandler#getLocalNetworkInterfaceA |
Local network interface that is used to receive UDP multicast packets for "A" feeds. If set to null then the LocalNetworkInterface value is used. | null |
biz.onixs.cme.md.handler.CmeMdHandler#setLocalNetworkInterfaceB biz.onixs.cme.md.handler.CmeMdHandler#getLocalNetworkInterfaceB |
Local network interface that is used to receive UDP multicast packets for "B" feeds. If set to null then the LocalNetworkInterface value is used. | null |
biz.onixs.cme.md.handler.CmeMdHandler#
setMaximumNumberOfQueuedIncrementalRefreshMessages biz.onixs.cme.md.handler.CmeMdHandler#
getMaximumNumberOfQueuedIncrementalRefreshMessages |
Maximum number of queued Market Data Incremental Refresh (X) messages. When the message gap is detected all subsequent Refresh(X) messages are queued until the Handler re-synchronizes with the market using Snapshot(W) messages. After that the queued Refresh(X) messages are re-played. This setting could be used to limit the memory usage when there is no Market Data - Snapshot/Full Refresh (W) messages. | 10000 |
biz.onixs.cme.md.handler.CmeMdHandler#setReceiveTimeoutInMilliseconds biz.onixs.cme.md.handler.CmeMdHandler#getReceiveTimeoutInMilliseconds |
Amount of time in milliseconds a Handler will wait to receive data once a read operation is initiated. | 36000 |
biz.onixs.cme.md.handler.CmeMdHandler#setRecoverSecurityDefinitionsOnGap biz.onixs.cme.md.handler.CmeMdHandler#getRecoverSecurityDefinitionsOnGap |
The flag to recover the Security Definition messages each time when the message sequence gap is detected. | false |
biz.onixs.cme.md.handler.CmeMdHandler#setReportNoDataWarning biz.onixs.cme.md.handler.CmeMdHandler#getReportNoDataWarning |
Report-no-data option value. | false |
biz.onixs.cme.md.handler.CmeMdHandler#setSecurityDefinitionsCacheRoot biz.onixs.cme.md.handler.CmeMdHandler#getSecurityDefinitionsCacheRoot |
Path where security definition cache will be stored. | "SecurityDefinitionsCache" |
biz.onixs.cme.md.handler.CmeMdHandler#setSocketBufferSize biz.onixs.cme.md.handler.CmeMdHandler#getSocketBufferSize |
UDP socket buffer size. | 65536 |
biz.onixs.cme.md.handler.CmeMdHandler#setTcpReplay biz.onixs.cme.md.handler.CmeMdHandler#getTcpReplay |
Set option to use the TcpReplay Channel for recovery of missed messages. This method recovers all missed messages. | false |
biz.onixs.cme.md.handler.CmeMdHandler#setTcpReplayPassword biz.onixs.cme.md.handler.CmeMdHandler#getTcpReplayPassword |
Password or passphrase to be used in the Logon (35=A) message from customer to CME Tcp Replay service. | ONIXS |
biz.onixs.cme.md.handler.CmeMdHandler#setTcpReplayReconnectAttempts biz.onixs.cme.md.handler.CmeMdHandler#getTcpReplayReconnectAttempts |
Number of attempts to receive missed messages via CME Tcp Replay service. | 10 |
biz.onixs.cme.md.handler.CmeMdHandler#
setTcpReplayReconnectIntervalInMilliseconds biz.onixs.cme.md.handler.CmeMdHandler#
getTcpReplayReconnectIntervalInMilliseconds |
Interval between attempts to receive missed messages via CME Tcp Replay service (in milliseconds). | 100 |
biz.onixs.cme.md.handler.CmeMdHandler#setTcpReplayUsername biz.onixs.cme.md.handler.CmeMdHandler#getTcpReplayUsername |
User ID or username to be used in the Logon (35=A) message from customer to CME Tcp Replay service. | ONIXS |
biz.onixs.cme.md.handler.CmeMdHandler#setUseIncrementalFeedA biz.onixs.cme.md.handler.CmeMdHandler#getUseIncrementalFeedA |
The flag to use or not incremental feed "A" option. | true |
biz.onixs.cme.md.handler.CmeMdHandler#setUseIncrementalFeedB biz.onixs.cme.md.handler.CmeMdHandler#getUseIncrementalFeedB |
The flag to use or not incremental feed "B" option. | false |
biz.onixs.cme.md.handler.CmeMdHandler#setUseInstrumentReplayFeedA biz.onixs.cme.md.handler.CmeMdHandler#getUseInstrumentReplayFeedA |
The flag to use or not instrument feed "A" option. | true |
biz.onixs.cme.md.handler.CmeMdHandler#setUseInstrumentReplayFeedB biz.onixs.cme.md.handler.CmeMdHandler#getUseInstrumentReplayFeedB |
The flag to use or not instrument feed "B" option. | false |
biz.onixs.cme.md.handler.CmeMdHandler#setUseOneReceiverOnSamePort biz.onixs.cme.md.handler.CmeMdHandler#getUseOneReceiverOnSamePort |
True, if one multicast socket should be created for A nd B feeds, if ports are same. | false |
biz.onixs.cme.md.handler.CmeMdHandler#setUseSnapshotFeedA biz.onixs.cme.md.handler.CmeMdHandler#getUseSnapshotFeedA |
The flag to use or not Market Recovery (UDP) Feed "A". | true |
biz.onixs.cme.md.handler.CmeMdHandler#setUseSnapshotFeedB biz.onixs.cme.md.handler.CmeMdHandler#getUseSnapshotFeedB |
The flag to use or not Market Recovery (UDP) Feed "B". | false |
Listening to Market Data
Events in Handler
Once Handler is started, it listens to a market data from the network, then processes it (decodes messages, validates message order, updates order books) and invokes client code for further processing.Handler processes market data working asynchronously and uses concept of events and event listeners to notify client code about a particular occasion like reception of security definition or direct book update.
- Note
- We recommend not to call handler methods which change handler state (like handler.stop()) from event listeners.
Listening to a particular Event
For each event like Error Occurred Handler provides the following interface likebiz.onixs.cme.md.handler.event.ErrorListener
. Client code must implement this interface to be able to handle events of a particular type. Handler also exposes a member like biz.onixs.cme.md.handler.CmeMdHandler#registerErrorListener
which allows to associate an instance of the event handler with a particular instance of the biz.onixs.cme.md.handler.CmeMdHandler
class.
- Note
- Associating listener for a particular event with an instance of the
biz.onixs.cme.md.handler.CmeMdHandler
class must be performed while Handler is in stopped stated. Once the Handler is started, changing listener-event associations is not allowed, and may lead to unpredictable behavior as well as may cause unexpected errors.
biz.onixs.cme.md.handler.CmeMdHandler
class is also called subscribing to an event.
Primary High-level Events
There are multiple events exposed by Handler. All events can be logically divided onto high- and low- level event sub-set. High-level events reflect various results of market data processing done by Handler like Security Definition Received, Market Book Updated, and Trading occurred. Low-level events are designed for more control over data processing and for more flexibility. They are described in different sections of this documentation.The table below describes primary high-level events exposed by Handler. It also depicts correspondence between events, interfaces for listeners and thes Handler's members to subscribe to an event.
Event | Listener interface to be implemented | Handler member to register listener | Description |
---|---|---|---|
Security Definition Received | biz.onixs.cme.md.handler.event.SecurityDefinitionListener |
biz.onixs.cme.md.handler.CmeMdHandler#registerSecurityDefinitionListener |
Fired if a definition of a new security is received. Security definitions normally are received at the beginning of the Handler's execution and before any other high-level event like Book Updated event. |
Security Status Changed | biz.onixs.cme.md.handler.event.SecurityStatusChangedListener |
biz.onixs.cme.md.handler.CmeMdHandler#registerSecurityStatusListener |
Fired if the remote system reports about changes in a status of a particular security. An example of such change is update of upper price or trading status. |
Direct Book Updated | biz.onixs.cme.md.handler.event.RegularBookUpdatedListener |
biz.onixs.cme.md.handler.CmeMdHandler#registerRegularBookUpdatedListener |
Occurs when direct book of a particular security is updated. |
Direct Book Changed | biz.onixs.cme.md.handler.event.RegularBookChangedListener |
biz.onixs.cme.md.handler.CmeMdHandler#registerRegularBookChangedListener |
Occurs when direct book of a particular security is changed. |
Implied Book Updated | biz.onixs.cme.md.handler.event.ImpliedBookUpdatedListener |
biz.onixs.cme.md.handler.CmeMdHandler#registerImpliedBookUpdatedListener |
Occurs when implied book of a particular security is updated. |
Implied Book Changed | biz.onixs.cme.md.handler.event.ImpliedBookChangedListener |
biz.onixs.cme.md.handler.CmeMdHandler#registerImpliedBookChangedListener |
Occurs when implied book of a particular security is changed. |
Consolidated Book Updated | biz.onixs.cme.md.handler.event.ConsolidatedBookUpdatedListener |
biz.onixs.cme.md.handler.CmeMdHandler#registerConsolidatedBookUpdatedListener |
Occurs when consolidated book of a particular security is updated. |
Consolidated Book Changed | biz.onixs.cme.md.handler.event.ConsolidatedBookChangedListener |
biz.onixs.cme.md.handler.CmeMdHandler#registerConsolidatedBookChangedListener |
Occurs when consolidated book of a particular security is changed. |
Statistics Changed | biz.onixs.cme.md.handler.event.StatisticsListener |
biz.onixs.cme.md.handler.CmeMdHandler#registerStatisticsListener |
Occurs when attributes like opening or closing price for a particular security are changed. |
Trade Information Received | biz.onixs.cme.md.handler.event.TradeListener |
biz.onixs.cme.md.handler.CmeMdHandler#registerTradeListener |
Occurs if trade information is received for a particular security. Trade attributes are collected into an instance of biz.onixs.cme.md.handler.event.TradeEventArgs class which is supplied as a parameter to the listener by Handler. |
News Received | biz.onixs.cme.md.handler.event.NewsReceivedListener |
biz.onixs.cme.md.handler.CmeMdHandler#registerNewsReceivedListener |
Fired when the system spreads arbitrary news to Handler. Headline and full text of the news are exposed via an instance of biz.onixs.cme.md.handler.event.NewsReceivedEventArgs class. |
Example
The following sample demonstrates how to listen to notifications about security definition reception.
public class MyListener implements SecurityDefinitionListener { private static final Logger LOG = LoggerFactory.getLogger(MyListener.class); public void onSecurityDefinition(SecurityDefinitionEventArgs args) { LOG.debug("onSecurityDefinition(): {}", args); } } public class Sample { private void run() throws Exception { final CmeMdHandler handler = new CmeMdHandler ("30"); // ... final MyListener listener = new MyListener(); handler.registerSecurityDefinitionListener(listener); } }
Manipulating Books
Books in Handler
Handler builds and maintains books for each security whose definition was previously received in bounds of a particular market data channel.
All books, maintained by Handler, represent price-level books. Book abstraction is exposed by the biz.onixs.cme.md.handler.IOrderBook
interface.
There are several types of books that are available for each security: direct, implied and consolidated. Handler builds and maintains all type of books for all securities.
Once a book of any type and for any security is updated, Handler notifies client code using appropriate event:
- To get notified about updates into a direct book for a particular security, implement
biz.onixs.cme.md.handler.event.RegularBookUpdatedListener
interface and register instance in the Handler usingbiz.onixs.cme.md.handler.CmeMdHandler#registerRegularBookUpdatedListener
member. - To get notified about updates into an implied book for a particular security, implement
biz.onixs.cme.md.handler.event.ImpliedBookUpdatedListener
interface and register instance in the Handler usingbiz.onixs.cme.md.handler.CmeMdHandler#registerImpliedBookUpdatedListener
member. - To get notified about updates into a consolidated book for a particular security, implement
biz.onixs.cme.md.handler.event.ConsolidatedBookUpdatedListener
interface and register instance in the Handler usingbiz.onixs.cme.md.handler.CmeMdHandler#registerConsolidatedBookUpdatedListener
member.
Instance of a book for a particular security is passed through a parameter of event listener interface. Although Handler differs a book update event for each type of a book, biz.onixs.cme.md.handler.IOrderBook#getType
member can be used to identify whether given instance of the book is direct, implied or consolidated.
The depth of a book of a particular type is usually defined by security definition and is exposed by biz.onixs.cme.md.handler.IOrderBook#getDepth
member. Returned value defines maximum number of price levels that may exist in the book.
biz.onixs.cme.md.handler.IOrderBook#getBids
and biz.onixs.cme.md.handler.IOrderBook#getAsks
members of the biz.onixs.cme.md.handler.IOrderBook
class can be used to access price value and quantity information for all levels available in book.
Behavioral Traits
Handler constructs books for each security when it receives Security Definition message for that security. Books are changed in time and their members are not thread-safe.
Handler uses its own synchronization strategies while updating the books. Therefore, do not access price levels information outside of book-related event handling as well as do not store references to any book instance and data returned by its members like asks and bids collections.
It happens because Handler may change internal book structures at the same time. If any data, which is exposed by the biz.onixs.cme.md.handler.IOrderBook
class instance, must be used outside of event handling, it should be copied before later using. To construct immutable copy of the book for using outside of callback, use the biz.onixs.cme.md.handler.IOrderBook#clone
member.
See Listening to Market Data topic for more information concerning how to subscribe for a book updates.
Building and Maintaining Books by Yourself
Books Building Bricks
As it was mentioned in the previous topic, Handler builds and maintains books for each security by itself and notifies client code once book update takes place.
Sometimes there is a necessity to build and maintain books by own forces (for example, if there's no need in information of the lowest price-levels). For this purpose, Handler provides advanced events, which contain detailed information about the changes, needed to be applied to a book, to bring it into an actual state.
To get notified about detailed changes into a direct book for a particular security, implement biz.onixs.cme.md.handler.event.RegularBookChangedListener
interface and register it in the Handler via biz.onixs.cme.md.handler.CmeMdHandler#registerRegularBookChangedListener
member.
To get notified about detailed changes into an implied book for a particular security, implement biz.onixs.cme.md.handler.event.ImpliedBookChangedListener
interface and register it in the Handler via biz.onixs.cme.md.handler.CmeMdHandler#registerImpliedBookChangedListener
member.
Both events expose detailed information of elementary change into book's bids and asks.
- Note
- There is no ability to listen to changes for consolidated books because these books are derived from direct and implied ones.
Book Change Machinery and Relation with 'Book Updated' Event
Book Updated and Book Changed events are not mutually exclusive ones. It is possible to subscribe to both types of events. Book changes represent more elementary (atomic) updates of the book. Therefore, Book Changed events may occur more frequently in compare to Book Updated event occasions. In fact, series of Book Changed events occur before single occasion of Book Updated event.
When an order book is delivered via Book Updated callbacks, it is always in a valid and up-to-date state. In contrast, a book change is an elementary action over a book and there're usually multiple changes inside single snapshot and/or incremental refresh. For this reason, the book may not be in a valid state between two changes. Only when all the changes are processed from the single network packet (a message like snapshot and incremental refresh), the book can be considered as valid. Book Updated callbacks are called exactly at the time when all changes are processed and the book appears to be valid and up-to-date.
To build an order book properly, it is also necessary to listen to Handler State changes. When Handler falls into biz.onixs.cme.md.handler.HandlerState#BOOKS_RESYNCHRONIZATION_STARTED
, all books must be empty. Afterwards, changes that are reported via Book Changed callbacks, must be applied to build a correct order book. An important aspect is that Handler doesn't differ changes from snapshots and incremental refreshes. From the API perspective, all the changes, reported through callbacks, are of the same structure. To detect the source of a change (e.g. snapshot and/or incremental refresh) as well as to determine the bounds of changes transaction it is necessary to subscribe to Message Processing events (Raw Market Data Processing).
- Note
- Handler exposes members in listening interfaces like
biz.onixs.cme.md.handler.event.RegularBookChangedListener#onRegularBookChangesReset
. These members can be used to reset all books instead of using Handler state listening.
Customizing Book Changes/Updates Notifications
Tracking the Best Bids and Asks Only
By default, Handler fires Book Changed and Book Updated events each time it receives incremental updates for a particular book from the remote system. In certain cases, there is a necessity to reduce amount of events, generated by Handler. In particular, it is quite often that only the top of a book is a point of intersect. For this reason, Handler allows to get notified only about changes and/or updates into the best bids and asks.
To take advantage of such facility, use overloads which accept instance of biz.onixs.cme.md.handler.BestBidAskTrackingOptions
class and register listeners like biz.onixs.cme.md.handler.event.TopOfTheRegularBookUpdatedListener
biz.onixs.cme.md.handler.BestBidAskTrackingOptions#setBestBidAskTrackingParameters
bit field specifies, of which available bid/ask attributes (e.g. price or quantity or both) should be tracked for changes. Handler will report about changes and/or updates into a book only if specified parameters of the best bid/ask were changed.
Other members of biz.onixs.cme.md.handler.BestBidAskTrackingOptions
class allow to control sensitivity of Handler to the changes and updates.
In particular, biz.onixs.cme.md.handler.BestBidAskTrackingOptions#setPriceThreshold
member allows to define sensitivity threshold for the best bid/ask price values exposed in percent. If price of the best bid or ask changes for more than specified value (in percent), Handler will raise appropriate event. In other case, it will not do it.
Similarly, biz.onixs.cme.md.handler.BestBidAskTrackingOptions#setQuantityThreshold
member controls occasion of Top of The Book Updated event, in case the best bid/ask quantity value changes.
Example
The following example demonstrates how to configure Handler to get notified only if price of best bid and ask changes for more than 10 percent or quantity changes for more than 50 percent.
public class MyListener implements TopOfTheRegularBookUpdatedListener { private static final Logger LOG = LoggerFactory.getLogger(MyListener.class); public void onTopOfTheRegularBookUpdated(TopOfTheOrderBookUpdatedEventArgs args) { LOG.debug("onTopOfTheRegularBookUpdated(): {}", args); } } public class Sample { private void run() throws Exception { final CmeMdHandler handler = new CmeMdHandler ("30"); handler.getBestBidAskTrackingOptions().setBestBidAskTrackingParameters(BestBidAskTrackingParameter.PRICE | BestBidAskTrackingParameter.QUANTITY); handler.getBestBidAskTrackingOptions().setPriceThreshold(new ScaledDecimal("10")); handler.getBestBidAskTrackingOptions().setQuantityThreshold(50); // ... final MyListener listener = new MyListener(); handler.registerTopOfTheRegularBookUpdatedListener(listener); } }
Filtering Securities
Defining Securities of Interest
By default, Handler processes market data for all securities that are available in a particular market data channel. For certain channels, it may be a huge amount of data which may cause Handler to allocate significant amount of system resources (like a memory). On the other side, it often happens that all securities are not the point of interest. For these reasons, Handler provides an ability to define a subset of securities for which it will monitor market data, maintain books and report about changes in statistics. No events will be fired by Handler for any security it they are not presented in the defined subset. This feature is called as securities filtering.
Operating over securities filter is simple. biz.onixs.cme.md.handler.CmeMdHandler#addSecurityIdFilter
member must be used to put a security into the collection of securities for which Handler must process market data. There are no limits on quantity of securities which can be included into filtering. Client code may be put from single up to all securities that are available in the single market data channel.
If at least one security is added to the filter, Handler stops raising events for all securities that are available in the market data channel. From that moment Handler will fire events only for securities that are included into filtering.
To remove security from the filter, it's necessary to call biz.onixs.cme.md.handler.CmeMdHandler#removeSecurityIdFilter
member. Handler will continue processing market data and raising events only for the securities that remain in the filter.
If collection of securities in the filter becomes empty, Handler starts processing market data and event notifying for all securities in the market data channel.
To ensure that no securities are available in the securities filter, use biz.onixs.cme.md.handler.CmeMdHandler#clearSecurityIdFilters
member. It will clear the filter.
In the same way securities can be filtered by Security Description, Security Group and Symbol.
- Note
- Filtering must be manipulated only when Handler is in
biz.onixs.cme.md.handler.HandlerState#STOPPED
,biz.onixs.cme.md.handler.HandlerState#STARTED
or inbiz.onixs.cme.md.handler.HandlerState#SECURITY_DEFINITIONS_RECOVERY_STARTED
state. Once Handler changes its state to any other one, filters must not be modified.
Filtering Securities using Sub-Channel Identifiers
In FIX/FAST 2.0 there's an ability to filter securities using sub-channel information (identifiers). Handler provides such facility in the same way as it does for single securities. There are three members that affect sub-channel filtering:
biz.onixs.cme.md.handler.CmeMdHandler#addSubchannelFilter
puts a sub-channel id into the collection of sub-channels for which Handler must process market data.biz.onixs.cme.md.handler.CmeMdHandler#removeSubchannelFilter
excludes single sub-channel id from the set of sub-channels for which Handler processes market data.biz.onixs.cme.md.handler.CmeMdHandler#clearSubchannelFilters
removes all sub-channel filters and restores default behavior when all the market data is processed by Handler.
Example
The following code demonstrates how to instruct Handler to process data only for two particular securities:
public class Sample { private void run() throws Exception { final CmeMdHandler handler = new CmeMdHandler ("30"); handler.addSubchannelFilter("1"); handler.addSecurityIdFilter("05930"); handler.addSecurityIdFilter("05931"); handler.start(); // ... handler.stop(); handler.clearSecurityIdFilters(); } }
Error Handling
Error Handling Concept
Being presented as a Java library, Handler uses exceptions to report about error occurred while performing certain action. For example, Handler will raise regular exception to report about inability to find actual license for the product. However, as far as Handler processes market data asynchronously, it is not able to report about any further errors from the moment market data processing is started. For this reason, Handler exposesbiz.onixs.cme.md.handler.event.ErrorListener
interface and biz.onixs.cme.md.handler.CmeMdHandler#registerErrorListener
member to be able to subscribe to and handle errors.
Once instance of biz.onixs.cme.md.handler.event.ErrorListener
is assigned to Handler, it will invoke biz.onixs.cme.md.handler.event.ErrorListener#onError
member each time whe the error occurs. biz.onixs.cme.md.handler.event.ErrorListener#onError
member has several incoming parameters one of which defines human-readable explanation (description) of the error.
An important aspect of the Handler's behavior is that once error occurred Handler will automatically restart market data processing starting from security definitions.
Example
The following sample demonstrates how to receive error notifications.
public class MyListener implements ErrorListener { private static final Logger LOG = LoggerFactory.getLogger(MyListener.class); public void onError(ErrorEventArgs args) { LOG.debug("onError(): {}", args); } } public class Sample { private void run() throws Exception { final CmeMdHandler handler = new CmeMdHandler("30"); // ... final MyListener listener = new MyListener(); handler.registerErrorListener(listener); } }
Licensing the Handler
Specifying License Location
Handler needs a license for successful execution. If Handler is not able to find a valid license, it will throw an exception at the initialization stage.
biz.onixs.cme.md.handler.CmeMdHandler
class exposes biz.onixs.cme.md.handler.CmeMdHandler#setLicenseFile
member that allows to instruct Handler where to look for a valid license. By default, Handler looks for a license in the current directory for the application that it uses, home directory of current user and a class path. However, by using noted parameter, it's possible to specify another folder anywhere on a file system.
- Note
- Handler looks for a valid license in the specified folder and selects the best one. If multiple licenses are available, it will select the most significant one.
Example
The following example demonstrates how we can supply license to Handler:
public class Sample { private void run() throws Exception { CmeMdHandler.setLicenseFile("license.lic"); final CmeMdHandler handler = new CmeMdHandler("30"); } }
Understanding States
Handler States
Once Handler is started, it processes securities definitions that are retrieved either from multicast channel or previously recorded local cache file. Afterwards, Handler starts listening for a book-related market data. At first, it reconstructs all books (direct, implied or both) for all available securities. When reconstruction is completed , Handler listens for an incremental updates and processes them accordingly. To check in which state Handler is currently, it's necessary to implement biz.onixs.cme.md.handler.event.HandlerStateChangedListener
interface and associate an instance with the Handler using biz.onixs.cme.md.handler.CmeMdHandler#registerStateChangedListener
member.
The table below describes all possible states for Handler:
State | Description |
---|---|
biz.onixs.cme.md.handler.HandlerState#STOPPED |
Handler is stopped or was not executed yet. |
biz.onixs.cme.md.handler.HandlerState#SECURITY_DEFINITIONS_RECOVERY_STARTED |
Handler started gathering security definitions either from multicast feed or previously stored cache file. |
biz.onixs.cme.md.handler.HandlerState#SECURITY_DEFINITIONS_RECOVERY_FINISHED |
Handler accomplished processing of security definitions. In normal flow Handler moves to the BooksResynchronizationStarted right after it achieves SecurityDefinitionsRecoveryFinished state. However, in certain cases (for example, in case of error while processing security definitions) Handler may move back to the SecurityDefinitionsRecoveryStarted state. |
biz.onixs.cme.md.handler.HandlerState#BOOKS_RESYNCHRONIZATION_STARTED |
Handler switches to this state when it joins the system for a market data and starts building the books for all securities whose definitions were obtained on previous stage. Moreover, Handler may move into this stage in case of message gap detection while it is maintaining books.
|
biz.onixs.cme.md.handler.HandlerState#BOOKS_RESYNCHRONIZATION_FINISHED |
Handler accomplishes process of books resynchronization. From this moment all books are valid. They are in the up-to-date state. Handler continues to listen to any changes in the books and notifies about updates through appropriate listeners. |
biz.onixs.cme.md.handler.HandlerState#TCP_REPLAY_STARTED |
Because of multicast unreliability, message gap may happen while processing market data. If Handler is configured to use TCP Replay facility, it suspends regular processing, moves to this state and requests remote system to resend missed messages. |
biz.onixs.cme.md.handler.HandlerState#TCP_REPLAY_FINISHED |
Handler switches into this state in case of success recover from message gap using TCP replay facility. Afterwards, normal market data processing is restored and Handler continues listening for books updates. |
Listening to Warnings
Warnings Concept
Miscellaneous non-critical issues may occur while Handler is being executed. Handler will handle such issues by itself, thus no special handling is required for such cases. However, sometimes it's reasonable to be notified about such events.
For this reason, handler exposes biz.onixs.cme.md.handler.event.WarningListener
class and will invoke its biz.onixs.cme.md.handler.event.WarningListener#onWarning
member each time a non-critical issue will take place.
Security Definitions Cache
Security Definitions Cache Concept
Once Handler is started it will listen to security definition channel and will receive all security definitions. Amount of security definitions can be large, so Handler startup time will be significantly increased. To prevent this, security definitions cache can be used.
If cache is enabled, Handler will try to load security definitions from cache instead of listening to the instrument channel during startup. If cache is absent or broken, Handler will start listening to the instrument channel. Then it will receive all security definitions, override cache and stop listenint to the instrument channel.
If cache is disabled, Handler will start listening to the instrument channel. Then it will receive all security definitions and stop listening to the instrument channel.
To enable security definitions we can use an overloaded biz.onixs.cme.md.handler.CmeMdHandler#start
member with parameter cacheSecurityDefinitions that is set to true.
- Note
- If Handler is restarted during the working week, a security definitions cache will be able to miss some security definitions (which were added in the middle of the week). In this case, the cache file must be manually deleted and Handler must be restarted.
Example
The following sample shows the case when Security Definitions caching is enabled:
public class Sample { private void run() throws Exception { final CmeMdHandler handler = new CmeMdHandler("30"); handler.start(true); } }
Raw Market Data Processing
Processing Raw FIX Messages
High-level Handler's API allows to get rid of protocol specific and uses high-level entities directly like security definitions, trading information and books. Advanced events are exposed by Handler to allow the client code to build books by itself using data of atomic changes.
Handler exposes additional level of flexibility by providing ability to process FIX messages (raw market data) as they come from the remote system.
To listen for such kind of data (raw FIX messages), it's necessary to implement biz.onixs.cme.md.handler.event.MessageProcessingListener
class interface and associate appropriate instance with the Handler using biz.onixs.cme.md.handler.CmeMdHandler.registerMessageProcessingListener
member.
Handler notifies the client code when it begins and accomplishes processing of a message. biz.onixs.cme.md.handler.event.MessageProcessingListener#onMessageProcessingStarted
member is called when Handler starts processing a message.
biz.onixs.cme.md.handler.event.MessageProcessingListener#onMessageProcessingFinished
is called when message processing is accomplished.
- Note
- FIX messages which are passed to the listener's members by Handler are previously verified and checked for a proper order. If Handler notifies listener about starting of message processing, its guaranteed message is not out of sequence. However, messages, that are obtained from different network sources (different feeds) have uncorrelated sequence numbers.
Thread Specific
Handler may call biz.onixs.cme.md.handler.event.MessageProcessingListener
members simultaneously from different threads.
Invocation of any member of biz.onixs.cme.md.handler.event.MessageProcessingListener
class is not synchronized by Handler.
Example
The following sample prints security description from instrument definition message:
public class MyListener implements MessageProcessingListener { private static final Logger LOG = LoggerFactory.getLogger(MyListener.class); public void onMessageProcessingStarted(MessageProcessingEventArgs args) { if (args.getMessage().getType() == "d") LOG.debug("Security Description: {}", args.getMessage().get(FIX50_SP2.Tag.SecurityDesc)); } public void onMessageProcessingFinished(MessageProcessingEventArgs args) { } }
Controlling Handler Logging
Controlling Logging in the Handler
By default, Handler logs all important aspects of its activity while processing market data. The SLF4J (Simple Logging Facade for Java) is used by Handler internally. The SLF4J detects and uses the concrete logging implementation configured by user. By default, the Logback logging implementation is recommended.
The Logback (or any other logging implementation) must be configured at the beginning of the application. The examples of Logback configuration can be found in the Handler samples. The details of Logback configuration can be found here.
By default logging is configured to log only errors and warnings. This is an example of logback.xml where detailed logging was switched on:
<logger name="biz.onixs" level="INFO"/>
<logger name="biz.onixs.cme.md.handler" level="INFO"/>
<logger name="biz.onixs.cme.md.handler.OrderBookRepository" level="DEBUG"/>
<logger name="biz.onixs.cme.md.handler.InstrumentRepository" level="DEBUG"/>
<logger name="biz.onixs.cme.md.handler.sample" level="DEBUG"/>
Using TCP Replay Facility
Recovering from Message Gaps
Handler receives market data through network multicast feeds. Due to the multicast nature, data may come in wrong order or may be completely lost. Handler does its best to fix all the network-related issues. However, message gaps may still happen while processing market data.
CME market data system provides its clients with an ability to request and obtain lost messages using reliable TCP connection. This facility is called a TCP Replay. If the Handler is configured to use TCP Replay facility, in case of message lost, it doesn't report error immediately and falls into books resynchronization. Instead, it suspends regular processing and requests remote system to resend missed messages. Once lost messages are successfully received from the system, Handler resumes regular processing without restarting its execution.
Enabling TCP Replay
To use TCP Replay facility, first of all, it's necessary to contact CME support for TCP Replay credentials. Once login and password information are obtained, they can be passed to Handler and TCP Replay feature activated.
The following example demonstrates how to enable to use TCP Replay feature in Handler:
public class Sample { private void run() throws Exception { final CmeMdHandler handler = new CmeMdHandler("30"); // Take advantage of TCP Replay feature. handler.setTcpReplay(true); // Fill connection credentials. handler.setTcpReplayUsername("MyLogin"); handler.setTcpReplayPassword("MyPassword"); // In case of absence of a response from the remote // system, Handler will try to reconnect three times // with half-of-minute interval between each attempt // before it reports about inability to establish // connection. handler.setTcpReplayReconnectAttempts(3); handler.setTcpReplayReconnectIntervalInMilliseconds(30000); } }
Replaying Log Files
Using Logs to Replay Market Data
In normal flow, Handler logs all important aspects of its execution onto the file system. Log files also include original market data that is processed by Handler. This information is usually saved for analysis of non-standard situations, which may occur during the using of Handler. However, it can be also used to reproduce normal Handler's behavior for a certain period of time.
Once Handler was executed with logging enabled, it is possible to use log files for further replay. First of all, logs must be backed up (copied to another location) or the Handler's configuration must be updated to use another directory for new logs. It happens because current implementation of Handler doesn't support replay from the same folder, in which new logs will be stored.
- Note
- Handler will not replay log files if packet logging are not enabled. To enable packet logging set biz.onixs.cme.md.handler.Feed in logback.xml at INFO or DEBUG level:
<logger name="biz.onixs.cme.md.handler" level="INFO"/>
or
<logger name="biz.onixs.cme.md.handler.Feed" level="DEBUG"/>
Afterwards, instance of biz.onixs.cme.md.handler.ReplayOptions
class must be constructed and initialized with the logs to replay.
When configuring the options is accomplished, replay can be started. For this purpose, biz.onixs.cme.md.handler.CmeMdHandler#start
overload is to be called to start replaying. Replaying is performed by Handler asynchronously in the same way as regular execution flows. From the client code point of view, there's no difference whether Handler processes market from the network or from log files.
- Note
- Handler fires appropriate events as soon as market data is received from the network and processed. However, since CME environment spreads market data with different frequency which depends on live market activity, there're delays of different duration between fired events. When log file is replayed, Handler does not do any pauses and fires corresponding events as soon as data is retrieved from the log file.
To stop Handler to replay log files, biz.onixs.cme.md.handler.CmeMdHandler#stop
member can be used.
When replay is accomplished Handler will call biz.onixs.cme.md.handler.event.ReplayListener#onReplayFinished
member of the biz.onixs.cme.md.handler.event.ReplayListener
class. To get notified about this and other replay-related events it's necessary to implement biz.onixs.cme.md.handler.event.ReplayListener
interface and supply an instance of the derived class to the Handler using biz.onixs.cme.md.handler.CmeMdHandler#registerLogReplayListener
member.
- Note
- When market data is being replayed, message receiving time will return time of corresponding log file entry. That is the value that returned the represent original time, when market data message was obtained from the network.
Currently, log replay must be run with the same set of Handler's settings that were originally used.
Example
The following example demonstrates how to replay log files backed up into 'replay' folder:
public class MyListener implements ReplayListener { private static final Logger LOG = LoggerFactory.getLogger(MyListener.class); public void onReplayError(ErrorEventArgs args) { LOG.debug("onReplayError(): {}", args); } public void onReplayFinished() { LOG.debug("onReplayFinished()"); } } public class Sample { private void run() throws Exception { final CmeMdHandler handler = new CmeMdHandler("30"); handler.registerLogReplayListener(new MyListener()); handler.start(BooksMaintenanceOptions.REGULAR_BOOKS_ONLY, new ReplayOptions("path_to_log")); // ... handler.stop(); } }
Connectivity Troubleshooting
Issues Related with Receiving Multicast Data
In most cases, absence of multicast data is caused by network-related configuration issues that are described below.
Firewall Blocks Program and GettingStarted Sample
Updating firewall rules or simply turning it off usually solves the issue.
- Note
- Disabling firewall explicitly may be dangerous. Consult your network and system administrator before turning firewall off.
Linux-like Operating System Turns on Reverse Path Filtering on Network Interfaces
Disabling reverse path filtering by updating system control parameters usually resolves the issue.
System control parameters (/etc/sysctl.conf) affecting reverse path filtering:
Parameter | Description |
---|---|
net.ipv4.conf.default.rp_filter | Defines default policy. |
net.ipv4.conf.<NetworkInterface>.rp_filter | Defines policy for a <NetworkInterface> network interface. |
To determine current status of a reverse path filtering, we must run the command below:
To disable a reverse path filtering from console, we must run the following command:
Explicit Network Interface Must Be Specified
The sample program configures Handler to listen to market data on all network interfaces on Windows and Linux. If network configuration supposes the specifying exact network interface (like 'eth0' on Linux) for market data listening, then biz.onixs.cme.md.handler.CmeMdHandler#setLocalNetworkInterface
member value must be updated accordingly.
FAQ
After switching the Operating System even the sample program is no longer able to receive market data. However, I am able to ping the CME router and iLink server. Any insight or suggestions on what may be going on here?
See Connectivity Troubleshooting.
If Handler joins while the market is busy, warnings Packet Cache Overflow are reported. Is there any possibility to raise the configured limit, if so how it can be done?
When Handler performs book resynchronization, it caches all the data that are received on incremental feeds. When the market is busy, the number of cached messages may exceed the configured limit defined by biz.onixs.cme.md.handler.CmeMdHandler#setMaximumNumberOfQueuedIncrementalRefreshMessages
parameter value. So, to avoid 'PacketCacheOverflow' warnings, it is necessary to increase value of the noted parameter which equals to 10000 by default.
What's the difference between "Book Changed" and "Book Updated" events?
If you subscribe to Book Updated, you will always have a consistent book. Book Changed callbacks are designed for those who want to build and maintain books by themselves. Book Changed is an elementary action over the order book and usually there are multiple changes inside single snapshot and/or incremental refresh. For this reason, the book may not be valid between two changes. If only all the changes are processed from the single network packet (message like snapshot and incremental refresh), the book is considered as valid. Book Updated callbacks are called exactly at the a time when all changes are processed and book appears to be valid and up-to-date.
While using the onConsolidatedBookUpdated callback, we seem to see crosses. Can you please assist?
For the implied eligible products, one may see the cross book momentarily and the book becomes normal after the next transaction at the engine. So it is possible to see the crossed book in the biz.onixs.cme.md.handler.event.ConsolidatedBookUpdatedListener#onConsolidatedBookUpdated
callback even when the market is in the opened state.