An important issue that the users of OnixS C++ ICE iMpact Multicast Price Feed Handler library need to consider is the threading model that is used by the components of the application. The threading model of a library determines how the methods of listeners are assigned to threads for execution and synchronization.

Overview

The Handler processes market data coming from multicast feeds asynchronously. The threads that receive and process that data are owned by the Feed Engine, not by the Handler itself. To receive live market data, a Handler must be bound to a FeedEngine instance (see Setting Up Feed Engine) before it is started; it then runs on that engine's worker threads. Two kinds of Handler do not need a Feed Engine to process data: those that replay a recorded log or PCAP file, and those configured to download product definitions only (HandlerSettings::receiveProductDefinitionsOnly).

The size of the engine's thread pool is defined by the FeedEngineSettings::threadCount parameter. Its default value is 1, so, unless configured otherwise, a single worker thread receives, parses, and dispatches all market data. The worker threads are spawned when the FeedEngine is constructed and torn down when it is destroyed; no additional threads are spawned while it is running.

Handlers, the Feed Engine, Threads, and Sockets

The FeedEngine encapsulates all network-related aspects. Its worker threads own the underlying sockets: they create and join the multicast groups, wait for incoming data, and drain the sockets. Each subscribed feed opens its own socket. A subscription is served by a live feed and, while it recovers, a snapshot feed - so it uses up to two sockets, dropping to one once the books are resynchronized and the snapshot feed stops. A single worker thread can own and service many sockets, so, with the default single-thread configuration, one thread handles every subscription across every multicast group.

A FeedEngine instance is created explicitly and attached to a Handler with Handler::bindFeedEngine. Binding is only allowed while the Handler is stopped, and the engine must outlive every Handler bound to it. Because binding stores a reference rather than taking ownership, a single Feed Engine can be shared by multiple Handler instances, in which case all their feeds run on that engine's shared thread pool. Sharing one engine is the recommended default: it avoids allocating redundant threads and system resources per Handler.

If you need to dedicate a thread (or a set of CPU cores) to a particular group of markets - for example, to isolate a latency-sensitive subscription - create a separate FeedEngine instance with its own settings and bind the relevant Handler to it.

Feeds and Feed IDs

A subscription targets a multicast group through a specific channel - a combination of book depth (for example, Top 5 Price Level or Full Order Depth) and price type (implied or non-implied). The Handler serves each enabled channel with an internal feed client and assigns it a sequential feed ID (0, 1, ...). The number of feed IDs therefore equals the number of enabled (group x channel) combinations - for example, subscribing to the same book depth for both non-implied and implied prices in a single group already produces feed IDs 0 and 1.

Each feed client maintains two feeds: a snapshot feed and a live (incremental) feed. The snapshot feed recovers the current state of the order books while the live feed is buffered; once the books are resynchronized, the Handler switches to the live feed and stops the snapshot feed. Both feeds belong to the same feed client and therefore share a single feed ID - the snapshot and live feeds are not reported as separate IDs.

The FeedListener::onFeedStarted callback reports a feed client's feed ID when that client starts: when the subscription begins, and again if the Handler restarts the client, for example on an automatic reconnect or a session-number change. (A mid-session snapshot recovery does not restart the client, so it does not re-fire the callback.) Seeing several feed IDs is expected and does not indicate that several Handlers or several threads were created: all feed clients share the Feed Engine's worker threads, and no feed requires a thread of its own. See Event Listeners for the full set of feed callbacks.

Sizing the Thread Pool and CPU Affinity

The default FeedEngineSettings::threadCount of 1 is sufficient for most deployments, as the ICE iMpact feed volume is comfortably handled by a single thread. Increasing the thread count allows the engine to receive and process different feeds concurrently; reception of any single feed is still serialized. Consequently, additional worker threads only help when the engine services several feeds - with effectively a single active feed, the extra threads find no work and spend their time parked in the idle spin/sleep path.

To avoid unnecessary context switching between processors, the worker threads can be pinned to specific CPUs through the FeedEngineSettings::threadAffinity parameter. The same CPU set is applied to every worker thread of the engine. For advanced per-thread tuning (setting thread priority or a thread name), the FeedEngineListener callbacks are invoked in the context of the worker threads themselves. See Low Latency Best Practices and Setting Up Feed Engine for details.

Reducing Wake-up Latency

When there is no data to process, a worker thread would normally block while waiting for incoming packets, which can evict hot data and code from the CPU cache and slow down the next processing cycle. Two FeedEngineSettings parameters control this behavior:

FeedEngineSettings::dataWaitTime - how long, in milliseconds, the engine blocks while waiting for I/O. Setting it to 0 turns the wait into a busy poll.
FeedEngineSettings::spinBeforeIdleTime - how long, in milliseconds, a redundant worker keeps cycling before it goes to sleep once it runs out of work. It applies only when several worker threads share the engine (threadCount > 1); with the default single worker there is always incoming I/O to wait on, so the setting has no practical effect.

Both trade CPU consumption for lower latency. The Low Latency Best Practices topic explains how to tune them.

Listener Callback Threading

The Handler delivers callbacks from a few internal threads:

Feed Engine worker thread(s) deliver the live market-data callbacks - order-book changes, exchange messages received over multicast (including option, strategy, and expiry definitions), and multicast message-block events. The FeedEngineSettings::threadCount setting controls how many of them run (default 1).
A security-definition thread delivers the TCP recovery product-definition callbacks while the Handler downloads and processes security definitions, and raises the initial feed-started callback as the subscription starts - before any Feed Engine worker delivers market-data callbacks for the feed. (The order books are still empty at that point, so no OrderBookChangeListener reset is emitted; a mid-session recovery reset, by contrast, runs on the Feed Engine worker that services the feed.)
In log or PCAP replay mode there is no Feed Engine; market-data callbacks and LogReplayListener::onReplayFinished are delivered from the dedicated replay thread.
Handler state changes, and feed-started/stopped events raised on stop or restart, may additionally come from internal Handler threads that perform asynchronous stop, restart, and reconnection.

The table below summarizes where each listener's callbacks come from and what synchronization the application can rely on:

Listener	Delivered from	Synchronization
ErrorListener	Any internal thread	None - may be invoked concurrently
WarningListener	Any internal thread	None - may be invoked concurrently
FeedListener	Any internal thread	None - may be invoked concurrently
ExchangeListener	Security-definition thread (TCP recovery product definitions); Feed Engine worker thread(s) or the replay thread (multicast messages, including option/strategy/expiry definitions)	Not a single thread - see the note below
OrderBookChangeListener	Feed Engine worker thread(s) or the replay thread	Serialized per feed ID within a Handler; concurrent across feed IDs and across Handlers sharing a Feed Engine
OrderBookUpdateListener	Feed Engine worker thread(s) or the replay thread	Serialized per feed ID within a Handler; concurrent across feed IDs and across Handlers sharing a Feed Engine
OrderBookBundleUpdateListener	Feed Engine worker thread(s) or the replay thread	Serialized per feed ID within a Handler; concurrent across feed IDs and across Handlers sharing a Feed Engine
HandlerStateChangeListener	Any internal Handler thread (the start/stop caller, the security-definition thread, an async stop/restart or reconnect thread, or a Feed Engine worker)	May be invoked from different threads over the Handler's lifecycle
LogReplayListener	The replay thread	Delivered from the single replay thread
PacketProcessingListener	Feed Engine worker thread(s) or the replay thread for multicast packets; the security-definition thread for TCP packets, or a Feed Engine worker during Historical Replay	Not a single thread

Note: With the default FeedEngineSettings::threadCount of 1, all live market-data callbacks are delivered on that single Feed Engine worker thread, which is the simplest model to program against. Raising the thread count lets the Handler process different feeds in parallel, so callbacks for different feed IDs may then be invoked concurrently, while callbacks for any one feed ID within a Handler stay serialized. Feed IDs are assigned per Handler and restart at 0, so two Handlers that share a Feed Engine can invoke a shared listener for the same feed ID at the same time. During startup the TCP recovery product-definition callbacks and the initial feed-started callback are raised on the security-definition thread, before any Feed Engine worker delivers market-data callbacks for the feed; all order-book and market-data callbacks - including multicast definition updates and any mid-session recovery reset - run on a Feed Engine worker (or the replay thread). A listener that shares state across feeds, across Handlers, or between the startup and live phases must synchronize that access itself. The ErrorListener, WarningListener, and FeedListener are never synchronized and may be invoked from several threads at once.