Setting up a Feed Engine

A Feed Engine receives data and delivers it to one or more Handler instances. Each Handler processes that data for its configured CME MDP channel and notifies your application through event callbacks.

Data flow for live market data

Arrows show data delivery. The boxes do not represent separate threads.

Choosing a Feed Engine

The SDK provides two Feed Engine implementations for live network reception:

Implementation	Reception	When to use it
SocketFeedEngine	Standard socket API	Default network reception.
SolarflareFeedEngine	Kernel-bypass multicast reception through Solarflare's ef_vi API	Latency-sensitive applications on Linux with supported Solarflare adapters.

In mixed mode, the Solarflare Feed Engine can fall back to standard sockets when initializing multicast reception; see Mixed mode.

Default and explicit engines

By default, each Handler owns an internally managed Feed Engine backed by OnixS::CME::MDH::SocketFeedEngine and an internal worker thread pool, so no explicit Feed Engine setup is required.

The internally managed default is intended primarily for examples and initial setup. For production applications, prefer an explicit Feed Engine to control reception and processing threads.

To use the Solarflare Feed Engine, construct an explicit engine and bind it to the Handler.

Configuration	Who owns the engine?	Who runs it?	When to use it
Default	Handler	Internal worker pool	Examples and initial setup.
Explicit	Application	Application loop or FeedEngineThreadPool	Control reception and processing threads in production, share an engine between Handlers, or choose another implementation.

Using an explicit Feed Engine

Bind an explicit engine through FeedSettings::engine(NetFeedEngine*) before starting the Handler. The application must run the engine using one of these approaches:

• Repeatedly call OnixS::CME::MDH::NetFeedEngine::process from an application thread.
• Use OnixS::CME::MDH::FeedEngineThreadPool to call it on background threads; see Multi-threaded processing.

Calling OnixS::CME::MDH::Handler::start alone does not run an explicitly bound engine. The following example constructs an explicit engine, binds it to the Handler, and calls OnixS::CME::MDH::NetFeedEngine::process in an application loop:

SocketFeedEngine dataProvider;
 
Handler handler;
 
// Selects the CME MDP channel and specifies the configuration file's location.
handler.settings().channel(310).connectivityConfigurationFile("config.xml");
 
// Binds the Handler to the Feed Engine.
handler.settings().feeds().engine(&dataProvider);
 
// Configures the processing session.
setSessionToAccurateLateJoin(handler.settings().session());
 
// Moves the Handler to the processing state.
handler.start();
 
// Runs the Feed Engine logic to receive live market data
// from the network and to push it to the Handler.
while (!userRequestedToEndLooping())
    dataProvider.process();
 
// Stops the Handler.
handler.stop();

Explicit engine lifetime

The application owns an explicitly bound engine. Keep it alive while it is bound to any Handler and while any processing pool or application-managed worker uses it. OnixS::CME::MDH::Handler::stop stops market data processing but does not remove the engine binding.

One safe shutdown sequence is:

1. Stop all Handlers bound to the engine.
2. Stop and join the engine's processing workers. Destroying OnixS::CME::MDH::FeedEngineThreadPool stops and joins its workers. For application-managed threads, signal them to stop and join them.
3. Destroy the stopped Handlers or explicitly remove their engine bindings.
4. Destroy the engine.

Other destruction orders can also satisfy the lifetime requirements: the engine must outlive all of its bindings and processing workers. For example, the multithreaded example below destroys its stopped Handler before its pool, then destroys the engine last.

When using automatic objects, declaring the engine first, the Handler second, and the pool last ensures that the pool and Handler are destroyed before the engine, including during exception unwinding.

To remove an explicit binding from a stopped Handler, pass nullptr to FeedSettings::engine(NetFeedEngine*). This leaves the Handler without a Feed Engine; bind a valid engine before calling OnixS::CME::MDH::Handler::start again.

Sharing the Feed Engine between multiple handlers

Multiple Handler instances can explicitly share a single Feed Engine instance:

void configure(HandlerSettings& settings, ChannelId channel, NetFeedEngine& dataProvider)
{
    // Selects the CME MDP channel and specifies the configuration file's location
    settings.channel(channel).connectivityConfigurationFile("config.xml").feeds().engine(&dataProvider);
 
    setSessionToAccurateLateJoin(settings.session());
}
 
void runHandlers()
{
    // Initializes the Feed Engine.
    SocketFeedEngineSettings feedEngineSettings;
    SocketFeedEngine dataProvider(feedEngineSettings);
 
    // Initializes the Handlers.
    Handler handlerForChannel310;
    Handler handlerForChannel312;
 
    // Configures Handlers.
    configure(handlerForChannel310.settings(), 310, dataProvider);
    configure(handlerForChannel312.settings(), 312, dataProvider);
 
    // Moves the Handler to the processing state.
    handlerForChannel310.start();
    handlerForChannel312.start();
 
    // Runs the Feed Engine logic to receive live market data
    // from the network and to push it to handlers.
    while (!userRequestedToEndLooping())
        dataProvider.process();
 
    // Stops the Handler.
    handlerForChannel310.stop();
    handlerForChannel312.stop();
}

Multi-threaded processing

If the Feed Engine implementation supports multithreaded processing, it is possible to distribute the Feed Engine logic across multiple threads instead of calling OnixS::CME::MDH::NetFeedEngine::process from one thread only.

OnixS::CME::MDH::FeedEngineThreadPool creates worker threads that repeatedly call OnixS::CME::MDH::NetFeedEngine::process on the engine passed to its constructor. Configure the number of workers through OnixS::CME::MDH::FeedEngineThreadSettings::pool and OnixS::CME::MDH::ThreadPoolSettings::size.

A pool size of 1 is the default and recommended starting point. Increase it only if the Feed Engine implementation supports concurrent processing and measurements show a benefit for your workload.

With the standard OnixS::CME::MDH::SocketFeedEngine, socket reads, data processing and application callbacks for one channel are serialized. While that channel is replaying queued packets or running a callback, additional workers cannot drain its sockets. Incoming traffic accumulates in the operating system receive buffers and can be lost if those buffers fill. Adding workers does not remove a processing bottleneck within that channel. See Keeping callbacks responsive.

The pool reports worker-thread events through OnixS::CME::MDH::FeedEngineThreadPoolListener. Pass a listener when constructing the pool.

class ThreadListener : public FeedEngineThreadPoolListener
{
public:
    void onFeedEngineThreadIssue(const FeedEngineThreadPool&, const Messaging::Char* issue) override
    {
        std::clog << issue << std::endl;
    }
};
 
ThreadListener listener;
 
SocketFeedEngine dataProvider;
 
// Configures thread pool
FeedEngineThreadSettings threadSettings;
threadSettings.pool().size(2).affinity().cpus().insert(1);
 
FeedEngineThreadPool threads(dataProvider, threadSettings, &listener);
 
Handler handler;
handler.settings().feeds().engine(&dataProvider);
 
handler.start();
 
while (!userRequestedToStop())
    CurrentThread::sleep();
 
handler.stop();
 

Note

The Feed Engine abstraction does not require implementations to support multithreaded execution. Therefore, the OnixS::CME::MDH::NetFeedEngine::process function may not support simultaneous invocation by multiple threads. Check the documentation of a particular implementation to see if it is safe to call OnixS::CME::MDH::NetFeedEngine::process simultaneously.

Custom Feed Engines

The Feed Engine abstraction is represented as a set of abstract classes. Implementing them allows users to build their logic by extracting market data from various sources, including networks, file-based stores, etc.

To implement a custom feed engine, please contact OnixS Support.