Built by Maven

Message encoding and decoding

General principles

  • Encoding and decoding of SBE messages is performed by Message objects. Each Message object handles only one SBE message.

  • Message objects are created using a ByteDecoder object. The ByteDecoder itself is created separately, using an XML SBE template, and can then be reused in a multithreaded context.

  • Objects for encoding and decoding SBE messages on both sides of the communication must be created using the same XML SBE template or compatible versions of such templates.

  • The same Message object can be used to encode and decode an SBE message; the only difference is the order in which it is initialized. Once created, a Message object can be used to both read and modify data.

  • A Message object does not contain intermediate application data, and all changes are immediately reflected in the buffer with which it was associated upon creation. The reverse is also true: most changes to binary data are immediately available through the Message object API. The exception to this rule is standard service header fields (custom header fields are processed the same as other application data).

Preparing ByteDecoder

An object of type ByteDecoder produces Message objects and it can be created once for any number of concurrent uses.

        ByteDecoder decoder = null;
        try {

            MessageSchema schema = new MessageSchema(ParseUtil.parse(new File("SbeXmlTemplate.xml")));
            decoder = new ByteDecoderFactory().create(schema);
        } catch (LicenseException e) {
            throw new RuntimeException(e);
        }

In some cases, it's necessary to additionally specify the Simple Open Framing Header (SOFH) format. This header isn't defined in the XML SBE template, and its definition must be created separately using special SBE Toolset classes based on the documentation of the specific SBE data provider.

The SBE Toolset provides definitions for several well-known SOFHs. The following shows how to use one of them to initialize a ByteDecoder:

ByteDecoder decoder = null;
try {
    // Use one of well-known Simple Open Framing Header specifications.
    MessageSchema schema = new MessageSchema(ParseUtil.parse(new File("SbeXmlTemplate.xml")),
            FramingHeaderPolicy.getCustomPolicy(FramingHeaderSpecification.B3_FRAMING_HEADER));
    decoder = new ByteDecoderFactory().create(schema);
} catch (LicenseException e) {
    throw new RuntimeException(e);
}

The details of creating SOFH descriptions are discussed separately.

Message encoding

Message encoding is the process of creating a binary SBE message suitable for transmission. To create an SBE message, you need to:

  • Prepare a buffer for encoding;
  • Select the template ID of the message to be created;
  • Create a pre-initialized Message object;
  • Populate the data fields using the IMessage, IFieldSet, and IGroup interfaces;
  • Copy the populated data buffer for further transmission.

Preparing buffer

The SBE message buffer must be large enough to accommodate the entire message, including headers. The SBE toolset provides the ability to specify the exact size required to encode the message, but this is typically not required. Most often, it's sufficient to reserve a reasonable size, such as 4K, and copy data from it as needed.

// Allocate buffer.
final int LargeBufferSize = 4096;
byte[] buffer = new byte[LargeBufferSize];

The created buffer must be passed as a parameter to the method ByteDecoder.encode. This call will return an object that implements the IMessage interface, associated with the buffer and the selected SBE template.

// Choose template ID for the message encoded
int templateID = 101;
// Obtain IMessage instance that will handle the message with chosen template ID:
IMessage msgEncoded = decoder.encode(buffer, 0, buffer.length, templateID);

Immediately after initialization, the buffer is filled with the following data:

  • Required header information. Both headers are filled: SOFH and Message Header;
  • Empty values for non-optional fields (exact value depends on the field specification);
  • Null values for optional fields (exact value also depends on the field specification);
  • All groups and variable-length fields at the top level of nesting has length of 0 (zero);
  • The message version is the latest possible version declared in the SBE template.

Fields filling

Message fields are populated using the IFieldSet interface setter methods. The methods for changing fields differ slightly for fixed-length fields, groups, and variable-length fields.

Fixed-length fields

Fixed-length fields are fields whose length at the lower level is always the same, regardless of their value. For example, these are integer, Boolean, or enum-type fields, as well as fixed-length strings and composite data types.

Fields can be modified in one of the following ways:

  • Set a specific value. These methods have similar prototypes: void setFieldType(tag, value). For example, IFieldSet.setInt or IFieldSet.setBoolean. Fields encoded as fixed-length strings can be modified using the method IFieldSet.setString.

  • Set a default value or, if it's an optional field, a value corresponding to null: IFieldSet.reset.

  • Modify a field's value by overwriting lower-level data. This is unsafe, since the validity of such data is not verified, but it can still be used: IFieldSet.setBytes.

Repeating groups

To populate a repeating group, follow this flow:

  • Request the IGroup object;
  • Set the number of elements in the group;
  • Use methods that select the current group element (IGroup.setPos, IGroup.next, IGroup.rewind), to iterate through all the group elements and fill their values.

Modifying fields in a group element is no different from modifying message fields, but keep in mind:

  • It's best to populate groups in the order they are declared in the SBE template;
  • Resizing a group with nested groups can be a relatively expensive operation.

Sample of group definition:

<group name="NoMDEntries" id="268">
    <field name="MDEntryType" id="269" type="char"/>
    <field name="MDEntryPx" id="270" type="decimal" presence="optional"/>
    <field name="Currency" id="15" type="char" length="3" presence="constant">USD</field>
</group>

And the sample of code which fills the group:

        // Obtain list of bids from outside
        ScaledDecimal[] bids = getBids();

        // Choose group template ID
        int groupTag = 268;

        // Obtain group object
        IGroup group = msgEncoded.getGroup(groupTag);

        // Set group length
        int groupLength = bids.length;
        group.setLength(groupLength);

        // Iterate throuth the group elements to update them
        for(int i = 0; i < group.getLength(); ++i)
        {
            // Set position inside the group
            group.setPos(i);

            // All field operations will be performed on the current group element
            group.setChar(269, '0');
            group.setDecimal(270, bids[i]);

            // No need to update the constant field 'Currency'.
        }

Variadic-length data

Variable-length fields are modified using the IFieldSet.setVarData methods. It is best to fill these fields in the same order as they are described in the SBE template.

Message decoding

Decoding a message is the process of binding a Message object to SBE data and extracting the fields of that data.

To decode SBE data, you need to:

  • Create a Message object bound to the SBE data. This is accomplished using the ByteDecoder.decode method;
  • Extract the field values using the IFieldSet, IMessage, and IGroup methods.

Preparing decoding

The decoding data must be obtained from a source providing an SBE stream. It's important that the entire message fits into a single byte array.

// Allocate buffer.
byte[] buffer;
// buffer = ... fill it somewhere else ...

The received data should be passed to the ByteDecoder.decode method, which will return a Message object that is bound to the data and ready for reading individual fields.

// Obtain IMessage instance that will handle the message:
IMessage msgDecoded = decoder.decode(buffer, 0, buffer.length);

If the data set provided to the decoder contains an SBE message, the IFieldSet.decode method will return a new Message object. If errors occur, an exception will be thrown; if there is insufficient data, null will be returned.

Obtaining decoded message service parameters

The Message object can be queried for specific service information about a message.

        // Number of bytes occupied by the SBE message.
        int bytesUsed = msgDecoded.getMsgSize();

        // Template ID of the message.
        int templateId = msgDecoded.getTemplateId();

        // Version of the message.
        int version = msgDecoded.getVersion();

Decode a long data buffer

The IMessage.getMsgSize method returns the length of the data used by the SBE message. This is the full length, including service headers. The resulting value can be used to calculate the offset for the next decoding if the input data buffer contains multiple SBE messages:

IMessage msgNext = decoder.decode(buffer, bytesUsed, buffer.length - bytesUsed);

Or, in a loop:

        int offset = 0;
        while(offset < buffer.length) {
            IMessage msg = decoder.decode(buffer, offset, buffer.length - offset);
            if (msg == null) {
                break;
            }
            offset += msg.getMsgSize();

            // ... Handle the message content ...
        }

Getting fixed-length fields

  • Normal fields
  • Optional fields (contains and tryXXXXX)

Repeating groups

Variadic-length fields