Composite data handling

A composite is a data type that consists of several components. Such data types are intended to encode information that can not be encoded using a simple type like integer or char. For instance, decimal numbers, date and time with timezone are encoded using composite types.

The structure of SBE composite data type is always described in the SBL XML template in accordance to the SBE standard. This standard defines the rules for such descriptions and introduces several composite types with predefined structure and semantics (decimal number, different headers, etc).

Composite types with predefined traits are hereinafter referred as predefined or well-known composites. Also, they are identified in metadata using the WellKnownComposite type. Other composite types that are defined by user are referred to as custom composite types or custom composites.

When the ByteDecoderFactory reads and compiles SBE XML template it handles predefined and custom composites in different ways. Handling of such data types is explained below.

Predefined composites

Predefined composites are composite types that are described in the SBE standard for particular purposes. Below is a list of such composite types:

  • Framing header - Simple Open Framing Header (part of SBE wire format);
  • Message header - header of the message itself;
  • Repeating group header;
  • Decimal value;
  • MonthYear to encode date;
  • UTCTimestamp to encode timestamp in UTC;
  • TZTimestamp to encode timestamp in particular timezone;
  • VarData header.

These types are recognized by the OnixS SBE Decoder on the basis of the set of their fields while the name of composite type often has no matter. For example, the decimal type can be named As DECIMAL9 or Price4Null, etc.

At the most of cases the composite type must have only the specified set of fields, no additional fields are allowed. If the type definition has extra fields, this type will be recognized as “custom” composite type.

Rules of predefined types recognition are described below. Please note that deep details of the type using and encoding are not subject of the documentation and follow the SBE Standard for such details.

Framing header

The framing header is never used directly from the SBE template but must be defined via API on the base of the venue specification. See the appropriate article Framing and message headers for details.

Message header

This header is automatically added at the beginning of each SBE message, just immediately after the Framing Header. All the header fields are handled automatically and user should noy touch them separately.

The message header consists of the following fields and can have extra fields:

Name Type Required Description
blockLength Integer 8-32 bits Yes Length of the fixed-size block of the given message.
version uint16 Yes Version of the message.
templateId uint8-uint32 Yes Template ID used to encode the message.
schemaId Yes Schema ID of the message (usually taken from the XML template).
numGroups No Number of repeating groups on the top level of the message.
numVarDataFields No Number of variadic-length data members on the top level of the message.

Repeating group header

This header prepends each repeating group inside the message (or inside another repeating group). All the header fields are handled automatically and user should noy touch them specifically.

Header of the repating group consists of the following fields:

Name Type Required Description
blockLength Integer 8-32 bits Yes Length of the fixed-size block of the given message.
numInGroup uint16 Yes The number of elements of the repating group.
numGroups No Number of repeating groups on the top level of the repeating group.
numVarDataFields No Number of variadic-length data members on the top level of the repeating group.

Decimal

This type intended to encode decimal data type.

User update fields of the type using methods that have the Decimal word in their names and biz.onixs.usil.ScaledDecimal argument or return type. Such a methods are: setDecimal, getDecimal, etc.

The type consists of the following fields only:

Name Type Description
mantissa Integer 8-64 bits Value of the mantissa.
exponent Value of the exponent.

An empty decimal value depends on the particular fields and its type definition. Such a value can be obtained via the IFeildSet.createNullDecimal.

MonthYear

This type intended to encode different data that denote a date without exact time.

User update fields of the type using methods that have the MonthYear word in their names and MonthYear argument or return type. For instance setMonthYear and getMonthYear.

The type consists of the following fields (at least one field must appear in the type definition):

Name Type Required Description
year Integer No Year.
month No Month.
day No Day.
week No Week number.

An empty MonthYear value depends on the particular fields and its type definition. Such a value can be obtained via the IFeildSet.createNullMonthYear.

UTC Timestamp and Timestamp with timezone

Thes types intended to encode UTC Timestamp with timezone information (TZTimestamp) or without it (UTCTimestamp).

User accesses and updates fields of the types using methods that have Timestamp word in their names and SbeTimestamp argument or return type. For instance setTimestamp or getTimestamp.

Fields of the SbeTimestamp object are updated in dependency on exact SBE type definition of the given field. If the field does not use timezone information, the properties belonged to the timezone are not touched in the object.

The UTC Timestamp type consists of the following fields only:

Name Type Description
time Integer 8-64 bits Number of time units.
unit Unit specification.

The TZTimestamp consists of the following fields only:

Name Type Description
time Integer 8-64 bits Number of time units.
unit Unit specification.
timezoneHour Hour of the timezone.
timezoneMinute Minute of the timezone.

An empty SbeTimestamp value depends on the particular fields and its type definition. Such a value can be obtained via the IFeildSet.createNullTimestamp.

VarData header

This type intended to encode header of variadic-length data fields.

All the header fields are handled automatically and user should noy touch them specifically.

The type consists of the following fields only:

Name Type Description
length Integer 8-64 bits Length of the variadic-length data.
data or varData The variadic-length data itself.

Custom composites

Composite that is not recognized as predefined composite type is a custom composite. Such a type can be handled in the stateful or stateless manner, or via “virtual fields”.

Advantages and disadvantages of the all methods described below:

  • Stateful data objects
    • Advantages
      • Independent data objects;
      • Straightforward usage as usual Java objects.
    • Disadvantages
      • Can be slow because updates and reads the entire field even only part of it must be updated;
      • Can not be applied to customized service composites (i.e. to headers).
  • Stateless data objects
    • Advantages
      • Fine-grained access to the composite field elements. Allows to read/update only part of the field without touching of another data;
      • The objects are usual Java interfaces.
    • Disadvantages
      • The stateless data depends on the lifecycle of the fieldset and can not be reused between different messages.
      • Can be applied to customized service composites (i.e. to headers).
  • Access via virtual fields
    • Advantages
      • Fine-grained access to the composite field elements. Allows to read/update only part of the field without touching of another data;
      • Fastest possible access to individual fields of the composite data;
      • Can be applied to customized service composites (i.e. to headers).
    • Disadvantages
      • Gives no integral representation of the composite data;
      • In some cases this can be difficult because it requires tracking the coordination between “virtual field identifiers” and the fieldsets from which those identifiers were obtained.

These methods are described below in details.

Creating custom composite definition in SBE template

Each custom composite must be defined in the SBE XML template.

<composite name="NewType" description="New non-standard custom type.">
    <type name="Number" type="int32" presense="optional"/>
</composite>

Stateful custom composite (using user-specified class)

The “stateful composite” is a data object that is detached from the field set and works as usual Java object. This object keeps data without dependency on the lifecycle of the message: its data copied to and from the message upon request.

Two following steps must be accomplished to implement the stateful composite data access:

  • It is needed to define an interface that will be used to access composite data object. This interface must specify all required composite fields access methods, and have to be marked with special attributes;
  • The interface class must be passed to one of the ByteDecoderFactory create method.

These steps are described below.

Defining custom interface

@SbeCompositeType("NewType")
public interface CustomComposite {
    @SbeCompositeField("Number")
    void setN1(int value);
    @SbeCompositeField("Number")
    int getN1();
    @SbeCompositeField("Number")
    void setNullNumber();
    @SbeCompositeField("Number")
    boolean isNullNumber();
}

Getting instance of the custom composite type

Instance of the custom composite can be obtained from the ByteDecoder object:

Error during retrieving content skip as ignoreDownloadError activated.

Please pay attention that the class passed to this method must be passed to the decoder constructor when the decoder is created.

Read and write field of the custom composite type

The composite type object can be applied to read the message field or to update its value:

// The custom2Obj gets content of the composite field
msg1.getCustomComposite(9001, customObj);
customObj.setN1(12345);
msg1.getCustomComposite(9001, customObj);

Stateless custom composite

The stateless composite is a data object that is bound to the fieldset (message or group) which field it handles, i.e. such an object can not be detached off the fieldset. The stateless composite provides ability to access composite data by its individual fields, without complete updating of the field. That makes the stateless composites faster than stateful ones.

Two following steps must be accomplished to implement the stateful composite data access:

  • It is needed to define an interface that will be used to access composite data object. This interface must specify all required composite fields access methods, and have to be marked with special attributes;
  • The interface class must be passed to one of the ByteDecoderFactory create method.

Creating custom composite definition in SBE template

<composite name="NewType" description="New non-standard custom type.">
    <type name="Number" type="int32" presense="optional"/>
</composite>

Defining custom interface

@SbeCompositeType("NewType")
public interface CustomComposite {
    @SbeCompositeField("Number")
    void setN1(int value);
    @SbeCompositeField("Number")
    int getN1();
    @SbeCompositeField("Number")
    void setNullNumber();
    @SbeCompositeField("Number")
    boolean isNullNumber();
}

Getting stateless composite data instance and access fields

The stateless composite instance can be created only when there is a message or group instantiated.

CustomComposite customObject = msg1.getStatelessComposite(9001, CustomComposite.class);

The obtained object is a direct interface to low-level data and can be used to read or write such data:

Error during retrieving content skip as ignoreDownloadError activated.

Access to custom composites using virtual fields

The virtual field is an elementary field included into composite type that is accessed individually via the IFieldSet interface without additional defining of an additional interface or class.

Virtual fields has own numeric identifiers that are assigned separately for each such field during compilation of the SBE template. These identifiers can be obtained via the MessageSchema.queryVirtualFieldId method.

The method requires “field path” as its argument to search the required field in the SBE template. Such a path should follow particular rules:

  • That is array of string where each string represents field name, or field ID, or special field name (or special ID).
  • Each element of the array corresponds to the particular nesting level;
  • The last element in the array must name elementary field in particular composite field.

Let's take the following sample instead of thousand words.

Access field via its virtual ID sample

SBE template of the composite type:

<composite name="Composite2" description="Custom composite type.">
    <type name="CountOfSomething" type="int32"/>
    <type name="SizeOfSomething" type="int64"/>
</composite>

SBE template of message:

<sbe:message name="CustomMessage" id="1" description="The message with custom composite field.">
    <!-- The 35'th field required for FIX messages -->
    <field name="messageType" type="NoMatter" id="35" presence="constant" value="NoMatter.Also"/>
    <!-- The custom composite field -->
    <field name="cust1" type="Composite2" id="9001"/>
</sbe:message>

Now we need to access composite member “SizeOfSomething” of the field “cust1” in the message “CustomMessage”. To accomplish this task we need to compose the path to the field and retrieve its virtual ID.

The requested path consists of the following components:

  • Message name (or ID);
  • Field name (or ID);
  • Composite field name.

In the case the path will be:

String[] fieldPath = new String[] {
        "CustomMessage", // Message name
        "cust1", // Field name
        "SizeOfSomething" // Composite field (element) name.
};

This path must be used for queryVirtualFieldId call after parsing of the SBE template to obtain the virtual field ID:

        Document xmlSchema = ParseUtil.parse(new File("MessageSchema.xml"));
        MessageSchema schema = new MessageSchema(xmlSchema);

        int virtualFieldId = schema.queryVirtualFieldId(fieldPath);

To successfully create the ByteDecoder it is needed to suppress custom composite recognition using empty classes.

Let's declare the empty class:

@SbeAutoRecognize
public class Composite2 {
}

…and create the decoder:

        Class<?>[] customComposites = {
                Composite2.class
        };

        ByteDecoder decoder = new ByteDecoderFactory().create(schema, customComposites);

Now we can access field using the virtual identifier:

        byte[] data = new byte[SIZE_ENOUGH_TO_ENCODE];
        IMessage msg = decoder.encode(data, 0, data.length, 1);

        // Set the virtual field value
        msg.setLong(virtualFieldId, 123L);

        // Get the virtual field value
        long v = msg.getLong(virtualFieldId);

Access message header field via its virtual field ID

Usually, there is no need to access message header fields directly: the ByteDecoder automatically updates all required information during the encoding process. But in some cases, the message header has non-standard fields. The user code must update such fields separately. The “virtual field” mechanics significantly help in the task.

Let's take the sample of non-standard message header:

<composite name="messageHeader" description="Message header with non-standard field.">
    <type name="blockLength" primitiveType="uint16"/>
    <type name="templateId" primitiveType="uint16"/>
    <type name="schemaId" primitiveType="uint16"/>
    <type name="version" primitiveType="uint16"/>
    <type name="SequenceNo" primitiveType="int32"/>
</composite>

Here, the messageHeader composite definition has a non-standard field SequenceNo. The user code must update the field explicitly.

Please note that even if the message header contains non-standard fields, there is no need to define a “custom composite” class for the header.

ByteDecoder decoder = new ByteDecoderFactory().create(schema);

This code shows how to obtain the virtual ID of the “SequenceNo” message header field. Please pay attention how the field name is declared:

        String[] headerSequenceNoFieldPath = new String[] {
                // Message header composite pseudo-name
                SpecialFieldIds.MESSAGE_HEADER_FIELD_NAME,
                
                // The message header field name
                "SequenceNo"
        };

        // Get virtual ID of the 'SequenceNo' message header field
        int seqNumID = schema.queryVirtualFieldId(headerSequenceNoFieldPath);

This message creating is trivial:

byte[] data1 = new byte[SIZE_ENOUGH_TO_ENCODE];
IMessage msg1 = decoder.encode(data, 0, data.length, 1);

And finally, the following code shows how to access SequenceNo field of the message header in the given message:

// Set the 'SequenceNo' message header field
msg1.setInt(seqNumID, 12345);
// Get the 'SequenceNo' message header field
int seqNum1 = msg1.getInt(seqNumID);