Composite data handling

A composite is an SBE type that contains several elementary fields. Composites are used for values that cannot be represented by one primitive field, for example:

  • decimal value;
  • date without time;
  • timestamp with unit and optional timezone;
  • service headers.

Composite layout is always defined in the XML template.

Predefined composites

The decoder recognizes several composite shapes as predefined:

Recognition is based on field composition, not on type name. If shape does not match predefined rules, the type is treated as custom composite.

Rules and semantics of wire format come from the SBE standard. This guide focuses on API behavior.

Framing header

Framing header is configured through API, not directly from template parsing flow. See Framing and message headers.

Message header

Message header is inserted after SOFH and is processed automatically. Standard fields are handled by decoder. Extra custom fields can be handled through virtual fields or custom composites.

Message header fields:

Name Type Required Description
blockLength Integer 8-32 bits Yes Length of fixed-size block of the message body.
version uint16 Yes Message version.
templateId uint8-uint32 Yes Message template ID.
schemaId Integer Yes Schema ID.
numGroups Integer No Number of top-level repeating groups.
numVarDataFields Integer No Number of top-level variable-length fields.

Repeating group header

Each group has its own header. Decoder fills it automatically.

Group header fields:

Name Type Required Description
blockLength Integer 8-32 bits Yes Length of fixed-size block for one group element.
numInGroup uint16 Yes Number of elements in the group.
numGroups Integer No Number of nested groups on top level of group element.
numVarDataFields Integer No Number of top-level variable-length fields in group element.

Decimal

Decimal composite is accessed with:

Structure:

Name Type Description
mantissa Integer 8-64 bits Mantissa value.
exponent Integer Exponent value.

MonthYear

MonthYear composite is accessed with:

Structure (at least one field must be present):

Name Type Required Description
year Integer No Year.
month Integer No Month.
day Integer No Day.
week Integer No Week number.

UTC Timestamp and TZ Timestamp

Timestamp composites are accessed with:

SbeTimestamp object is populated according to concrete composite layout. For UTC timestamp, timezone fields stay untouched.

UTCTimestamp structure:

Name Type Description
time Integer 8-64 bits Number of time units.
unit Integer Time unit.

TZTimestamp structure:

Name Type Description
time Integer 8-64 bits Number of time units.
unit Integer Time unit.
timezoneHour Integer Timezone hour offset.
timezoneMinute Integer Timezone minute offset.

VarData header

Var-data fields are encoded as header + payload and handled automatically.

Structure:

Name Type Description
length Integer 8-64 bits Length of payload.
data/varData Bytes Payload bytes.

Custom composites

Any composite that is not recognized as predefined is a custom composite. You can handle it in three ways:

Approach Strengths Limitations
Stateful custom object Detached Java object, straightforward usage Reads/writes full object each time; not suitable for custom service headers
Stateless composite interface Field-level access, no full copy Bound to current message lifecycle
Virtual fields Fast field-level access, no extra interface No single aggregate object

Creating custom composite in template

Each custom composite must be declared in XML template:

<composite name="NewType" description="New non-standard custom type.">
    <type name="Number" type="int32" presense="optional"/>
</composite>

Stateful custom composite (using user class)

Stateful object is detached from fieldset. Data is copied between message buffer and object on read/write calls.

Required steps:

  1. Define custom interface and mark fields with required annotations.
  2. Pass this interface class to ByteDecoderFactory.create(...).

Interface example:

@SbeCompositeType("NewType")
public interface CustomComposite {
    @SbeCompositeField("Number")
    void setN1(int value);
    @SbeCompositeField("Number")
    int getN1();
    @SbeCompositeField("Number")
    void setNullNumber();
    @SbeCompositeField("Number")
    boolean isNullNumber();
}

Create empty instance:

Error during retrieving content skip as ignoreDownloadError activated.

Read and write field value:

// The custom2Obj gets content of the composite field
msg1.getCustomComposite(9001, customObj);
customObj.setN1(12345);
msg1.getCustomComposite(9001, customObj);

Stateless custom composite

Stateless composite is bound to current fieldset and field. It gives direct field-level access without full object copy.

Setup steps are the same:

  1. Define interface.
  2. Pass class to decoder factory.

Template example:

<composite name="NewType" description="New non-standard custom type.">
    <type name="Number" type="int32" presense="optional"/>
</composite>

Interface example:

@SbeCompositeType("NewType")
public interface CustomComposite {
    @SbeCompositeField("Number")
    void setN1(int value);
    @SbeCompositeField("Number")
    int getN1();
    @SbeCompositeField("Number")
    void setNullNumber();
    @SbeCompositeField("Number")
    boolean isNullNumber();
}

Get stateless instance:

CustomComposite customObject = msg1.getStatelessComposite(9001, CustomComposite.class);

Read/write through stateless interface:

int n1 = customObject.getN1(); // Reads data.
customObject.setN1(12345); // Immediately updates low-level data.

Stateful vs stateless on the same field

The following example shows both approaches for one composite field:

        // Stateful composite: detached object, requires explicit write-back.
        CustomComposite stateful = decoder.createEmptyCustomComposite(CustomComposite.class);
        msg1.getCustomComposite(9001, stateful);
        int statefulN1 = stateful.getN1();
        stateful.setN1(statefulN1 + 1);
        msg1.setCustomComposite(9001, stateful);

        // Stateless composite: direct view over message bytes.
        CustomComposite stateless = msg1.getStatelessComposite(9001, CustomComposite.class);
        int statelessN1 = stateless.getN1();
        stateless.setN1(statelessN1 + 1);

Access to custom composites using virtual fields

A virtual field is a composite member that can be addressed as a normal IFieldSet field by synthetic ID. No extra Java interface is needed.

Virtual IDs are generated during schema parsing. Use MessageSchema.queryVirtualFieldId to get an ID.

Field path rules:

  • path is String[];
  • each element is name or numeric ID of schema element;
  • last element is member of target composite;
  • first element is message name/ID, or "*" for message header.

Access field by virtual ID

Composite definition:

<composite name="Composite2" description="Custom composite type.">
    <type name="CountOfSomething" type="int32"/>
    <type name="SizeOfSomething" type="int64"/>
</composite>

Message definition:

<sbe:message name="CustomMessage" id="1" description="The message with custom composite field.">
    <!-- The 35'th field required for FIX messages -->
    <field name="messageType" type="NoMatter" id="35" presence="constant" value="NoMatter.Also"/>
    <!-- The custom composite field -->
    <field name="cust1" type="Composite2" id="9001"/>
</sbe:message>

Goal: access SizeOfSomething inside field cust1 of message CustomMessage.

Path components:

  1. Message name (or ID).
  2. Field name (or ID).
  3. Composite member name.

Path example:

String[] fieldPath = new String[] {
        "CustomMessage", // Message name
        "cust1", // Field name
        "SizeOfSomething" // Composite field (element) name.
};

Query virtual ID:

        Document xmlSchema = ParseUtil.parse(new File("MessageSchema.xml"));
        MessageSchema schema = new MessageSchema(xmlSchema);

        int virtualFieldId = schema.queryVirtualFieldId(fieldPath);

If you want pure virtual-field access, suppress custom composite recognition by passing placeholder class.

Placeholder class:

@SbeAutoRecognize
public class Composite2 {
}

Decoder creation:

        Class<?>[] customComposites = {
                Composite2.class
        };

        ByteDecoder decoder = new ByteDecoderFactory().create(schema, customComposites);

Use virtual ID:

        byte[] data = new byte[SIZE_ENOUGH_TO_ENCODE];
        IMessage msg = decoder.encode(data, 0, data.length, 1);

        // Set the virtual field value
        msg.setLong(virtualFieldId, 123L);

        // Get the virtual field value
        long v = msg.getLong(virtualFieldId);

Access custom message-header fields by virtual ID

Standard header fields are handled automatically. If the header has custom members, you can update them via virtual IDs.

Header example with extra SequenceNo field:

<composite name="messageHeader" description="Message header with non-standard field.">
    <type name="blockLength" primitiveType="uint16"/>
    <type name="templateId" primitiveType="uint16"/>
    <type name="schemaId" primitiveType="uint16"/>
    <type name="version" primitiveType="uint16"/>
    <type name="SequenceNo" primitiveType="int32"/>
</composite>

Decoder creation (no custom header class required):

ByteDecoder decoder = new ByteDecoderFactory().create(schema);

Path for header field:

        String[] headerSequenceNoFieldPath = new String[] {
                // Message header composite pseudo-name
                SpecialFieldIds.MESSAGE_HEADER_FIELD_NAME,
                
                // The message header field name
                "SequenceNo"
        };

        // Get virtual ID of the 'SequenceNo' message header field
        int seqNumID = schema.queryVirtualFieldId(headerSequenceNoFieldPath);

Message creation:

byte[] data1 = new byte[SIZE_ENOUGH_TO_ENCODE];
IMessage msg1 = decoder.encode(data, 0, data.length, 1);

Header field access by virtual ID:

// Set the 'SequenceNo' message header field
msg1.setInt(seqNumID, 12345);
// Get the 'SequenceNo' message header field
int seqNum1 = msg1.getInt(seqNumID);