File-Based Session Storage

File-Based Session Storage keeps messages in set of files arranged in directory, and specified by OnixS::FIX::EngineSettings::logDirectory settings. This storage usually keeps data at two kinds of files: messages are kept in StorageId.summary and session state kept in StorageId.state file. StorageID is a generic name of storage and it is available through the OnixS::FIX::Session::storageId() method.

In particular cases, it is needed to save space on disk and there is no need to keep the whole history of incoming and outgoing messages. To satisfy such requirements, File-Based Storage could be configured to store messages in a set of .summary files, instead of a single one. The entire set of logged messages is split through several files called as "segments", and the oldest segments could be just removed without affection on the Engine functionality. Details of this mode are described in the next section.

Segmented File-Based Storage

Segmented mode of file-based storage is initiated when the OnixS::FIX::EngineSettings::fileBasedStorageSegmentSize setting is configured to non-zero before Engine startup. This value specifies the maximum size of each one file where incoming and outgoing messages stored and affects all sessions created.

Names of segments composed as: StorageID[-part-N].summary, where:

• StorageID - generic name of storage, available through the OnixS::FIX::Session::storageId() method;
• -part-N - suffix of the part. The N is 1-based number of the part. One segment does not have this suffix - it is the segment where freshmost messages are written.

During message interchange, files are created and fulfilled by the following common algorithm:

• The new message is always logged to StorageID.summary file;
• After the new message logged, StorageID.summary file size is compared to the EngineSettings::fileBasedStorageSegmentSize value, and if the actual file size exceeds, a value new segment is created;
• To create new segment, the Engine renames StorageID.summary to StorageID-part-N.summary and creates a new empty StorageID.summary file to log subsequent messages. The N is a sequential number of part starting from 1.

So during a long session, there is a set of files created: StorageID.summary, StorageID-part-1.summary, StorageID-part-2.summary, etc. Oldest messages are arranged at StorageID-part-1.summary, and the freshmost ones - at StorageID.summary and StorageID-part-N.summary, where N is of maximum value.

At the same time, the entire number of parts is registered at StorageID.state with the key of FilePartCount. For instance, if there are 5 parts (StorageID.summary and StorageID-part-1.summary - StorageID-part-4.summary) the following line will be written at StorageID.summary:

FilePartCount=5

When the session restarts only a particular part of the entire set of segments, they could be used to retrieve the resend request queue. The exact number of segments used to retrieve this queue depends on the OnixS::FIX::EngineSettings::resendingQueueSize value and on traits of messages, which were written in .summary files. Messages are retrieved in backward order, from the tail to a start of the file set, so segment content is analyzed in the following order:

• Getting value of FilePartCount parameter from StorageID.state file;
• Analyzing a freshest segment (i.e. - just StorageID.summary);
• If more messages are needed, analyze StorageID-part-N.summary files down-counting N from (FilePartCount - 1) to 1.

Unneeded segments (which obviously will have lowest N in -part-N suffixes) are never touched by Engine and could be deleted or backed up by a user.

While the session is active, all segments, which could be used by the service resend request, are kept opened to prevent unexpected deleting. When new messages arrived, older segments are closed and detached from Engine and could be removed or backed up by a user.

Combinations between segmented and non-segmented File-Based Storages

It is important to understand that Engine decides whether the storage working mode is just on the startup. Assume the following situation:

• Configure EngineSettings::fileBasedStorageSegmentSize to non-zero and start Engine;
• Start the session which has a lot of exchanges, and creates, for instance, 5 segments;
• Stop Engine;
• Configure EngineSettings::fileBasedStorageSegmentSize to 0, start Engine;
• Start the same sessions as above, trying to restore session state from the existing storage. The Engine will read FilePartCount=5, but it will count that there is no upper limit for segment size. Therefore, messages still will be read from (possibly) all available segments. But during the session activity, no more segments will be created, and StorageID.summary file will grow without limit. From some moments, this file will carry all messages required for the resend request, and -part-N.summary files will not be needed.

Also, assume another situation.

• Configure EngineSettings::fileBasedStorageSegmentSize to 0 and start Engine;
• Start the session which has a lot of exchanges, and creates big single segment with the name of StorageID.summary;
• Stop Engine;
• Configure EngineSettings::fileBasedStorageSegmentSize to non-zero value which is less than the StorageID.summary size, start Engine;
• Start the same sessions as above, trying to restore a session state from the existing storage. Now the Engine will treat existing .summary as just a single big segment, which size exceeds limit of a segment size. Therefore, Engine will log just a single message to the file (that will be Logon message), which will rename the StorageID.summary to StorageID-part-1.summary, and will create new StorageID.summary to log subsequent messages.

Checking integrity of segmented File-Based Storage

During startup the Engine, check integrity of segmented file storage. The storage is treated as "correct" when all files, required to retrieve a resending queue, are available and accessible to read. If some files are not available, or inaccessible behavior depends on OnixS::FIX::EngineSettings::ignoreFileBasedStorageIntegrityErrors setting:

• OnixS::FIX::EngineSettings::ignoreFileBasedStorageIntegrityErrors=false (default value) Throw exception during session creation and prevent the session against created.
• OnixS::FIX::EngineSettings::ignoreFileBasedStorageIntegrityErrors=true Create a new storage with new StorageID. All sequence numbers are reset to initial values, and appropriate message is written to Engine log.

Attaching File-Based Storage

    Session initiator("SenderCompId", "TargetCompId", ProtocolVersion::FIX_42, &listener, SessionStorageType::FileBased);

Examples

Segmented storage

    EngineSettings settings;
    settings.fileBasedStorageSegmentSize(256 * 1024)
            .ignoreFileBasedStorageIntegrityErrors(true);

    Engine::init(settings);
    Session initiator("SenderCompId", "TargetCompId", ProtocolVersion::FIX_42, &listener, SessionStorageType::FileBased);

File-Based Storage without segmentation

    EngineSettings settings;
    settings.fileBasedStorageSegmentSize(0);

    Engine::init(settings);
    Session initiator("SenderCompId", "TargetCompId", ProtocolVersion::FIX_42, &listener, SessionStorageType::FileBased);