Detecting TCP congestion using OnixS FIX Engines and Trading Handlers

Version 1.0.0

Exchanging the data via TCP is a complex process that consists of several layers:

1. Sending Application: The data is generated by the sending application.
2. TCP Segmentation: The TCP protocol divides the data into smaller segments. These segments contain the necessary information for reassembling the data at the receiving end.
3. IP Layer: Each TCP segment is then encapsulated into an IP packet. The IP layer adds routing information, including the source and destination IP addresses.
4. Routing through Network: The IP packet travels through the network, possibly passing through multiple routers and switches. These devices use routing tables to determine the best path for the packet to reach its destination.
5. Data Link Layer: At each hop, the IP packet is encapsulated into the appropriate data link layer protocol frame (e.g., Ethernet frame for Ethernet networks). This layer adds physical addressing information (MAC addresses in the case of Ethernet).
6. Physical Transmission: The frames are transmitted over the physical medium (e.g., copper wires, fiber optic cables, or wireless connections) to reach the next hop in the network.
7. Receiving Host: Once the packet reaches the destination host, it goes through the reverse process. The data link layer removes the frame, the IP layer extracts the IP packet, and finally, the TCP layer reassembles the segments into the original data stream.
8. Receiving Application: The data is delivered to the receiving application.

The network card handles physical communication with the network medium, the operating system manages network communication at the kernel level, and user-space network stack software provides networking capabilities to applications running in user space. Together, these components enable data transfer between applications over the network.

Network Card (NIC) Layer

The network card is a hardware component responsible for interfacing between the computer and the network. It connects the computer to the physical network medium, such as Ethernet cables or wireless signals. The network card handles tasks such as encoding and decoding data into electrical signals, transmitting data onto the network medium, and receiving incoming data from the network medium. Modern network cards often include specialized hardware features, such as TCP/IP offload engines (TOE), that can accelerate certain networking tasks, including TCP/IP processing and checksum calculations.

Operating System

The operating system manages communication between applications and the network card. It provides a set of networking APIs (Application Programming Interfaces) that allow applications to send and receive data over the network. The operating system is responsible for configuring and managing network interfaces, including initializing the network card, assigning IP addresses, configuring routing tables, and handling network-related events. The operating system's network stack implements networking protocols such as TCP/IP, UDP/IP, and ICMP/IP. It encapsulates data into packets, adds necessary headers (such as IP and TCP/UDP headers), and manages data transmission and reception. The operating system's network stack interacts with the network card through device drivers, which are software components that enable the operating system to communicate with hardware devices like network cards.

User-Space Network Stack Software

User-space network stack software refers to networking libraries, protocols, and utilities that operate outside the kernel space of the operating system. These software components interact with the operating system's networking APIs to send and receive data over the network. The user-space network stack is an optional extension to the kernel-space network stack provided by the operating system. It is used to implement custom networking functionality, optimize networking performance and perform network monitoring and analysis tasks. Users typically leverage user-space network stacks when they require specialized networking features, performance optimization, or flexibility beyond what is offered by the kernel-space network stack. Various socket acceleration libraries, such as Intel's Data Plane Development Kit (DPDK), Xilinx (Solarflare) OpenOnload, and Mellanox VMA (Virtual Machine Acceleration), provide optimized networking APIs and drivers for high-performance networking applications. These libraries bypass the kernel network stack and interact directly with the network hardware to achieve low-latency packet processing and high throughput.

Network Congestion

Network congestion, characterized by exceeding a network's capacity, can result in increased latency and decreased throughput. TCP congestion, occurring for various reasons at both the network and end-system levels, poses significant challenges. On the end-system side, several factors contribute to TCP congestion, including application behaviour, socket buffer sizes, TCP window scaling, delayed acknowledgements (ACKs), congestion control algorithms, network interface configurations, CPU and memory utilization, and background traffic.

Factors contributing to TCP Congestion at the host level:

• Application Behaviour: The behaviour of applications running on the host can affect TCP congestion. For example, applications that generate a high volume of traffic may overwhelm the network stack, leading to congestion. When the receiving application processes incoming data slower than the sending application generates it, it can lead to congestion due to TCP flow control mechanisms signalling the sender to slow down transmission and potential buffer overflow situations resulting in packet loss.
• Socket Buffer Sizes: Inadequate socket buffer sizes may cause congestion and packet loss, while excessively large buffers can increase latency.
• TCP Window Scaling: Misconfigured or disabled window scaling can hinder TCP performance and contribute to congestion.
• Delayed Acknowledgments (ACKs): Delayed ACKs can worsen congestion by impeding TCP's ability to respond promptly to network changes.
• TCP Congestion Control Algorithms: Different algorithms (e.g., Reno, Cubic, New Reno) may perform variably under different network conditions, necessitating the selection of the appropriate one for optimal performance.
• Network Interface Configuration: Settings such as Maximum Transmission Unit (MTU) can impact congestion, with misconfigurations leading to fragmentation and retransmissions.
• Network Interface Card (NIC) Settings: Inappropriate NIC settings may introduce latency or increase CPU utilization, exacerbating congestion.
• CPU and Memory Utilization: High CPU or memory usage can hinder TCP performance, causing delays and packet loss.
• Background Traffic: Other processes generating network traffic may compete with TCP flows, aggravating congestion. Prioritizing or throttling background traffic can help mitigate congestion for critical TCP flows.

Congestion Detection

The OnixS products can (if configured so) probe the timestamp of the received data and provide it to the user along with the data.

If accessible (supported by the hardware and configured), the network interface card timestamp is used ("hardware time stamping"); otherwise, the timestamp supplied by the OS kernel or the user-level network stack is used ("software time stamping"). The user then can calculate the delay between the data received by the host system and the data delivered to the application and suspect that the receiving application processes incoming data slower than the sending application generates it.

For example, please see the "Taking timestamps of the received data" section in the OnixS C++ CME iLink 3 Binary Order Entry Handler Programming Guide.

Note: According to our measurements, enabling this feature adds around 50-100 nanoseconds to the resulting latency.

Note: This functionality is available on Linux only.

Note: To compare hardware-based timestamps with the host time, it is necessary to synchronize the network interface card clock with the host clock, as described in IEEE 1588-2008. For the Xilinx (Solarflare) cards, the sfptpd utility can be used to perform this synchronization.

In conclusion, detecting and managing TCP congestion requires thoroughly understanding various factors influencing network performance. OnixS products offer valuable tools for monitoring and diagnosing congestion issues, empowering users to optimize TCP connections and application data handling logic effectively.