TCP in depth
Properties
TCP (Transmission Control Protocol) provides:
- Connection-oriented communication — a three-way handshake before data (detailed in the handshake topic) and an orderly close.
- Reliable delivery — every byte is acknowledged; lost data is retransmitted.
- In-order delivery — the receiver reassembles bytes in sequence.
- Byte stream — no message boundaries; the application must frame messages (length prefixes, newlines, HTTP headers).
- Flow control — the sender never overwhelms the receiver.
- Congestion control — the sender adapts to network capacity.
- Full duplex — both directions independently.
The unit is a segment.
The TCP header
| Field | Purpose |
|---|---|
| Source / destination port | 16 bits each |
| Sequence number | Byte position of this segment's first data byte |
| Acknowledgement number | Next byte the sender of this segment expects to receive |
| Data offset | Header length (header is 20 bytes minimum, up to 60 with options) |
| Flags | SYN, ACK, FIN, RST, PSH, URG, ECE, CWR |
| Window size | How many bytes the sender of this segment can receive (flow control) |
| Checksum | Detects corruption |
| Urgent pointer | Rarely used |
| Options | MSS, window scale, SACK permitted, timestamps |
Reliability: sequence numbers, ACKs and retransmission
Every byte has a sequence number. The receiver sends cumulative ACKs: "ACK 5001" means "I have everything up to byte 5000; send 5001 next".
Loss is detected two ways:
- Retransmission timeout (RTO) — no ACK within a timeout computed from measured round-trip time (minimum ~200 ms on Linux, doubling on each retry — exponential backoff). Timeouts are slow and severely cut throughput.
- Fast retransmit — the receiver keeps acknowledging the last in-order byte when later segments arrive; three duplicate ACKs tell the sender a segment is missing, and it retransmits immediately without waiting for the timeout.
SACK (selective acknowledgement) lets the receiver report exactly which blocks it has, so the sender retransmits only the gaps. Timestamps improve RTT measurement and protect against wrapped sequence numbers on fast links.
Flow control: the receive window
Each side advertises a receive window (rwnd) — free space in its receive buffer. The sender may have at most that many unacknowledged bytes in flight. A slow application that doesn't read its socket shrinks the window to zero; the sender pauses ("zero window" in packet captures) and probes periodically.
The window field is only 16 bits (max 65,535 bytes); the window scaling option (negotiated in the SYNs) multiplies it, allowing windows of many megabytes — essential for fast, long-distance links.
Congestion control: why loss kills throughput
Flow control protects the receiver; congestion control protects the network. The sender keeps a congestion window (cwnd); it may send min(cwnd, rwnd) unacknowledged bytes.
- Slow start — a new connection starts with a small cwnd (commonly 10 segments ≈ 14 KB) and doubles it every round trip until loss or a threshold.
- Congestion avoidance — beyond the threshold, cwnd grows roughly one segment per RTT.
- On loss (detected by duplicate ACKs) — cwnd is cut sharply (Reno halves it; CUBIC, Linux's default, reduces by about 30% and regrows along a cubic curve). On a timeout, cwnd collapses back to a minimum and slow start begins again.
- BBR (Google) models bottleneck bandwidth and RTT instead of reacting to loss — much better on lossy, long-distance links. Enable with
sysctl net.ipv4.tcp_congestion_control=bbr(with thefqqdisc).
Consequences you should be able to explain:
- Even 1% packet loss can cut TCP throughput dramatically on high-latency paths, because each loss shrinks cwnd and recovery takes many RTTs.
- New connections are slow at first (slow start) — one reason connection reuse, keep-alive and HTTP/2 multiplexing matter.
- Throughput ≤ window / RTT. This is the bandwidth-delay product (BDP): to fill a 1 Gbps link with 100 ms RTT you need ~12.5 MB in flight. Small buffers or windows cap throughput no matter how big the pipe — a classic cause of slow cross-region transfers (Mumbai ↔ US). Fix: larger socket buffers (
net.core.rmem_max,net.ipv4.tcp_rmem), parallel streams, or tools designed for it (S3 multipart,aws s3 cpconcurrency).
Nagle, delayed ACKs and latency
- Nagle's algorithm batches small writes into fewer segments while data is unacknowledged — efficient, but adds latency for chatty request/response protocols.
- Delayed ACK — receivers wait briefly (up to ~40 ms on Linux) to piggyback ACKs.
- Together they can cause mysterious ~40 ms stalls. Latency-sensitive apps set
TCP_NODELAY(most HTTP clients, databases and RPC frameworks already do).
Keepalive and idle timeouts
TCP keepalive sends probes on idle connections to detect dead peers and keep NAT/firewall/load-balancer state alive. Linux defaults are very long (first probe after 2 hours), so apps usually configure their own or rely on application-level pings.
Middleboxes drop idle connections:
| Device | Idle timeout |
|---|---|
| AWS ALB | 60 s default (configurable) |
| AWS NLB | 350 s for TCP |
| AWS NAT gateway | 350 s |
| Many corporate firewalls | Minutes |
A long-lived connection (DB pool, message consumer) that sits idle longer than that is silently dropped; the next use fails with a reset or timeout. Fix: keepalives below the shortest timeout, pool max-idle settings, and retries.
Head-of-line blocking
Because TCP delivers bytes strictly in order, one lost segment holds up all later data until it's retransmitted, even if later segments already arrived. HTTP/2 multiplexes many streams over one TCP connection, so a single loss stalls every stream — the motivation for QUIC.
TCP Fast Open
Lets a client send data inside the SYN on repeat connections (using a cookie from a previous connection), saving one round trip. Limited deployment because middleboxes interfere.