Opscraft
TCP vs UDP · Lesson 5 of 8 · 4 min read

Operating TCP and UDP services

Firewalls and NAT

  • Stateful firewalls (security groups, iptables conntrack) track TCP by its handshake and FIN/RST. UDP has no connection, so they track pseudo-connections by the address/port tuple with an idle timeout: an outbound UDP packet opens a short window for replies.
  • NAT treats UDP the same way — short idle timeouts mean long-lived UDP flows need periodic keepalive packets.
  • Scanning: a TCP port is clearly open (SYN-ACK), closed (RST) or filtered (no reply). UDP gives no handshake — a scanner sending to a UDP port often gets no response and reports open|filtered; a closed UDP port may return ICMP port unreachable (if not blocked).

Load balancers

  • ALB — HTTP/HTTPS over TCP only (HTTP/2 and gRPC supported).
  • NLB — TCP, UDP, TCP_UDP and TLS listeners. Use it for UDP services (game servers, DNS, syslog, VoIP signalling). Since UDP has no handshake, NLB health checks for UDP targets use TCP or HTTP on a port you choose.
  • gRPC behind an L4 load balancer: gRPC uses long-lived HTTP/2 connections; an NLB balances connections, so all of a client's requests stick to one backend and load becomes uneven as you scale. Use L7 balancing (ALB with gRPC, Envoy/service mesh) or client-side balancing (gRPC with a headless Service and round-robin policy).

Kubernetes

  • Services support TCP, UDP and SCTP (and both TCP and UDP on the same port — CoreDNS's Service does exactly that for 53).
  • The 5-second DNS delay: glibc sends A and AAAA queries in parallel from the same socket; a race in conntrack's handling of simultaneous UDP packets on the same tuple can drop one, and the client waits for the 5-second timeout before retrying. Symptoms: random 5 s latency on outbound calls from pods. Fixes: NodeLocal DNSCache (uses TCP upstream / avoids the DNAT race), options single-request-reopen or use-vc in dnsConfig, musl-based images behave differently, or kernel fixes.
  • Conntrack table limits (nf_conntrack_max) on busy nodes cause dropped packets for both protocols.
Enlarge diagramPod (glibc)Node conntrackCoreDNSAAAA dropped by a conntrack racewaits 5 s for AAAA, then retriesA query ⎫ sameAAAA query ⎭ socketA query passesA answerFixes: NodeLocal DNSCache, options single-request-reopen, or use-vc (TCP) in resolv.conf.
Figure 1. The famous 5-second DNS delay in Kubernetes: two queries from the same socket race inside conntrack, one is dropped, and the client waits 5 seconds to retry.
Swipe sideways to see all of it, or tap to open it full screen.

Security group rules to remember

  • Instances must reach DNS: outbound UDP 53 and TCP 53 (to the VPC resolver this is handled implicitly — the Route 53 Resolver at base+2 is always reachable — but NACLs and custom resolvers need explicit rules).
  • NTP: outbound UDP 123 (or use the Amazon Time Sync Service at 169.254.169.123, always reachable).
  • NACLs: return traffic for both TCP and UDP uses ephemeral ports — allow 1024–65535 in the reverse direction.

Application patterns

  • Connection pooling for TCP (databases, HTTP clients): reuse connections to avoid handshake and TLS cost and slow start on every request; size pools sensibly; set max-idle below middlebox idle timeouts.
  • Timeouts everywhere: connect timeout (short, e.g. 1–3 s), read timeout, overall deadline. A missing timeout turns a network blip into hung threads.
  • Retries with exponential backoff and jitter, only for idempotent operations; budget retries to avoid retry storms.
  • For UDP-based logging (syslog), losses under load are silent — use TCP/TLS syslog, or a local agent (Fluent Bit) that buffers and forwards reliably.
Enlarge diagramnew connectionevery requestTCPTLSslow startrequestTCPTLSslow startrequestpooledreuse a warm onerequestrequestrequestrequestrequestrequestSet pool max-idle below the shortest idle timeout on the path, and always set connect and read timeouts.
Figure 2. Why connection pools matter: every new connection pays for a handshake, TLS setup and slow start before it is fast.
Swipe sideways to see all of it, or tap to open it full screen.