Safeguard
Supply Chain Security•

TCP Explained: RFC 9293 Handshake, State Machine and Hardening

TCP is the reliable byte stream under most of the internet. How RFC 9293's handshake and 11 states work, the weakness classes, and how to harden hosts.

Safeguard Security Team
46 min read

TCP, the Transmission Control Protocol, is the transport layer protocol that turns unreliable IP packets into a reliable, ordered, flow controlled byte stream between two programs. It is defined today by RFC 9293 (Internet Standard STD 7, August 2022), which replaced the original 1981 RFC 793. Web, mail, SSH and database traffic all ride on it.

Most of what goes wrong with TCP in production is not about the bytes on the wire. It is about state: how many half open connections a listener will hold, how long a socket sits in TIME-WAIT, whether a stray reset is believed, and which kernel knobs someone changed three years ago. This post walks through the RFC 9293 connection lifecycle state by state, explains congestion control at a concept level, sets out the weakness classes the standards themselves describe, and gives the commands a defender runs on Linux, Windows and macOS to check their own hosts. What an eavesdropper can read and alter in the headers is covered in our companion post, TCP vs UDP: what an eavesdropper can see and change, so this one stays on the state machine and host hardening.

Key takeaways

  • RFC 9293 (August 2022) is the current TCP specification. It obsoletes RFC 793 and six RFCs that patched it (879, 2873, 6093, 6429, 6528 and 6691), updates RFCs 1011, 1122 and 5961, and folds in the keyed initial sequence number algorithm that used to live in RFC 6528.
  • A TCP connection moves through 11 states: LISTEN, SYN-SENT, SYN-RECEIVED, ESTABLISHED, FIN-WAIT-1, FIN-WAIT-2, CLOSE-WAIT, CLOSING, LAST-ACK, TIME-WAIT and the fictional CLOSED. Every one of them is visible with ss, Get-NetTCPConnection or netstat, and odd counts in particular states are often the first sign of trouble.
  • RFC 9293 is blunt about security: TCP has "no built-in cryptographic capabilities". Its non-cryptographic defences (keyed ISNs, RFC 5961 reset and SYN checks, SYN cookies) only make blind attacks harder. Confidentiality and authentication come from TLS, IPsec or, for long lived router sessions, TCP-AO (RFC 5925).
  • The main weakness classes are handshake state exhaustion (SYN flooding, RFC 4987), blind in-window injection (RFC 5961), sequence number predictability (fixed by RFC 6528, now in RFC 9293), congestion control abuse (RFC 5681) and host fingerprinting.
  • On Linux, the defaults are mostly sensible: SYN cookies on (tcp_syncookies = 1), timestamps with random per connection offsets, server side Fast Open off, keepalive at two hours. Hardening is mostly about verifying those defaults, closing listeners you do not need and not tuning knobs the kernel documentation tells you to leave alone.
  • Pick TCP for reliable, ordered streams where middlebox compatibility matters most; QUIC when you want built in encryption, stream multiplexing without head of line blocking and connection migration; SCTP for message oriented telecom signalling with multi-homing; UDP when the application must own reliability and timing.

What TCP is for, where it sits and which standard defines it

IP delivers individual packets on a best effort basis. Packets can arrive out of order, twice or not at all, and IP does nothing about it. TCP sits one layer up, at the transport layer, and gives applications a different abstraction: a full duplex stream of bytes that arrives complete and in order, with the sender slowing down when the receiver or the network cannot keep up. In the IANA protocol numbers registry TCP is IP protocol number 6, referenced to RFC 9293. If you want a refresher on how the transport layer maps to the OSI and TCP/IP models, see TCP/IP model vs OSI model.

TCP provides five things the application would otherwise have to build itself:

  1. Connection setup and teardown, so both ends agree a conversation has started and finished.
  2. Reliability, through sequence numbers, acknowledgments and retransmission.
  3. Ordering, so the receiver hands bytes to the application in the order they were sent.
  4. Flow control, through the receive window, so a fast sender cannot overrun a slow receiver.
  5. Congestion control, so senders back off when the network path is overloaded.

What RFC 9293 consolidated

For four decades the TCP specification was RFC 793 (September 1981) plus a pile of later documents that changed parts of it. RFC 9293, edited by Wesley Eddy and published in August 2022 as Internet Standard 7, collects those changes into one document. Its abstract says it brings together changes that had "only been documented in a piecemeal fashion."

According to its header and section 5, RFC 9293:

ActionRFCsWhat that means
Obsoletes793, 879, 2873, 6093, 6429, 6528, 6691The original spec, plus updates on MSS (879, 6691), the IP precedence field (2873), the urgent pointer (6093), zero window probing and resource reclamation (6429) and keyed ISN generation (6528)
Updates1011, 1122The TCP portions of the 1989 host requirements (RFC 1122) are effectively replaced
Updates5961Adds a clarification on reset handling in SYN-RECEIVED
Incorporates3168The CWR and ECE control bits from Explicit Congestion Notification
IncorporatesAccepted RFC 793 errataNineteen errata IDs are listed in section 5

RFC 9293 does not obsolete everything about TCP. Window scaling and timestamps still live in RFC 7323, SACK in RFC 2018, congestion control in RFC 5681 and its successors, and the retransmission timer in RFC 6298. RFC 7414 (February 2015) is the IETF's roadmap to that wider family of documents.

How TCP works on the wire

The header, read as a state carrier

Every TCP segment carries a header of at least 20 bytes followed by options and data. Our eavesdropper post goes through what each field reveals to someone watching the network. Here the question is different: what does each field do for the connection's state?

FieldSizeRole in the connection lifecycle
Source port, destination port16 bits eachWith the two IP addresses, identify the connection (the four tuple)
Sequence number32 bitsPosition of the first data byte; on a SYN it carries the initial sequence number (ISN)
Acknowledgment number32 bitsNext byte the sender expects to receive; always sent once the connection is established
Data offset4 bitsHeader length in 32 bit words, so where the options end and data begins
Reserved4 bitsMust be zero when sent
Control bits8 bitsCWR, ECE, URG, ACK, PSH, RST, SYN, FIN; these drive every state transition
Window16 bitsHow many bytes the receiver will accept; shifted when window scaling is negotiated
Checksum16 bitsCovers header, data and a pseudo header with the IP addresses; mandatory to send and check
Urgent pointer16 bitsOffset to the end of urgent data when URG is set; RFC 9293 discourages new use
Options0 to 40 bytesMSS, window scale, SACK, timestamps and others

The checksum detects accidental corruption. It does not authenticate anything, because anyone who changes a segment can recompute it.

Options: MSS, window scale, SACK and timestamps

RFC 9293 requires every implementation to support three options: End of Option List (kind 0), No-Operation (kind 1) and Maximum Segment Size (kind 2). It must also ignore, without error, any option it does not implement. The IANA TCP parameters registry lists the rest. The ones you will see on almost every connection:

KindLengthOptionDefined inWhat it does
24Maximum Segment SizeRFC 9293Largest segment the sender of the SYN can receive. Sent only on SYN segments. Without it, a sender must assume 536 bytes for IPv4 or 1220 bytes for IPv6
33Window ScaleRFC 7323A shift count applied to the 16 bit window. The exponent is capped at 14, so the largest window is 1 GiB
42SACK PermittedRFC 2018Says "I understand selective acknowledgments". Only valid on a SYN
5variableSACKRFC 2018Lists up to four blocks of out of order data already received (three when timestamps are also present), so only the real gaps are resent
810TimestampsRFC 7323TSval and TSecr values for round trip measurement and PAWS (Protection Against Wrapped Sequences)
29variableTCP-AORFC 5925Message authentication code over the segment
30variableMultipath TCPRFC 8684Signals MPTCP; subtypes carry the individual messages
34variableFast Open CookieRFC 7413Lets a returning client send data in its SYN

Kind 19, the old TCP MD5 Signature option from RFC 2385 (August 1998), is marked in the registry as obsoleted by option 29.

Window scale and timestamps exist because the original 16 bit window and 32 bit sequence space were sized for 1981 networks. On a fast, long path, 64 KB in flight is nowhere near enough, and at high rates the sequence number can wrap around within the lifetime of a segment, which PAWS uses timestamps to detect.

The three-way handshake

RFC 9293 section 3.5 gives the canonical example. Peer A picks an initial sequence number of 100, peer B picks 300.

     Peer A (client)                                    Peer B (server)
1. CLOSED                                             LISTEN
2. SYN-SENT    --> SEQ=100 CTL=SYN               -->  SYN-RECEIVED
3. ESTABLISHED <-- SEQ=300 ACK=101 CTL=SYN,ACK   <--  SYN-RECEIVED
4. ESTABLISHED --> SEQ=101 ACK=301 CTL=ACK       -->  ESTABLISHED
5. ESTABLISHED --> SEQ=101 ACK=301 CTL=ACK DATA  -->  ESTABLISHED

Three details matter for hardening:

  • The SYN consumes one sequence number. That is why B acknowledges 101, not 100. A bare ACK does not consume sequence space, which is why line 5 reuses 101.
  • Options are negotiated here and only here. MSS, window scale and SACK Permitted ride on the SYN and SYN-ACK. If a middlebox or a SYN cookie implementation strips them, the connection runs without them for its whole life.
  • The server holds state from line 2 to line 4. Between receiving the SYN and receiving the final ACK, B has allocated something to remember the half open connection. That window is what SYN flooding targets.

RFC 9293 says the principal reason for the three-way handshake is "to prevent old duplicate connection initiations from causing confusion." The handshake also handles simultaneous open, where both ends send a SYN at the same time and each moves CLOSED, SYN-SENT, SYN-RECEIVED, ESTABLISHED. Implementations must support it.

Data transfer: sequence numbers, acknowledgments and windows

Once established, each side numbers every byte it sends. The receiver acknowledges the next byte it expects, cumulatively. The sender keeps unacknowledged data and retransmits it if no acknowledgment arrives before the retransmission timeout (RTO). RFC 6298 (June 2011) specifies that timer and sets the initial RTO to 1 second, rounding any computed value below 1 second up to 1 second.

The receive window tells the sender how much more it may send. The congestion window, covered below, is the sender's own estimate of what the network can carry. RFC 5681 states the rule simply: the minimum of the two governs transmission.

RFC 9293 section 3.8.3 also sets two retransmission thresholds. At R1 (at least three retransmissions) the stack tells IP to check its route and should warn the application. At R2 (at least 100 seconds) it gives up and closes the connection. On Linux, tcp_retries2 controls R2; its default of 15 yields a timeout of about 924.6 seconds according to the kernel documentation.

Teardown: FIN, half close and RST

TCP closes each direction separately. A FIN means "I have no more data to send", not "go away". RFC 9293 shows the normal sequence:

     Peer A (closes first)                              Peer B
1. ESTABLISHED                                        ESTABLISHED
2. FIN-WAIT-1  --> SEQ=100 ACK=300 CTL=FIN,ACK   -->  CLOSE-WAIT
3. FIN-WAIT-2  <-- SEQ=300 ACK=101 CTL=ACK       <--  CLOSE-WAIT
4. (B's application calls close)
     TIME-WAIT   <-- SEQ=300 ACK=101 CTL=FIN,ACK   <--  LAST-ACK
5. TIME-WAIT   --> SEQ=101 ACK=301 CTL=ACK       -->  CLOSED
6. (waits 2 x MSL, then) CLOSED

Between steps 3 and 4, B can keep sending data to A. That is a half closed connection, and it is legal. Step 4 depends on B's application actually calling close, which is why a pile of sockets stuck in CLOSE-WAIT almost always points at an application bug rather than a network one.

The other way out is an abort. A segment with the RST bit tells the peer to discard the connection immediately. RFC 9293 requires the stack to tell the application whether a connection closed normally or was aborted.

The state machine, state by state

RFC 9293 names 11 states. CLOSED is "fictional" because it means there is no transmission control block (TCB) at all. The table adds what each state usually means when you see it on a real host.

StateRFC 9293 meaningWhat a pile of these on your host usually means
LISTENWaiting for a connection requestA service is accepting connections. Each one should be something you intend to expose
SYN-SENTSent a SYN, waiting for a matching oneOutbound connections to something that is not answering, often a dropped firewall rule or a dead peer
SYN-RECEIVEDReceived and sent a SYN, waiting for the final ACKNormal in small numbers. Large counts suggest a SYN flood, lost ACKs, or an asymmetric route
ESTABLISHEDOpen connection, data flowsNormal. Compare against expected client counts
FIN-WAIT-1Sent FIN, waiting for its ACK or the peer's FINShort lived. Many suggests peers not acknowledging, sometimes a sign of resource holding
FIN-WAIT-2Our FIN acknowledged, waiting for the peer's FINPeer has not closed its side. Orphaned ones are reaped by tcp_fin_timeout on Linux
CLOSE-WAITPeer sent FIN, waiting for our application to closeAlmost always an application that forgot to close sockets
CLOSINGBoth sent FIN simultaneously, waiting for ACKRare
LAST-ACKSent our FIN after the peer's, waiting for its ACKShort lived
TIME-WAITWaiting so delayed segments die and the final ACK can be resentNormal on the side that closed first. Large counts on busy clients or proxies
CLOSEDNo connection stateNot shown by most tools

The state diagram in RFC 9293 (Figure 5) carries a warning: "This diagram is only a summary and must not be taken as the total specification." Two transitions it leaves out matter in practice. A RST received in any state sends the connection to CLOSED, or back to LISTEN for a passively opened connection still in SYN-RECEIVED, and a FIN-WAIT-1 connection that receives a FIN together with the ACK of its own FIN goes straight to TIME-WAIT.

Why TIME-WAIT exists

TIME-WAIT is the state people most often try to "fix", so it is worth being precise. RFC 9293 requires the side that closes first to linger in TIME-WAIT for twice the Maximum Segment Lifetime (MSL). It takes MSL to be two minutes, calling it "an engineering choice" that may change with experience.

The wait does two jobs. If the final ACK is lost, the peer will resend its FIN, and the TIME-WAIT socket is still there to acknowledge it. And it stops a new connection with the same four tuple from receiving stray segments left over from the old one. Shortcutting it trades a small, rare data integrity risk for port reuse. The Linux kernel documentation says tcp_tw_reuse "should not be changed without advice/request of technical experts", and that tcp_max_tw_buckets exists "only to prevent simple DoS attacks" and must not be lowered artificially.

Ports and IANA

Ports are 16 bit numbers. RFC 6335 divides them into System Ports (0 to 1023), User Ports (1024 to 49151) and Dynamic Ports (49152 to 65535, never assigned). IANA keeps the service name and port number registry. Operating systems choose their own ephemeral range for outbound connections; on Linux, the kernel documentation gives the default ip_local_port_range as 32768 to 60999, which is narrower than the IANA dynamic range and is worth knowing when a busy proxy runs out of source ports.

Congestion control, at a concept level

Flow control protects the receiver. Congestion control protects the network. RFC 5681 (September 2009) defines four algorithms every TCP sender must follow, and says a TCP "MUST NOT be more aggressive" than they allow:

  • Slow start. A new connection does not know the path's capacity, so it starts with a small congestion window (cwnd) and grows it quickly as acknowledgments arrive. RFC 5681 describes this as starting the "ACK clock".
  • Congestion avoidance. Once cwnd passes the slow start threshold (ssthresh), growth becomes roughly linear.
  • Fast retransmit. Duplicate acknowledgments suggest a segment was lost; the sender resends it without waiting for the timer.
  • Fast recovery. After a fast retransmit, the sender reduces cwnd instead of collapsing back to slow start.

All four treat loss as the congestion signal. Explicit Congestion Notification (RFC 3168, September 2001) lets routers mark packets instead of dropping them, which is what the CWR and ECE header bits are for.

CUBIC, now RFC 9438 (August 2023), replaces the linear growth of congestion avoidance with a cubic function of time since the last reduction, which scales better on fast, long distance paths. RFC 9438 states that CUBIC "has been adopted as the default TCP congestion control algorithm by the Linux, Windows, and Apple stacks". It cuts the window by a factor of 0.7 on loss, where classic Reno halves it, and moves CUBIC from Informational (RFC 8312) to the Standards Track. On Linux, the active algorithm is the tcp_congestion_control sysctl; on Windows, the CongestionProvider property of the TCP setting templates.

Congestion control has its own security angle. RFC 5681's Security Considerations note that an attacker who can drop packets or forge duplicate acknowledgments can slow a connection down, and that forging acknowledgments could make senders more aggressive, which "could drive a portion of the network into congestion collapse." Its fix for the ACK division attack is to grow cwnd by bytes acknowledged rather than by number of ACKs.

Versions and history

TCP has never had a "version 2". It has evolved through extensions negotiated per connection and through specification updates. The milestones:

DateDocumentWhat changed and why
Sep 1981RFC 793The original TCP specification, STD 7
Oct 1989RFC 1122Host requirements: corrections and clarifications to TCP, including keepalive rules
May 1992RFC 1323Window scale, timestamps and PAWS for high bandwidth paths
May 1996RFC 1948Keyed, per connection ISNs to resist sequence number guessing
Oct 1996RFC 2018Selective acknowledgment
Aug 1998RFC 2385TCP MD5 Signature option to protect BGP sessions; later deprecated
Sep 2001RFC 3168Explicit Congestion Notification and the CWR and ECE bits
Aug 2007RFC 4987Informational survey of SYN flooding and its defences
Sep 2009RFC 5681Current congestion control baseline
Jun 2010RFC 5925TCP Authentication Option, obsoleting TCP MD5
Aug 2010RFC 5961Defences against blind in-window resets, SYNs and data injection
Jun 2011RFC 6298Retransmission timer, initial RTO of 1 second
Feb 2012RFC 6528Keyed ISN algorithm on the Standards Track, obsoleting RFC 1948
Sep 2014RFC 7323Replaces RFC 1323; recommends random per connection timestamp offsets
Dec 2014RFC 7413TCP Fast Open, Experimental
Mar 2020RFC 8684Multipath TCP version 1, obsoleting RFC 6824
Feb 2021RFC 8985RACK-TLP loss detection
Aug 2022RFC 9293Consolidated TCP specification, STD 7
Aug 2023RFC 9438CUBIC on the Standards Track, obsoleting RFC 8312

What is deprecated: TCP MD5 (RFC 9293 says it "had flaws and is now deprecated"), use of the urgent pointer by new applications, and the old global counter ISN scheme. What is deployed almost everywhere: SACK, window scaling, timestamps, keyed ISNs and the RFC 5961 checks. What is deployed selectively: TCP-AO (mainly between routers), Fast Open (client support is common, server support is opt in) and Multipath TCP.

Security properties: what TCP guarantees and what it does not

RFC 9293 section 7 opens with the sentence every TCP security discussion should start from: TCP includes only rudimentary robustness features and "no built-in cryptographic capabilities to support any form of confidentiality, authentication, or other typical security functions."

PropertyDoes TCP provide it?Where it comes from instead
ConfidentialityNoTLS above, IPsec below
Integrity against tamperingNo. The checksum only catches accidentsTLS for the payload; TCP-AO or IPsec for headers too
Peer authenticationNo. Addresses are claims, not proofTLS certificates, SSH keys, IPsec, TCP-AO shared keys
Replay protectionPartly, through sequence numbers, against stale segments, not attackersTLS records; TCP-AO includes replay protection
Reliable, ordered deliveryYes, against loss and reordering, not against a malicious path
AvailabilityNo. State can be exhausted and connections resetHost and network defences, rate limiting, capacity

RFC 9293 adds that IPsec is "the only current effective method" to fully protect TCP connections including their control flags, that TCP-AO gives integrity and authentication but no privacy, and that TLS protects the payload but none of the TCP header. Non-cryptographic features such as ISN generation and sequence checks "are only capable of partially hindering attacks."

Weakness classes, described as concepts

These are taken from the Security Considerations of RFC 9293 and the RFCs it cites. They describe what goes wrong and why the design allows it, not how to do it.

Handshake state exhaustion: SYN flooding

The server must remember a connection between receiving the SYN and receiving the final ACK. If someone sends many SYNs from forged addresses, the final ACKs never come, and the listener's queue of half open connections fills until real clients are refused. RFC 4987 (August 2007) explains that the main damage is not CPU or memory but spoofed SYNs "filling up the often small listener's queue."

RFC 4987 lists the common defences: filtering, increasing the backlog, reducing the SYN-RECEIVED timer, recycling the oldest half open entry, SYN caches, SYN cookies, hybrids, and firewalls and proxies. SYN cookies are the best known. The server keeps no state at all for the half open connection; instead it encodes what it needs into the sequence number of its SYN-ACK using a keyed hash with time dependent bits, and rebuilds the connection only when a genuine ACK comes back.

SYN cookies have costs RFC 4987 spells out. Common schemes cannot encode every option, so window scaling and SACK may be lost, and an MSS has to be squeezed into a few bits. They cannot handle data carried on the SYN. And when the server speaks first, as an SMTP server does with its greeting, a lost final ACK leaves the client thinking it is connected while the server never heard of it. That is why the Linux kernel documentation describes tcp_syncookies as a "fallback facility" that "MUST NOT be used to help highly loaded servers to stand against legal connection rate." Our post on SYN floods, DDoS and API abuse covers the network side of flood defence.

Blind in-window injection and resets

A TCP stack accepts a segment if its four tuple matches and its sequence number falls inside the receive window. An attacker who cannot see the traffic has to guess, and older rules made the guess easy: any RST inside the window was accepted. RFC 5961 (August 2010) addresses three variants.

  • RST. Accept a reset only if its sequence number exactly matches the next expected byte. If it is merely in the window, send a "challenge ACK" to the real peer, which only a genuine peer can answer with an exact reset.
  • SYN in a synchronized state. RFC 793 told the stack to reset the connection on an in-window SYN. RFC 5961 says to send a challenge ACK for any SYN, regardless of sequence number, so a spoofed SYN cannot tear the connection down.
  • Data. Optionally require the acknowledgment number to fall within a bounded range too, which also hardens against spoofed FINs.

RFC 5961's own Security Considerations are clear about the limits: the changes "do NOT protect an implementation from on-path attacks", make off-path injection harder but "does NOT make it impossible", and full protection requires IPsec. The eavesdropper post walks through reset injection and the challenge ACK side channel in more depth.

Initial sequence number predictability

The ISN is what a blind attacker must guess to complete a forged handshake. RFC 6528 traces the problem to at least 1985, when Morris described attacks on predictable ISNs. RFC 9293 now requires a clock based ISN and recommends this form:

ISN = M + F(localip, localport, remoteip, remoteport, secretkey)

M is a timer that ticks roughly every 4 microseconds, and F is a keyed pseudorandom function that "MUST NOT be computable from the outside". The clock keeps ISNs unique across the 2 minute MSL (it cycles about every 4.55 hours); the keyed hash means learning one connection's ISN tells you nothing about another's. Embedded and firmware TCP/IP stacks written from older texts are where weak ISN generation still turns up, which is why it remains worth asking about when you assess a device. Our post on CVE-2023-45237 in the EDK II network stack is one example.

Connection state and resource holding

Beyond the SYN queue, RFC 9293 lists "wasting resources on non-progressing connections" as a resource attack. Connections that complete the handshake and then barely move, peers that advertise a zero window and never open it, and sockets orphaned in FIN-WAIT states all hold memory. RFC 6429 (now obsoleted into RFC 9293) clarified that a stack may reclaim resources from such connections. The Linux kernel exposes limits such as tcp_max_orphans, whose documentation warns that each orphan "eats up to ~64K of unswappable memory", and tcp_fin_timeout, which defaults to 60 seconds for orphaned FIN-WAIT-2 sockets.

Abusing congestion control

Covered above: an attacker who can drop or forge acknowledgments can slow a connection or, in theory, make senders too aggressive. RFC 9293 names ACK division as an example and Appropriate Byte Counting as the mitigation that went into the congestion control specifications.

ACK loops

The Linux kernel documentation describes "ack loop" denial of service, where a buggy or malicious middlebox rewrites header fields so each endpoint thinks the other is sending invalid segments, and both answer with an endless stream of duplicate ACKs. The tcp_invalid_ratelimit sysctl limits how fast the kernel answers out of window segments to break such loops.

ICMP interactions

ICMP errors can change TCP behaviour, for example by signalling that a destination is unreachable or that the path MTU is smaller. RFC 9293 points to RFC 5927 (July 2010), "ICMP Attacks against TCP", for the attack classes and the mitigations stacks adopted. A related operational problem is the PMTU black hole, where ICMP is filtered and large segments vanish; Linux offers tcp_mtu_probing to fall back to packetization layer probing.

Fingerprinting and timestamp leaks

RFC 9293 notes that the options a stack sends, their order, its timing and its reactions to odd conditions can identify the operating system. RFC 7323 adds a specific privacy point: a timestamp taken straight from the system uptime clock "may unintentionally leak this information", so it recommends a random per connection offset. On Linux, tcp_timestamps = 1 (the default) does exactly that; 2 turns the random offsets off.

TCP Fast Open and TCP-AO: when they matter

TCP Fast Open (RFC 7413)

Normally no application data is delivered until the handshake completes, costing a full round trip. Fast Open lets a client that has connected before present a cookie (option 34) and put data in its SYN, which the server can hand to the application straight away.

RFC 7413 is Experimental and comes with two cautions. First, data in a SYN can be delivered more than once if the SYN is duplicated in the network, so a client or server "that cannot handle receiving the same SYN data more than once MUST NOT enable TFO". Second, the server does work before the handshake proves the client's address, so RFC 7413 requires a limit on pending Fast Open requests, after which the server falls back to an ordinary handshake and SYN cookies can take over. It also says implementations "MUST NOT use TFO by default", only when an application asks for it per service port.

On Linux the tcp_fastopen sysctl is a bitmap. Client support (0x1) is on by default; server support (0x2) is off by default. Turn it on only for idempotent first requests and pair it with key management through tcp_fastopen_key, which supports a primary and a backup key for rotation. Note that TLS 1.3 has its own early data mechanism with similar replay concerns; our TLS 1.3 handshake post covers it.

TCP Authentication Option (RFC 5925)

TCP-AO (June 2010) adds a message authentication code to each segment, computed with a key both peers already share. It covers the TCP header, including the control flags, and the payload, so a forged reset or injected segment fails verification. It obsoletes TCP MD5, adding algorithm agility, key rollover within a connection and replay protection for long lived connections. The Linux kernel's TCP-AO documentation names HMAC-SHA-1-96 and AES-128-CMAC-96 as its MAC algorithms, with implementations free to add others, against a plain MD5 of data and key for the old option.

TCP-AO does not encrypt anything, and it needs keys configured out of band on both ends. That makes it a fit for a small number of long lived, high value sessions between known peers, the classic case being BGP between routers (see our BGP explainer). For anything with many clients, use TLS.

Multipath TCP (RFC 8684)

MPTCP version 1 (March 2020) lets one connection use several paths at once, for example Wi-Fi and cellular, using option 30. Linux ss can show MPTCP sockets with -M. From a security point of view, it means a single logical connection can involve more than one address pair, which matters for firewalls and monitoring that assume one four tuple per connection.

Secure configuration, with defender-side check commands

Only run these against systems you own or are authorised to test. Every command below is taken from the official documentation cited: the Linux ss(8) man page and kernel ip-sysctl documentation, Microsoft Learn for the Windows cmdlets and netstat, and the macOS netstat(1) and lsof(8) manual pages.

Step 1: list what is listening

Every LISTEN socket is attack surface. Start here.

# Linux: TCP listeners, numeric, with owning process
ss -t -l -n -p

# Linux: every TCP socket in every state
ss -t -a
# Windows: listening sockets and their owning process IDs
Get-NetTCPConnection -State Listen

# Windows: classic view with process names (-b fails without sufficient permissions)
netstat -anob
# macOS: TCP sockets in LISTEN with numeric addresses and ports
lsof -iTCP -sTCP:LISTEN -n -P

# macOS: all TCP sockets numeric
netstat -an -p tcp

For each listener, ask three questions: does it need to exist, does it need to bind to all interfaces rather than loopback, and is the process the one you expect.

Step 2: count connections by state

Odd state distributions are your early warning.

# Linux: summary counts across socket types and states
ss -s

# Linux: half open connections waiting for the final ACK
ss -t -n state syn-recv

# Linux: sockets whose application has not closed them
ss -t -n state close-wait

# Linux: TIME-WAIT and SYN-RECV minisockets together
ss -t -n state bucket

# Linux: established SSH sessions with timer information
ss -o state established '( dport = :ssh or sport = :ssh )'
# Windows: connections in a given state
Get-NetTCPConnection -State SynReceived
Get-NetTCPConnection -State CloseWait
Get-NetTCPConnection -State TimeWait

# Windows: per protocol statistics
netstat -s -p tcp
# macOS: listen queue sizes (unaccepted, incomplete, maximum)
netstat -L

# macOS: per protocol statistics (root needed for TCP counters)
netstat -s -p tcp

A rising SYN-RECEIVED count with the listener's backlog full points at a flood or at lost ACKs. A rising CLOSE-WAIT count points at your own code. On macOS, the second figure from netstat -L is the number of incomplete connections waiting on a listener.

Step 3: verify the Linux kernel's TCP defences

Read the values first. Change them only with a reason, and persist changes in your configuration management.

# SYN cookies: 1 = send when the SYN backlog overflows (documented default)
sysctl net.ipv4.tcp_syncookies

# Per listener half open queue limit; also check the accept queue cap
sysctl net.ipv4.tcp_max_syn_backlog
sysctl net.core.somaxconn

# SYN-ACK retransmissions: default 5, about 63 seconds to give up
sysctl net.ipv4.tcp_synack_retries

# Timestamps: 1 = on with random per connection offset (default)
sysctl net.ipv4.tcp_timestamps

# Fast Open bitmap: client 0x1 on by default, server 0x2 off by default
sysctl net.ipv4.tcp_fastopen

# RFC 5961 challenge ACK limit: default unlimited, per socket limits apply
sysctl net.ipv4.tcp_challenge_ack_limit

# Duplicate ACK rate limit for invalid segments, in milliseconds
sysctl net.ipv4.tcp_invalid_ratelimit

# Orphaned FIN-WAIT-2 lifetime: default 60 seconds
sysctl net.ipv4.tcp_fin_timeout

# TIME-WAIT reuse: default 2, loopback only; leave unless advised
sysctl net.ipv4.tcp_tw_reuse

# Congestion control in use and available
sysctl net.ipv4.tcp_congestion_control
sysctl net.ipv4.tcp_available_congestion_control

# Ephemeral port range for outbound connections
sysctl net.ipv4.ip_local_port_range

What good looks like, based on the kernel documentation:

SettingDocumented defaultHardening stance
tcp_syncookies1Keep at 1. Do not set 2 (always on) except for testing, because cookies drop TCP extensions
tcp_max_syn_backlog128 minimum, scales with memoryRaise on genuinely busy listeners, together with somaxconn and the application's own backlog
tcp_synack_retries5Lowering it shortens how long half open entries linger; RFC 4987 lists a reduced SYN-RECEIVED timer as a defence, at some cost to slow clients
tcp_timestamps1Keep at 1 for PAWS and the random offset. Turning timestamps off removes PAWS protection on fast links
tcp_sack, tcp_window_scaling1, 1Keep on. Disabling them hurts performance and does not fix vulnerable stacks
tcp_fastopenclient on, server offLeave server side off unless the service's first request is idempotent
tcp_challenge_ack_limitunlimitedLeave as is; the kernel documentation warns a per namespace limit "can allow some side channel attacks"
tcp_abort_on_overflowoffLeave off. The documentation says enabling it "can harm clients of your server"
tcp_tw_reuse, tcp_max_tw_buckets2, memory basedDo not lower bucket limits; the documentation says to raise them if needed
tcp_rfc13370When 0, the kernel deviates from RFC 1337 to "prevent TCP TIME_WAIT assassination"; read the documentation before changing
tcp_keepalive_time2 hoursApplications that need faster dead peer detection should set keepalive per socket, not globally

Step 4: check Windows TCP settings

Windows applies TCP settings through templates such as Internet, Datacenter, Compat and Custom. Microsoft documents reading them with Get-NetTCPSetting and changing the custom templates with Set-NetTCPSetting.

# All TCP setting templates in a table
Get-NetTcpSetting | Format-Table

# The Internet template only
Get-NetTCPSetting -Setting Internet

# Which template live connections use
Get-NetTCPConnection -AppliedSetting Internet

The properties to read include CongestionProvider (CTCP, DCTCP or Default), EcnCapability, Timestamps, InitialRtoMs, MaxSynRetransmissions and AutoTuningLevelLocal. Microsoft's documentation says the -SettingName parameter of Set-NetTCPSetting accepts only Custom templates, and its example modifies InternetCustom. As on Linux, verify first and change only with a measured reason.

Step 5: restrict exposure

Kernel tuning is the last line, not the first. In order of impact:

  1. Do not listen where you do not need to. Bind admin interfaces, databases and metrics endpoints to loopback or a private interface.
  2. Filter at the host. A default deny inbound policy with explicit allows, using nftables, Windows Defender Firewall or the macOS application firewall, shrinks what a SYN flood or a scan can reach.
  3. Put a proxy or load balancer in front of public listeners. RFC 4987 lists firewalls and proxies as a SYN flood defence because they absorb the handshake before your application sees it.
  4. Encrypt and authenticate above TCP. TLS 1.3 for applications, SSH for administration, TCP-AO for router sessions, IPsec where you need the whole header protected.
  5. Set application timeouts. Idle, header and body read timeouts in your servers are what stop slow, non-progressing connections from holding resources, since TCP keepalives default to off per connection and to two hours when on.
  6. Log state anomalies. The kernel logs SYN flood warnings when cookies are sent; alert on them and on unusual SYN-RECEIVED or CLOSE-WAIT counts.

Step 6: keep the stack itself current

TCP lives in your operating system kernel, and in some cases in user space libraries or firmware. The fixes for TCP stack vulnerabilities ship as kernel and firmware updates from your distribution or vendor, so patch cadence and reboots (or live patching) matter as much as configuration.

TCP vs UDP vs QUIC vs SCTP: when to use which

NeedTCP (RFC 9293)UDP (RFC 768)QUIC (RFC 9000)SCTP (RFC 9260)
Delivery modelReliable, ordered byte streamUnreliable datagramsReliable streams, multiplexed, over UDPReliable messages over multiple streams
Built in encryptionNoneNoneIntegrated TLS 1.3 handshake, always onNone (use DTLS or IPsec)
Head of line blockingYes, one lost segment stalls the streamNot applicableOnly streams with data in the lost packet stallPer stream
HandshakeThree-way, no data until complete (unless Fast Open)NoneCombined transport and crypto handshakeFour-way with a state cookie, resists SYN style floods by design
Multi-homing or migrationOnly with MPTCPApplication's jobConnection migrationBuilt in multi-homing
Typical usesWeb over TLS, SSH, databases, mail, most APIsDNS, NTP, VoIP media, games, QUIC itselfHTTP/3, modern web and mobileTelecom signalling, some cluster interconnects
Middlebox compatibilityExcellentGood, though some networks filter itGood where UDP 443 is openVaries; check NAT and firewall support on your path
Operational cost of securingAdd TLS and manage certificates; tune kernelApplication must handle spoofing and amplificationCertificates as for TLS; UDP visibility toolingDTLS or IPsec, specialised firewalling

For the UDP side, see UDP explained. RFC 9260 describes SCTP's handshake as a four-way exchange using a cookie mechanism "to provide protection against synchronization attacks", which is the design choice TCP later bolted on with SYN cookies. QUIC's RFC 9000 calls avoiding head of line blocking across streams one of its benefits, while noting that a lost packet carrying several streams still blocks all of them.

Common mistakes

  • Treating SYN cookies as a capacity plan. They are a fallback for overflow. If the kernel logs SYN flood warnings during normal peaks, the fix is a bigger backlog, a faster accept loop or more servers.
  • Disabling timestamps "for security". The uptime leak was addressed by random per connection offsets. Turning timestamps off removes PAWS and better round trip measurement.
  • Fighting TIME-WAIT with global knobs. Lots of TIME-WAIT sockets on a client or proxy are normal. Connection reuse (HTTP keepalive, connection pools) fixes the cause; aggressive reuse settings trade away the protection TIME-WAIT exists for.
  • Ignoring CLOSE-WAIT. These are your application failing to close sockets. They accumulate until file descriptors run out.
  • Assuming TLS protects the connection itself. TLS protects the bytes. A forged reset still ends the connection, which is why long lived router sessions use TCP-AO.
  • Still using TCP MD5 for BGP where TCP-AO is available. RFC 9293 calls MD5 deprecated; RFC 5925 explains why.
  • Enabling server side Fast Open for non idempotent requests. A replayed SYN could deliver the same request twice.
  • Binding everything to 0.0.0.0. The cheapest TCP hardening is a listener that does not exist on the public interface.
  • Relying on TCP keepalive for dead peer detection. It is off by default per socket and, when on, waits two hours. Use application level timeouts or heartbeats.
  • Assuming a container scan covers the host's TCP stack. The TCP implementation is in the host kernel, which image scanning does not see. Track kernel versions separately.

FAQ

What port does TCP use?

TCP itself does not have a port; it is IP protocol number 6. Each application chooses a port from the 16 bit space. IANA assigns well known System Ports (0 to 1023) and User Ports (1024 to 49151) through its service name and port registry, for example 443 for HTTPS and 22 for SSH. Ports 49152 to 65535 are Dynamic Ports and are never assigned.

Is TCP encrypted?

No. RFC 9293 says TCP has no built in cryptographic capabilities for confidentiality or authentication. Encryption comes from TLS running over TCP, from IPsec underneath it, or from a different transport such as QUIC.

What is the TCP three-way handshake?

The client sends a SYN with its initial sequence number, the server replies with a SYN-ACK carrying its own initial sequence number and acknowledging the client's, and the client acknowledges the server's. Both sides are then ESTABLISHED. Options such as MSS, window scale and SACK are negotiated in the first two segments.

What are the TCP connection states?

RFC 9293 lists LISTEN, SYN-SENT, SYN-RECEIVED, ESTABLISHED, FIN-WAIT-1, FIN-WAIT-2, CLOSE-WAIT, CLOSING, LAST-ACK, TIME-WAIT and CLOSED. CLOSED is fictional, meaning no connection state exists.

Why are so many sockets in TIME-WAIT?

The side that closes a connection first stays in TIME-WAIT for twice the Maximum Segment Lifetime, so a lost final ACK can be resent and stray segments from the old connection cannot confuse a new one. Busy clients and proxies naturally accumulate many. Reusing connections is the usual fix.

Should I enable SYN cookies?

On Linux they are already enabled by default (tcp_syncookies = 1) and only activate when a listener's SYN queue overflows. Keep that default. Do not force them always on, because they drop TCP extensions such as window scaling and SACK for affected connections.

What is the difference between FIN and RST?

FIN closes one direction gracefully after all data is delivered, and the other side can keep sending. RST aborts the whole connection immediately and discards its state. Applications are told which one happened.

TCP vs UDP: which is more secure?

Neither has built in security. TCP's handshake makes address spoofing harder to use for data exchange, while UDP's lack of a handshake makes reflection and amplification easier. Both need TLS, DTLS, QUIC or IPsec for real protection. See our eavesdropper comparison for the detail.

What is TCP Fast Open?

An extension (RFC 7413, Experimental) that lets a returning client send data inside its SYN using a server issued cookie, saving a round trip. Because SYN data can be replayed, it should only carry idempotent requests, and RFC 7413 says it must not be on by default.

How do I check open TCP ports on my server?

On Linux, ss -t -l -n -p. On Windows, Get-NetTCPConnection -State Listen or netstat -anob. On macOS, lsof -iTCP -sTCP:LISTEN -n -P. Only run these on systems you own or administer.

Where Safeguard fits

Safeguard does not inspect, capture or filter TCP traffic, and it does not configure kernels, firewalls or load balancers. Nothing in the hardening steps above depends on it. The honest connection is the software around TCP.

The protocols that make TCP safe to use, TLS, SSH, HTTP/2 and gRPC, live in open source libraries inside your applications and container images. Safeguard's software composition analysis, generally available, builds a full transitive dependency inventory of those libraries, and its SBOM generation and analysis, also generally available, records them in SPDX or CycloneDX. Vulnerability management with EPSS and KEV enrichment and reachability analysis, both generally available, tell you which of those library findings your code actually calls. Container scanning shows what is inside each image, though not the host kernel that runs the TCP stack, which you still patch through your distribution. For HTTP and API endpoints you own and verify, Safeguard's first party DAST, generally available, tests the running service. And policies and gates, generally available, can stop a build that pulls in a known vulnerable TLS or HTTP library. Our post on using SBOMs to find affected products fast shows the workflow when the next transport library advisory lands, and the HTTP/2 hardening guide covers the layer most TCP services expose.

Fix it without waiting for upstream

TCP stack bugs themselves are fixed by your operating system or firmware vendor. The libraries that sit on top of TCP in your own software, TLS, SSH, HTTP/2 and gRPC implementations among them, are a different story, and that is where you can act today.

Whatever the vulnerability, most of the risk sits in two gaps. The first is the time between a fix being published and that fix actually running in your production builds. The second, and worse, is the time before any fix exists at all, while a maintainer works out a patch and a release schedule. Waiting through both gaps is a habit, not a requirement. Safeguard gives teams three ways to close them.

1. Autonomous remediation: pull requests, not tickets

When a fixed version exists, Safeguard's Griffin models open a tested, policy gated pull request that moves the affected dependency to the safe version. You can run it on one repository or in bulk across every project that carries the vulnerable version. Remediation covers eight categories, including dependencies, licenses, security posture and findings. Reachability analysis ranks the vulnerable code paths your application actually calls first, so the first pull requests you merge are the ones that matter most. Autonomous remediation ships switched off, with every category set to Manual, so your team decides how much runs on its own and what still needs a human approval.

2. Gold Open Source: the safe version in seconds, free

gold.safeguard.sh is free and needs no login. Search any CVE or package to see affected and fixed versions, CVSS, EPSS, CISA KEV status and a recommended safe version, or use the "Am I affected?" checker. The same data is available through a free read API, README status badges and RSS threat feeds. The safeguard-sh/gold-scan-action GitHub Action can fail a build on a known exploited vulnerability, on any known vulnerability, or never, so a version you have just removed cannot quietly come back in next week's merge.

3. Gold Registry: start from components that are already fixed

registry.safeguard.sh provides hardened, drop in replacements for upstream packages and container images. Every Gold artifact is held continuously to zero exploitable critical or high CVEs, zero malware and a completed license check, with SLSA L3 or higher provenance, a Sigstore signature and a public evidence page you can hand to an auditor. An artifact loses its Gold badge if it regresses. Teams can point their own package directory at the registry so hardened equivalents resolve without changing how developers install anything.

When upstream has no fix yet, or will not ship one, Safeguard first works to get the fix accepted upstream. If the issue is severe or upstream will not move, Safeguard builds a fork, fixes the vulnerability there and publishes a hardened Gold build, so you are not stuck on a maintainer's release schedule. You can also request a hardened, compatibility tested build of a specific package. This fork and patch flow is in early access today, so ask which packages it covers before you plan around it.

Your situationWhat to use
A fixed version existsAutonomous remediation opens the upgrade pull request; Gold Open Source confirms the safe version
You need to know today whether you are affectedGold Open Source lookup and "Am I affected?" checker, free and without login
You want vulnerable versions to stop shippingThe Gold scan GitHub Action in CI, plus Safeguard policies and gates
Upstream has no fix yet, or will not ship oneA custom Gold request; fork and patch (early access)
You want to start clean on new projectsPull hardened components and images from Gold Registry

One honest limit: when the vulnerable code lives in a vendor appliance, a hosted service or an operating system runtime you do not build yourself, the fix still has to come from that vendor or distribution. The steps above apply to the open source dependencies and container images you build and ship, which is where most teams carry the largest and least tracked share of their exposure.

See how Safeguard would help

These demos show dependency, SBOM and remediation workflows, not network monitoring or kernel tuning. You can download Safeguard and follow the CLI docs to try them on your own projects.

This demo sets up the Safeguard runner on your own machine, approves it with its device code and then scans from the terminal. Pointed at the container image of a TCP service you run, the same scan lists the TLS, HTTP and gRPC libraries inside it with their versions and known vulnerabilities, so you know exactly what is handling your connections above the kernel.

Click through it at your own pace, or open it full screen.

This demo is a real CLI session: install, open the interactive terminal, ask it to fix the lodash CVE in a repository, and watch it read the finding and advisory, ask before each command, move lodash to 4.17.21, run npm install and npm test with approval, and print a summary without pushing anything. The same approve each step flow applies when the dependency to bump is a TLS or HTTP/2 library in a service that terminates TCP connections.

Click through it at your own pace, or open it full screen.

This demo connects Claude to the Safeguard MCP server with OAuth and lets you choose which tools Claude can use. Once connected, you can ask in plain language which of your projects ship a particular version of a networking or TLS library and get the answer from your own scan data.

Click through it at your own pace, or open it full screen.

Quick check

Your web server shows thousands of sockets in CLOSE-WAIT. Is that a SYN flood?

No. CLOSE-WAIT means the peer has already sent its FIN and TCP is waiting for your application to close the socket. A growing CLOSE-WAIT count almost always means application code is not closing connections. A SYN flood shows up as a large SYN-RECEIVED count and kernel warnings about SYN cookies.

Does turning on TLS protect a long lived TCP session from a forged reset?

No. TLS protects the payload but none of the TCP header, so a reset the stack accepts still ends the connection. RFC 5961's exact match rule makes blind resets much harder, but only TCP-AO or IPsec authenticate the TCP header itself, which is why router sessions such as BGP use TCP-AO.

Sources

Last reviewed: 23 October 2026. Kernel defaults change between releases; check the documentation for the kernel and Windows build you actually run.

Never miss an update

Weekly insights on software supply chain security, delivered to your inbox.

Self-healing security runs on Safeguard.

Your first fix PR is minutes away.

No sales call required, even your agent can complete the purchase over MCP.