Protocol Attacks: the SYN Flood in Depth

The retransmission behaviour worsens it: the server, expecting an ACK, retransmits each SYN/ACK several times before giving up, holding the entry for the full timeout, which is often many seconds. So each attacking SYN ties up an entry for a long time, and a modest rate of SYNs keeps the backlog permanently full.

The mitigation: SYN cookies

The elegant defence, and the reason it is worth teaching, is that it removes the resource the attack exhausts, so there is nothing left to fill.

The insight is that the server allocates state at SYN-RECEIVED before it knows the connection is real, which is exactly what the attacker abuses. SYN cookies invert this: the server allocates no state when it receives a SYN. Instead, it encodes the information it would have stored into the sequence number of the SYN/ACK it sends back, computed cryptographically from the connection's details. This encoded value is the "cookie".

Then:

• If the client is legitimate, it sends the final ACK, which (by the protocol) carries the cookie value back. The server reconstructs the connection state from the cookie, verifies it, and establishes the connection, having stored nothing in the meantime.
• If the SYN was part of a flood and no ACK ever comes, the server has spent no backlog entry, because it allocated none. There is nothing to fill.

So with SYN cookies, a flood of SYNs that are never completed costs the server almost nothing: it answers each with a computed SYN/ACK and forgets it, and only a genuine ACK causes any state to be created. The backlog can no longer be exhausted by half-open connections, because half-open connections no longer consume it.

SYN cookies have minor trade-offs (some rarely-used connection options cannot be carried in the cookie), so they are typically enabled to activate under attack, when the backlog is filling, rather than always. But they are a standard, widely available mitigation, and their existence is why the SYN flood, once devastating, is now a manageable and well-understood attack.

Other protocol attacks, briefly

The SYN flood is the archetype, and the same idea, exhausting a specific finite structure with little traffic, appears in other forms a student should recognise as the same category:

• Attacks that open connections and then keep them half-open at the application level, holding a request incomplete so the server keeps a worker waiting (a slow-request attack, which shades into the application-layer chapter).
• Attacks that exhaust other finite tables or state, such as connection-tracking tables in firewalls.

In each case the defence follows the SYN-cookie logic where possible: do not commit a scarce resource until the request is proven legitimate, and set timeouts and limits so that incomplete requests cannot accumulate.

A worked example, framed defensively

An assessor reviews a server's resilience to protocol denial of service, by inspecting configuration rather than by flooding it.

• The server's operating system has SYN cookies available but not enabled to activate under attack. Finding: the backlog could be exhausted by a SYN flood. Recommendation: enable SYN cookies, which removes the resource the attack targets.
• The network does not perform egress filtering, so spoofed SYNs could be sent from within, and inbound spoofed SYNs are not filtered. Finding: anti-spoofing would raise the difficulty of a SYN flood, since the flood relies on forged sources.
• The backlog size is at the default; Finding, minor: increasing it raises the bar slightly, but is not a real fix on its own, because an attacker can send more SYNs; SYN cookies are the actual answer.
• Upstream, the provider offers scrubbing that recognises SYN-flood patterns. Recorded as a positive control for large distributed floods that exceed what the host alone can absorb.

Recommendations, in order: enable SYN cookies (the definitive host mitigation); ensure anti-spoofing filtering; and rely on upstream scrubbing for distributed floods too large for the host. The assessor notes the pleasing point that the mitigation is not "more backlog" (an arms race the attacker wins) but "no backlog until the connection is real", which defeats the attack in principle rather than by degree.

What beginners get wrong

• Confusing a SYN flood with a volumetric attack. A SYN flood exhausts a specific structure, the backlog, with little traffic; a volumetric attack fills the connection with sheer volume. Different resource, different mitigation.
• Thinking a bigger backlog is the fix. It raises the bar slightly and the attacker simply sends more SYNs. SYN cookies remove the resource, which is the real fix.
• Missing why the source is spoofed. Spoofing hides the attacker and ensures no ACK returns, so each entry is held for the full timeout; anti-spoofing filtering therefore helps.
• Forgetting the retransmission cost. The server retransmits each unanswered SYN/ACK, holding the entry for many seconds, so a modest SYN rate keeps the backlog full.
• Not connecting it to the handshake. The attack is a direct consequence of the handshake's cost of remembering half-open connections, which the handshake chapter named.
• Believing SYN cookies have no trade-offs. They cannot carry some rarely-used options, so they are often enabled to activate under attack rather than always; still the standard mitigation.

Quick revision

• The handshake leaves the server half-open (SYN-RECEIVED) after SYN/ACK, holding a backlog entry until the final ACK or a timeout; that entry is the resource.
• A SYN flood sends many SYNs and never the final ACK, filling the backlog so legitimate clients cannot connect. A protocol attack: little bandwidth, usually spoofed sources so no ACK returns, worsened by SYN/ACK retransmission holding entries for the full timeout.
• SYN cookies: allocate no state on SYN; encode the needed state into the SYN/ACK sequence number (the cookie); reconstruct it only when a valid ACK returns. A flood then costs almost nothing because half-open connections consume no backlog. Standard, often enabled to activate under attack.
• The same "do not commit a scarce resource until the request is proven" logic answers other protocol attacks. Anti-spoofing filtering and upstream scrubbing support the defence.

Test yourself

1. What resource does a SYN flood exhaust, and how does the attack exhaust it?

It exhausts the server's finite backlog of half-open connections. After the server replies to a SYN with SYN/ACK it holds a backlog entry awaiting the final ACK; a SYN flood sends many SYNs and never sends the ACK, so the server allocates an entry for each and holds it until timeout, filling the backlog so that legitimate clients' connection requests are dropped for lack of room.

1. Why is a SYN flood classed as a protocol attack rather than a volumetric one?

Because it exhausts a specific finite structure, the connection backlog, using very little bandwidth, rather than saturating the target's network connection with sheer volume. Only enough small SYN packets to fill the backlog are needed, so the attack is efficient and its effect depends on the size of a data structure, not on out-flooding the pipe.

1. How do SYN cookies mitigate the attack?

They cause the server to allocate no state when it receives a SYN, instead encoding the information it would have stored into the sequence number of the SYN/ACK it returns, computed cryptographically from the connection details. The state is reconstructed and the connection established only when a legitimate final ACK returns carrying that value, so a flood of SYNs that are never completed consumes no backlog entries and the backlog can no longer be exhausted.

1. Why does the attacker usually spoof the source addresses of the SYN packets?

To hide their identity and, more importantly, to ensure that the server's SYN/ACK is sent to an address that will not respond, so no final ACK ever arrives and each backlog entry is held for the full timeout. Because the attack relies on forged sources, anti-spoofing ingress and egress filtering raises its difficulty.

1. Why is increasing the backlog size not a real solution?

Because it only raises the threshold slightly, and the attacker can simply send more SYN packets to fill the larger backlog, making it an arms race the attacker wins. SYN cookies solve the problem in principle by removing the resource the attack targets, allocating no backlog state until a connection is proven legitimate, so there is nothing left to exhaust.