The Three-Way Handshake, and Why Scanning Works

Why the SYN and the ACK can share packet 2 is worth stating: they are independent bits in the same header, so the server acknowledges and initiates simultaneously. That is why the exchange is three packets and not four.

The other outcome: RST

If nothing is listening on the port, the standard requires the host to reject the attempt, and it does so by replying with RST (usually RST with ACK). It says: there is nothing here, do not continue.

And that is the whole basis of port scanning:

Two different, unambiguous, mandatory responses. The target is not choosing to inform the scanner; the specification requires it to answer this way so that ordinary connections work. A host that stopped answering would break every legitimate client.

Why that is enough

Everything else follows from those two lines.

To find open ports, send a SYN to each port and record which reply with SYN/ACK. That is a SYN scan, and it is the reason port scanning is fast and reliable rather than a matter of guesswork.

To avoid completing connections, send the SYN, read the reply, and if it is SYN/ACK send RST instead of the final ACK. The connection never reaches ESTABLISHED, so the application on the target may never be told a connection occurred, and historically it was not logged by the service. This is the half-open or SYN scan, and its "stealth" is exactly this and nothing more.

To fingerprint through firewalls, note that a third response exists that the table does not list: no reply at all. That cannot happen in a correctly functioning direct conversation, because the standard requires SYN/ACK or RST. So silence means something in between interfered, which is what "filtered" means and what the firewall-detection chapter reads.

To scan without opening connections at all, exploit a different rule. The specification also says what a host must do with a segment that arrives for a connection that does not exist: a closed port must reply RST, and an open port must ignore it. That single asymmetry is what FIN, NULL and XMAS scans use, and it is why those scans can probe without ever sending a SYN.

The examinable point: a scanner is not breaking anything. It is asking questions the standard obliges the target to answer, or sending packets the standard specifies a response to. Port scanning is a consequence of TCP being well specified.

The half-open state and what it costs the target

Dwell on packet 2 for a moment, because it matters twice more in this book.

When the server sends SYN/ACK it must remember the half-formed connection: the client's address and port, the sequence numbers, the options negotiated. It holds that in a finite structure, often called the backlog, and waits for the third packet. If the ACK never arrives, the entry sits there until a timeout expires, and the server may retransmit the SYN/ACK several times first.

Two consequences:

• A SYN scan leaves the target holding half-open entries until they time out. A scan of many ports is therefore not free for the target, even though the scanner has done nothing wrong.
• A deliberate flood of SYNs that are never completed fills the backlog, so genuine clients cannot connect. That is the SYN flood, the protocol denial-of-service attack of Module 2, and it is the handshake's cost turned into a weapon. The defence, SYN cookies, works by refusing to allocate the state at all until a valid ACK proves the client is real.

Closing a connection, briefly

For completeness, since FIN appears in the scan types. An orderly close is four packets, because each direction closes independently: one side sends FIN, the other ACKs it, then the other side sends its own FIN, which is ACKed. Either side may also abort at any time with RST, which is immediate and unacknowledged.

The point that matters for scanning: FIN is meaningful only within an existing connection. Sending a bare FIN to a port with no connection is nonsense, which is precisely why the standard's rule for nonsense packets is what the FIN scan exploits.

A worked example

A tester, with authorisation, probes three ports on one host and reads the exchanges.

Port 443. Sends SYN. Receives SYN/ACK. Concludes open, and immediately sends RST to tear down the half-open connection rather than completing it. The target briefly held a half-open entry and its web service was probably never informed.

Port 25. Sends SYN. Receives RST/ACK. Concludes closed: the host is alive and reachable, and no mail service is listening. Note that this is still useful, because it proves the host exists.

Port 3306. Sends SYN. Receives nothing, twice, after retransmission. Concludes filtered: since the standard requires one of the two answers, silence means an intermediate device dropped the packet. Whether a database is behind it is unknown.

Three ports, three distinct conclusions, all derived from two lines of the specification. And one further observation for the report: the mixture is itself informative. A host that returns RST on some ports and silence on others has a firewall with per-port rules; a host that returns silence on everything except two ports has a default-deny firewall, which is the reasoning the firewall chapter formalises.

What beginners get wrong

• Thinking the handshake is four packets. It is three, because SYN and ACK are independent flags that can be set in the same segment.
• Believing a half-open scan is undetectable. It avoids completing the connection, so the application may not log it, but the pattern of many SYNs without ACKs is exactly what a network sensor looks for.
• Forgetting that a closed port is informative. RST proves the host is alive, which silence does not.
• Treating no reply as "closed". Closed has a specific signal, RST. Silence means filtered, and the state is unknown.
• Thinking scanning exploits a flaw. It uses required behaviour; a host that refused to answer would break normal connectivity.
• Missing that half-open connections cost the target resources. That cost is what the SYN flood weaponises.

Quick revision

• Three packets: SYN (client, sequence x), SYN/ACK (server, acknowledges x + 1, sends its own y), ACK (client, acknowledges y + 1). States: SYN-SENT, SYN-RECEIVED (half-open), ESTABLISHED.
• The handshake confirms willingness, exchanges initial sequence numbers, and proves the return path works.
• The basis of scanning: a SYN to an open port must be answered SYN/ACK; to a closed port, RST. Both are mandatory, so the target cannot decline to inform the scanner.
• A third outcome, silence, is impossible in a direct exchange and therefore means an intermediate device dropped the packet: filtered.
• A second rule, that a segment for a non-existent connection gets RST from a closed port and is ignored by an open one, is what FIN, NULL and XMAS scans use.
• The server allocates state at SYN-RECEIVED; scanning leaves half-open entries, and flooding them deliberately is the SYN flood, answered by SYN cookies.
• Closing is four packets (FIN and ACK each way), or an immediate RST. FIN is meaningful only inside an existing connection.

Test yourself

1. Describe the three-way handshake, naming the flags and what each packet accomplishes.

The client sends a segment with SYN set and its initial sequence number; the server, if a service is listening, replies with SYN and ACK together, acknowledging the client's number and supplying its own; the client replies with ACK acknowledging the server's number, after which the connection is established. It confirms mutual willingness, exchanges initial sequence numbers and proves the return path works.

1. Why is port scanning possible at all, and why is it not an exploitation of a flaw?

Because the TCP specification requires a host to answer a SYN with SYN/ACK when a service is listening and with RST when none is, so the response distinguishes open from closed. This is required behaviour that makes ordinary connections work; a host that refused to answer would be unable to serve legitimate clients.

1. What does a complete absence of response indicate, and why can it not occur in a direct exchange?

It indicates the port is filtered, meaning an intermediate device such as a firewall silently discarded the packet, so the port's real state is unknown. It cannot occur directly because the standard requires either SYN/ACK or RST, so silence proves something intervened.

1. What is the half-open state, what does it cost the target, and which attack weaponises it?

After sending SYN/ACK the server has allocated connection state and is waiting for the final ACK, which is the half-open or SYN-RECEIVED state. It consumes an entry in a finite backlog until it times out. Deliberately flooding a target with SYNs that are never completed fills that backlog so genuine clients cannot connect, which is the SYN flood.

1. Why can a bare FIN sent to a port be used for scanning?

Because FIN is meaningful only within an existing connection, so a bare FIN refers to a connection that does not exist, and the specification dictates the response to such a segment: a closed port replies RST while an open port ignores it. That asymmetry lets a scanner infer state without ever sending a SYN.