How SYN cookies stop a SYN flood
For a networks and security project, my group analysed a packet capture of a TCP SYN flood attack on a file server. My part was the mitigations. Here’s the short version.
The attack. A TCP connection starts with a three-way handshake: the client sends SYN, the server replies SYN-ACK and waits, and the client sends ACK. While it waits, the server holds a slot in memory for that half-open connection.
A SYN flood sends huge numbers of SYNs, usually from spoofed addresses, and never finishes the handshake. The server’s queue of half-open connections fills up and real users can’t connect.
Mitigation 1: SYN cookies. Instead of storing state for every SYN, the server encodes the connection details into the sequence number of its SYN-ACK. If a real client replies with an ACK, the server can rebuild the connection from that number. A flood of SYNs no longer fills memory, because nothing is stored until the handshake completes.
- Where: Layer 4, on the server itself.
- Limitation: it protects the connection queue, not your bandwidth. A big enough flood can still saturate the link.
Mitigation 2: Ingress filtering (BCP38). Networks drop incoming packets whose source address couldn’t legitimately come from where they arrived. That makes spoofing much harder in the first place.
- Where: Layer 3, at network edges.
- Limitation: it only works if networks other than yours deploy it, so you depend on everyone else doing their part.
The two work well together: SYN cookies protect the server now, and ingress filtering reduces spoofed traffic across the internet over time.