How a Stateful Firewall Matches a Packet

A stateless filter compares each packet against an ordered list and forwards or drops it. That is a pure function of the packet header, and it is easy to reason about. A stateful firewall is not that: it keeps a table of flows, consults the table before the rule set, and lets the table’s contents decide most packets. The rule set then describes how flows are created, not how individual packets are judged. Most surprising firewall behaviour follows from that split.

The five-tuple is the unit of decision

Both the rule set and the state table are keyed on a five-tuple: source address, destination address, source port, destination port, and transport protocol. The port fields are only meaningful for TCP, UDP, SCTP, and a handful of other protocols; ICMP has no ports, so implementations key ICMP flows on the type, code, and identifier fields instead, which is why an ICMP echo request and its reply form a state entry just as a TCP connection does.

A packet that carries no transport header at all still has to be classified. IPv4 protocol 1 is ICMP, 6 is TCP, 17 is UDP, 47 is GRE, 50 is ESP, 51 is AH. Protocol 58 is ICMPv6. A firewall that only knows how to build state for TCP and UDP must decide what to do with the rest, and the usual answer is to treat them as stateless traffic matched only against explicit rules. This is why IPsec and GRE tunnels so often need a permissive rule covering both directions while ordinary TCP services need only one.

Because the tuple is symmetric, the reply direction is not a separate rule. A connection-tracking entry records the original tuple and the reply tuple, and the reply tuple is what allows the return traffic. That is the single largest semantic difference from a stateless access list, which must describe both directions explicitly.

Rules are a first-match program

Rule sets are evaluated top to bottom, and the first rule whose predicates all match produces the verdict. Most implementations therefore require the rule set to be ordered from most specific to least, with a default policy at the end. In nftables the default policy is attached to the chain rather than being a terminal rule:

table inet filter {
    chain input {
        type filter hook input priority filter; policy drop;
        ct state established,related accept
        ct state invalid drop
        iif "lo" accept
        ip protocol icmp accept
        tcp dport { 22, 443 } accept
    }
}

Two details in that snippet carry most of the semantics. The first rule is a state test, not a five-tuple test: it accepts anything the tracking table considers part of an existing flow, before any port is examined. The second detail is the chain type. A chain of type filter attached to the input hook sees packets addressed to the host itself, which is a different set from packets being forwarded; routers enforce policy in a forward chain. Getting the hook wrong produces a rule set that looks correct and filters the wrong traffic.

Set-valued matches such as tcp dport { 22, 443 } are not syntactic sugar in the kernel. They compile to a lookup against a set object, which is typically a hash table or a red-black tree, so a rule with a thousand destination ports costs a lookup rather than a thousand comparisons. The performance of a rule set therefore depends on how many distinct predicates appear, not on how many literal values.

What the state table adds

The tracking table stores, per flow, the tuple in both directions, the current TCP state, a timeout, and counters. The timeouts are protocol-specific and are the source of most operational surprises. A TCP flow that has completed the handshake but never carried data sits in a short-lived state; a flow that has seen data in both directions gets a longer idle timeout. UDP has no handshake at all, so its entries are purely timeout-driven, and a reply seen within the window is what refreshes them.

Consider a UDP service that answers from a different source port than the one addressed. The reply arrives with a tuple that does not match the expected reply tuple. A strict tracker drops it as invalid. Implementations vary here, and the variation is exactly what the NAT behavioural documents catalogue: whether an incoming packet matching the original direction of an existing mapping but arriving from outside is accepted, whether ICMP errors are related to the flows they quote, and how long an unreplied UDP mapping survives. Any firewall behaviour that seems inconsistent across vendors usually reduces to one of these policy choices rather than a bug.

The state test also runs before the rule set, so a rule cannot be used to deny traffic that the table already accepted unless the deny rule is placed earlier in the chain than the state accept. This is the mechanism behind the common pattern of putting a blocklist ahead of the established rule. Placing it after is a no-op, and the rule set reads as if it blocks traffic it never sees.

Asymmetric routing, failover, and the table’s blind spots

Tracking assumes both directions traverse the same device. When they do not, as with a multi-homed host, a router pair with per-packet load balancing, or a policy-routed return path, the device that sees only one direction marks the flow incomplete and drops the returning packets. Some deployments handle this by disabling the state check for the affected prefixes, which converts the firewall back into a stateless filter for exactly the traffic that needed state most.

High availability adds a second failure mode. If the state table is not synchronised to the standby node, a failover leaves every established flow unknown to the new active device, and the rule set’s default policy silently kills long-lived sessions. Synchronising the table fixes that, but the synchronisation channel itself becomes security-relevant: it carries the full flow state, and an attacker who can inject entries there can open holes.

Fragmentation is the third blind spot. A non-initial fragment carries no transport header, so a rule that matches on port numbers cannot classify it. RFC 1858 describes the resulting attacks, in which tiny fragments are used to slip past filters that only inspect the first fragment, and the standard mitigations are to drop fragments outright, to require a minimum fragment size, or to reassemble before filtering. Reassembly before filtering is the only approach that is correct for all cases, and it is also the one that turns the firewall into a target for memory exhaustion, because it must buffer incomplete datagrams.

Where the model ends

A stateful filter sees headers and, optionally, a small amount of payload for protocol helpers. It does not see whether a request on port 443 is an HTTP GET for an allowed path or a tunnel carrying something else. Protocols that negotiate their data channel out of band, such as FTP’s data connection, SIP’s media streams, and some VoIP signalling, need a helper that parses the control channel and opens the expected data flow in advance. Those helpers are themselves parsers exposed to hostile input, and they historically produced both bypasses and memory-safety bugs.

Raising the inspection point above the transport layer removes the ambiguity but changes the trust model, since the proxy must terminate and re-originate the connection. That trade is the subject of Application Proxy Firewalls and Parser Differentials. Below the network layer, the isolation question is different again: a filter enforces policy between addresses, while VLAN Segmentation and the Limits of Layer 2 Isolation enforces it between broadcast domains. And when the address rewriting that accompanies most edge deployments is in play, the tuple the rule set sees is not the tuple the endpoints use, as Connection Tracking Under NAT describes.

The practical consequence is that a firewall policy is a statement about flow initiation. Its enforcement of anything after initiation is limited to timeouts, state validity, and whatever a helper or proxy chooses to parse. Reading a rule set as if it described per-packet permissions overstates what the device does.