Security Groups and the Stateful Connection Tracker

A security group is a list of rules, and the list has an unusual property: it contains no rule for return traffic. An outbound rule permitting a request to a destination port is sufficient for the reply to arrive, even though no inbound rule mentions that port. This is not a convenience feature or a default-allow. It is the visible consequence of a connection tracker sitting between the rules and the packets, and the tracker’s behaviour under stress determines what the group actually permits.

The rule set is one-directional by design

Each rule names a direction, a protocol, a port range, and a source or destination. An outbound rule evaluates the destination address and port of a packet leaving the interface; an inbound rule evaluates the source address and port of a packet arriving. Nothing else in the packet is matched: there is no payload inspection, no application-layer field, and no notion of a rule that applies only during business hours.

When a packet matches an outbound rule and leaves, the tracker records a flow entry. The entry is keyed on the five-tuple: source address, source port, destination address, destination port, and protocol. It is marked as established in the outbound direction. When a packet arrives with the reverse five-tuple, the tracker finds the entry and passes the packet without consulting the inbound rules at all.

The consequence is that the rule set does not describe reachability; it describes flow initiation. A group that permits outbound TCP to port 443 permits exactly one thing: the ability to start a connection to that port. Everything that follows from a connection started under that permission is allowed, including a reply of any size, on the matching reverse tuple.

The reverse is also true and is the source of a common misconfiguration. An inbound rule permitting TCP from a client on port 22 allows the client to open a connection, but the server’s replies are permitted by the tracker, not by the rule. If the outbound rules are restrictive and the tracker entry is absent, as after a restart, the replies are dropped and the connection stalls after the handshake. A group whose rules look symmetric can still fail asymmetrically, because the tracker is a separate mechanism with its own lifetime and capacity.

What the tracker keys on, and why ICMP is different

For TCP, the tracker follows the state machine loosely. A SYN creates an entry in a new state; the SYN-ACK and ACK move it to established; a FIN or RST begins teardown. The tracker does not need to be strict about the sequence numbers, but it does need to see the handshake, because a mid-stream packet with no entry is treated as a new flow and must match an inbound rule.

UDP has no handshake, so the tracker treats the first packet as the flow’s beginning and relies on idle timeout for the end. This creates an asymmetry worth naming: for UDP, any outbound datagram opens a return path for any inbound datagram on the reverse tuple, including one from an unrelated sender that happens to guess the port. The tracker cannot tell the difference because the protocol carries no state to tell it with.

ICMP is handled specially because its identifiers and types do not fit the port model. An echo request carries a type and an identifier rather than ports, and the reply carries a different type with the same identifier. The tracker maps the identifier to a pseudo-port so that replies are recognised, which is why ping works under a rule that permits outbound ICMP echo without any inbound rule. ICMP error messages are harder: a Destination Unreachable quotes the header of the packet that triggered it, so the tracker must parse the embedded header to associate the error with a flow. A firewall that does not perform that association drops path MTU discovery messages, and the symptom is a connection that works for small packets and hangs for large ones. Internet gateway, NAT, and source address translation covers the related case where the same quoted header must survive translation.

Timeouts are policy, not protocol

An entry does not persist forever. Each state has an idle timeout, and a flow that sends nothing for longer than the timeout has its entry removed. Once removed, the next packet on that tuple is a new flow and must match a rule, which for a mid-stream TCP segment it usually cannot.

The timeouts are set by the provider, not by the tenant, and they are shorter than the equivalent timeouts on a general-purpose host firewall. This produces a failure that is distinctive because it is silent: a long-lived idle connection, such as a database connection pool or an SSH session left open, is dropped by the tracker while both endpoints still believe the connection is established. The next write from either side is discarded, and the application sees a timeout or a reset that neither end can explain from its own logs.

/* Sketch of the lookup a tracker performs per packet. */
struct flow_key {
    uint32_t saddr, daddr;
    uint16_t sport, dport;
    uint8_t  proto;
};

/* Returns 1 if the packet belongs to an established flow. */
int tracker_established(struct tracker *t, const struct flow_key *k)
{
    struct flow_key rev = reverse_of(k);   /* arriving packet mirrors the flow */
    struct entry *e = hash_lookup(t->table, &rev);
    if (!e)
        return 0;
    if (now_ms() - e->last_seen > t->idle_timeout_ms[e->state]) {
        hash_remove(t->table, e);   /* expiry is lazy: on next reference */
        return 0;
    }
    e->last_seen = now_ms();
    return e->state == ESTABLISHED;
}

Two details in that sketch matter. The lookup uses the reversed tuple, because the arriving packet is the mirror of the one that created the entry. And expiry is lazy: entries are removed when they are next referenced, not by a scan, which means an idle entry occupies memory until something touches the tuple. The table therefore holds more than the number of active flows, and its capacity is consumed by flows that are already dead but not yet reclaimed.

Applications that must survive the timeout have two options, and only one of them is reliable. Sending periodic keepalive traffic on the connection resets the idle timer, which works as long as the interval is shorter than the timeout. Relying on the transport layer’s own keepalive is less reliable, because the default interval on most systems is measured in hours, far longer than any tracker timeout. Enabling TCP keepalive with a short interval, or sending an application-level heartbeat, is the standard remedy. A subnet-level filter, by contrast, has no timer to reset at all, which is one of the reasons it is used in front of workloads with long idle connections; see network ACLs and stateless filtering at the subnet boundary.

Capacity and the behaviour at the limit

The tracker has a fixed maximum number of entries per interface or per host, sized by the provider for what it considers a normal workload. When the table is full, new flows cannot be admitted. What happens then is the important part: packets for flows with no entry are dropped, and because they match no rule that would permit them, the drop looks exactly like a rule denial.

A host that opens a very large number of short-lived outbound connections can fill the table with entries that are not yet expired, and the symptom is a sudden refusal of new connections while existing ones continue. Because the drop is silent and looks like a policy problem, the usual first response is to inspect the rules, which are not the cause. A tracking table that is full is also a state exhaustion vector: an attacker who can open many flows from a permitted source can displace legitimate entries, which is why rules that permit a wide source range are worth narrowing even when the destination port is restricted.

The table size is not a documented number in most offerings, and the safe assumption is that it is finite and much smaller than the number of concurrent connections a large host could open.

Rules are evaluated as a set, with a default deny

Within a direction, rules are not ordered and are not evaluated first-match. They form a set, and a packet is permitted if it matches any rule in the set. The default when nothing matches is deny, and there is no way to express an explicit deny that takes priority over a permit, because there is nothing to prioritise: a packet that matches any permit is allowed.

That design removes a whole class of error. In an ordered firewall, a rule appended after a broader permit is dead, and diagnosing why a specific address is still allowed requires reading the whole list. In a set, adding a narrow permit cannot be negated by an earlier broad one, and the only question is whether the packet matches something.

What the set model cannot express is a negative rule: “permit the subnet except this one address”. The workaround is to permit the specific addresses that are wanted, which turns the rule set into an enumeration, and enumerations have size limits. A group with hundreds of rules is within the documented per-group limit in most offerings, but a group per interface multiplied by many interfaces reaches the per-interface attachment limit, and the failure at that boundary is a refused attachment rather than a slow evaluation.

Where the group sits relative to the interface

The group is attached to the network interface, and the evaluation happens in the virtual switch that owns that interface, before the packet is delivered. This means a packet between two instances in the same subnet is filtered twice: once on the sender’s outbound rules and once on the receiver’s inbound rules. There is no path that avoids both, because there is no shared medium.

It also means the group follows the interface rather than the instance. An address that is reassigned to a different interface takes the new interface’s group with it, and an instance that has several interfaces can have a different group on each. Traffic that arrives on one interface and leaves on another is subject to both groups, and the tracker entries are per-interface, so a flow that is redirected between interfaces mid-connection loses its state. This is the mechanism behind a common surprise in appliances that route between two interfaces: the reply to a connection that arrived on one interface leaves on the other, and the tracker on the receiving side has no entry for it, because the entry lives on the interface where the flow started.