Security Engineering · Network Security
Flow Records and Traffic Baselining
How routers summarise traffic into flow records, what the cache timeouts decide about those records, and which questions the resulting telemetry can and cannot answer.
A sensor that inspects payload needs to be positioned on a choke point and sized for the traffic it sees. Flow export takes the opposite approach: the device forwards packets normally and, as a side effect, emits a compact record for each conversation it observed. The record carries the endpoints, ports, protocol, packet and byte counts, and timestamps, and no payload at all. That omission is the whole design. It makes the telemetry cheap enough to run everywhere, and it fixes the set of questions it can answer.
What a flow is, and how one ends
A flow is a set of packets sharing a key. The key is usually the five-tuple of source address, destination address, source port, destination port, and protocol, plus interface or direction identifiers at some exporters. Packets that share the key belong to the same flow, and the record is a summary of them.
A flow ends in one of three ways. It is removed when the cache is flushed whole; it is removed when the key is needed for a different flow and the cache is full; or it times out. Timeouts come in two flavours. The active timeout forces a record out for a flow that is still running, so long-lived conversations appear as a series of records rather than one enormous one. The inactive timeout ends a flow that has stopped carrying packets. Implementations configure these per protocol: TCP flows tend to get longer windows than UDP, on the reasoning that TCP has explicit teardown, and the differences show up in the data as a floor on how short a recorded conversation can be.
The consequence is that a flow record does not correspond to a connection. A TCP connection that lasts an hour and transfers nothing for two minutes in the middle may appear as several records if the inactive timeout is short. Two connections between the same endpoints on the same ports at different times are indistinguishable unless the records’ timestamps separate them. Any analysis that treats one record as one session is wrong whenever either timeout has fired, and the exporter’s configuration is not visible in the data unless the collector records it.
Records are defined by templates
IPFIX and NetFlow version 9 both solve the variable-content problem the same way: the exporter sends templates that declare the ordered list of fields in subsequent data records, and the collector matches data against the template it last received. Data records contain values only, so the wire format is compact even when the field set is wide. This is why a collector that restarts and has not yet seen the template cannot decode the data that follows, and why templates are re-sent periodically.
IPFIX replaced the fixed field numbering of earlier NetFlow versions with an information model, so a field is identified by a combination of element identifier and enterprise number. The information model is defined separately from the protocol, which allows a vendor to add fields without changing the protocol; RFC 7012 is that model. The protocol message header carries an export time, a sequence number, and an observation domain identifier. The sequence number counts data records, not messages, so a collector can detect loss, because a gap in the sequence means records were dropped, though not which ones.
Transport is a real design decision. Plain UDP export is cheap and loses records silently under congestion, which a collector can detect only by counting the gap. SCTP with partial reliability gives the exporter a mechanism to abandon records that cannot be delivered in time rather than blocking, and TCP gives reliable, head-of-line-blocking delivery at the cost of backpressure in the router. RFC 7011 specifies all three, and the choice is usually made by what the exporter platform supports rather than by analysis.
Sampling and the error it introduces
At line rate, exporting a record for every flow is not always feasible, so exporters sample. The standard model is to select one packet in every n, which is systematic sampling, or to select packets with probability p, and to report the sampling parameters to the collector so it can estimate real volumes. RFC 5476 specifies the protocol for communicating packet sampling configuration and reports.
The statistical consequence is straightforward and often ignored. A scan that touches ten thousand hosts, one packet each, is a very large flow count, not a large byte count; sampling preserves flow counts much better than it preserves evidence of any individual flow, because a one-packet flow is either selected or it is not. Conversely, a bulk transfer’s byte count is estimated from a sample, so its precision depends on the sample size. Sampling is therefore acceptable for volume baselines and unreliable for low-volume reconnaissance, which is precisely the traffic most worth detecting.
Baselining with flow data
The productive use of flow telemetry is not matching against known bad hosts. It is comparing current behaviour against a learned normal, per host or per group.
Four measurements do most of the work. Fan-out is the number of distinct destinations a source contacts in a window; a host that suddenly contacts hundreds of peers in one segment is either scanning or has been repurposed. Fan-in is the same from the other side and catches hosts being probed. Byte volume per peer distinguishes a bulk copy from interactive traffic, since the ratios differ by orders of magnitude. Port diversity catches a host using protocols it never used before, which is the signature of both lateral movement and legitimate software changes.
Fan-out on an internal interface is the measurement that finds lateral movement, and it depends on the exporter seeing that interface at all. Traffic between two workloads on the same host never appears in a flow record, which is the visibility gap that Microsegmentation and Policy at the Workload Boundary addresses by moving both enforcement and observation into the host.
The baseline itself has to tolerate normal variation. Diurnal traffic patterns mean a fixed threshold produces alerts every morning and none overnight. A model that tracks a per-host distribution and flags deviation from it is more sensitive, at the cost of needing enough history per host to have a distribution at all, and a host that has been compromised long enough contributes to its own baseline. That contamination is the fundamental limit of behavioural baselining and does not have a clean solution.
Long-lived and periodic flows deserve separate treatment. A pair that exchanges packets at a regular interval with roughly constant small volumes is either a keepalive, a monitoring probe, or a command channel; the three are not distinguishable from flow records, which is why this pattern generates investigation rather than an automatic verdict. The quantitative version of that analysis is described in Detecting Periodic Beacons in Flow Telemetry for the case where the interval structure is the evidence.
The cost of collecting it everywhere
Aggregation is what makes flow data manageable and also what destroys detail. A collector that merges records per host pair before storing them loses the per-port view and the temporal ordering; one that merges per port over a day loses the burst structure. Storage grows as the product of distinct keys and events, and distinct keys grow with the environment: a large flat network generates a very large number of short flows, and a busy host talking to many peers generates more keys than a busy server talked to by many.
Retention is the second pressure. The value of a baseline is that it spans enough history to distinguish normal from abnormal, and the value of raw records is that they support retrospective questions about a period before anyone knew to ask. Storing aggregated summaries for months and raw records for days is the usual compromise, and its adequacy depends entirely on how quickly an incident is noticed.
Finally, the export channel is itself traffic in the path. The exporter sends records to the collector over the same network that carries production traffic, from a device that also forwards that traffic. The confidentiality of those records is not guaranteed by the protocol: IPFIX over UDP has no cryptographic protection of its own, so an environment that treats the records as sensitive has to protect them in transit or accept that a party on the path can read the map of who talks to whom. Flow data reveals topology, service usage, and timing, which is enough to plan an attack even without any payload.