Near-Real-Time Refresh and the Commit Boundary

A classic batch index makes documents searchable only after a full rebuild, which can lag the source data by hours or days. Near-real-time search closes that gap to about a second, and it does so without sacrificing durability, by carefully separating three things that a naive design conflates: making data visible to queries, making it recoverable after a crash, and making it permanent. The boundaries between those three states are the commit boundary, and understanding it is the key to reasoning about any search system’s consistency.

Three separate concerns

The first concern is visibility. A document becomes searchable when the query layer can see it, which happens when the in-memory buffer that holds it is flushed to a segment and a new searcher is opened over the updated segment set. The second is durability. A document is durable when it has been written to the filesystem’s write cache and, strictly, when that cache has been synced to the physical device with an fsync. The third is permanence, the point at which the document is part of an immutable on-disk segment rather than a replayable log record.

These three can be satisfied at different times, and the art of near-real-time refresh is choosing how much work to do per second of latency. Making data visible costs a flush plus a searcher switch, both cheap. Making data durable costs an fsync, which is expensive because it forces the device to flush its volatile cache, often tens of milliseconds on a rotating disk. Making data permanent costs a merge or a full flush, the most expensive of all.

The reason these three are decoupled at all is that they have different consumers. Visibility serves the reader, who wants the newest document now. Durability serves the operator, who wants not to lose accepted documents on a crash. Permanence serves the storage layer, which wants the log shrunk and the segments immutable. A design that bundled all three into a single operation would be forced to pay the sum of their costs on every document, which is exactly the batch-index behaviour near-real-time search exists to escape.

The refresh interval

A refresh is the operation that moves documents from the in-memory buffer into a searchable segment. Systems run it on a timer, commonly once per second by default, so a document added just after a refresh becomes visible at the next one, roughly a second later. The refresh interval is the single tunable that trades indexing throughput against latency: a short interval makes documents visible sooner but flushes many tiny segments, which the merge operation then has to combine, while a long interval batches more documents per segment and amortizes fixed costs better.

The interval is a statistical guarantee, not a wall-clock one. A document’s visibility latency is the time until the next refresh, which is uniform between zero and the interval, so the average latency is half the interval and the worst case is the full interval. Halving the interval halves the worst case, but it doubles the number of segments flushed per unit time and therefore doubles the merge load. The one-second default is an engineering choice sitting at the knee of that curve, fast enough to feel interactive and slow enough that the merge and the segment count stay manageable.

Refresh does not guarantee durability. A document flushed to a segment but not yet synced to disk can be lost if the machine loses power, because the segment exists only in the operating system’s page cache. That is an acceptable risk for many workloads, and it is exactly the trade the system makes: the cost of an fsync on every refresh would make the one-second interval impossible on mechanical disks.

The commit and the translog

Durability is handled by a separate, older mechanism: a write-ahead log, called a translog in several systems. Every incoming document is appended to the translog first, before it is indexed, and the translog is periodically synced. Because a log append is a sequential write, syncing it is cheap compared to syncing a random collection of segment files. If the machine crashes, the system replays the translog to reconstruct the documents that were in the buffer but not yet safely on disk.

A commit, or full flush, is the operation that syncs the segments to disk and then truncates the translog, because at that point the translog is no longer needed for recovery. Commits are expensive, so they run infrequently; the translog is what carries the durability burden in between. The failure mode this pairing prevents is the silent window where a document is searchable (it was refreshed) but not durable (it was not yet committed): the translog covers that window, so the system can honestly report that it indexed the document while knowing it can still recover it.

The translog and the refresh are independent timers, and their independence is the source of most confusion about this mechanism. The refresh timer controls visibility; the translog sync controls durability; the commit controls permanence. A document can be visible and durable but not permanent, if it has been refreshed and its translog synced but not yet committed. It can be durable but not visible, in the brief interval after a translog sync but before the next refresh. And it can be visible but not durable, after a refresh but before the next translog sync, which is the window where a power loss drops it. None of these three timers implies the others, and a correct mental model of near-real-time search keeps them strictly separate. The full account of the log and its recovery is the subject of write-ahead logging.

The searcher switch

Every refresh ends with a searcher switch, the atomic swap of the query layer’s view from the old segment set to the new one. Because segments are immutable, the old searcher remains valid for in-flight queries; the switch is a change to a pointer, not to any file a reader holds open. A query that started before the refresh finishes against the old view, and a query that starts after finishes against the new view. This copy-on-write publication is why concurrent readers can run without locks while refreshes happen underneath them.

The switch has a cost that scales with the number of open segments and the size of the term dictionary, because the new searcher must be warmed: its files opened, its dictionary mapped, and its hottest blocks pulled into cache. Warming is the difference between a refresh that adds a segment invisibly and one that causes a latency spike as the next queries fault in the new segment’s pages.

Edge cases

The commit boundary produces several named edge cases. The “empty refresh” is a refresh that runs when the buffer is empty; it should be a no-op, but a naive implementation still opens a new empty segment and pays the switch cost. The “uncommitted but searchable” state is the one that surprises operators who copy an index directory: a directory copied without the translog contains the segments but not the in-buffer documents, so recovery from a copied directory replays nothing and silently loses the tail. And the interval is a ceiling, not a guarantee: a system under heavy merge load can delay a scheduled refresh, stretching the one-second promise outward under exactly the conditions where latency already hurts.

Near-real-time refresh is the mechanism that turned search from a batch operation into an interactive one, and its entire design is an exercise in decoupling visibility from durability so that each can be paid for on its own schedule.