Topic domain
Inference Systems
Serving trained models under latency, cost, and accuracy constraints.
- Articles
- 80
- Clusters
- 8
- Foundational
- 24
- Intermediate
- 32
- Advanced
- 24
Batching, routing, and model placement.
- Iteration-Level Batching: Reforming the Batch at Every Token
- Paged KV Cache Allocation and the Block Table
- Chunked Prefill and the Interference It Removes
- Least-Outstanding Routing When Request Costs Vary
- Prefix-Cache-Aware Routing and Cache Locality
- Prefill and Decode Disaggregation
Quantization
10Precision reduction and calibration.
- Affine Quantization: Scale, Zero Point, and the Error They Introduce
- Per-Channel and Group Scales: Granularity Against Metadata Cost
- Range Calibration: Trading Clipping Error Against Rounding Error
- Weight-Only Quantization and the Memory-Bandwidth Bound
- Activation Outlier Channels and Mixed-Precision Decomposition
- Requantization Inside the Kernel: From Int32 Accumulator to Int8 Output
Fusion, scheduling, and kernels.
- Computational Graphs as an Intermediate Representation
- Constant Folding and Algebraic Rewrites on a Fixed Graph
- Operator Fusion and the Conditions That Block It
- Static Memory Planning from Tensor Liveness Intervals
- Layout Assignment and the Elimination of Transposes
- Dynamic Shapes, Guards, and the Cost of Recompilation
Memory residency and utilization.
- Iteration-Level Batching and the Cost of a Fixed Batch
- How Key-Value Cache Bytes Set the Batch Ceiling
- Splitting Prefill From Decode Inside a Step
- Paged Key-Value Residency and Block Tables
- Admission Control and the Memory Watermark
- Preemption by Swap Versus Recomputation
Indexing, chunking, and grounding.
- Why a Larger Context Window Does Not Remove the Need for an Index
- Chunk Boundaries Decide What Can Be Retrieved
- Approximate Nearest Neighbour Indexes and the Recall They Trade Away
- Similarity Metrics and the Normalisation They Require
- Hybrid Lexical and Dense Retrieval: Fusing Two Ranked Lists
- Cross-Encoder Reranking and the Cost of Reading Every Pair
Quality measurement for generated text.
- Clipped N-Gram Precision and the Brevity Penalty
- Recall-Oriented Summary Scoring with ROUGE
- Why Perplexity Is Not Comparable Across Tokenizers
- Answer Normalization and the Fragility of Exact Match
- Position Bias in Pairwise Preference Judging
- Confidence Intervals for Accuracy on Small Evaluation Sets
Cost per token and capacity planning.
- Prefill and Decode: The Two Cost Curves Behind One Token Price
- The KV Cache Memory Budget and What It Costs per Token
- Batch Size, Occupancy, and the Marginal Cost of a Token
- Sizing a Replica Pool from Arrival Rate and Latency Target
- Prompt Cache Economics: Hit Rate, Prefix Length, and Break-Even
- Speculative Decoding Cost Accounting: The Acceptance Rate Threshold
Deployment, monitoring, and rollback.
- Model Artifact Identity: Why a Checkpoint Hash Is Not a Deployment
- Shadow Traffic: Duplicating Requests to a Candidate Without Serving Them
- Canary Promotion as a Repeated Statistical Test
- Rolling Back a Stateful Serving Fleet Under Live Traffic
- Point-in-Time Feature Reads and the Skew They Prevent
- Where Latency Is Measured, and Why Percentiles Do Not Average
All Inference Systems articles
- Oct 2, 2026 Inter-Operator Parallelism and Stream Scheduling Independent branches of a graph can run concurrently on separate queues, but overlap only helps when the branches use different resources and the dependencies are exact.
- Sep 29, 2026 Weight-Only Quantization and the Memory-Bandwidth Bound Why 4-bit weights speed up token generation while leaving prefill almost untouched, how the dequantize-on-load kernel works, and where the arithmetic intensity ceiling sits.
- Sep 20, 2026 Adapter Residency When One Replica Serves Many Fine-Tunes Low-rank adapters let dozens of fine-tunes share a base model, but each token still needs one adapter applied. Residency, batch grouping, and routing decide whether that sharing pays.
- Aug 21, 2026 Similarity Metrics and the Normalisation They Require Why cosine, inner product, and Euclidean distance rank differently, what L2 normalisation changes about a vector index, and which failures come from mixing conventions.
- Aug 8, 2026 Iteration-Level Batching: Reforming the Batch at Every Token How a serving loop rebuilds its batch before every forward pass, why request-level batching wastes half the machine, and which budget actually limits concurrency.
- Jul 21, 2026 Autoscaling Signals for Token Servers CPU utilization and request rate both mislead when capacity is pinned by KV cache and iteration time. Queue depth in tokens, cache occupancy, and time-to-first-token measure what matters.
- Jun 24, 2026 Windowed Drift Detection and the Arithmetic of Its Alarms Comparing a live feature distribution against a reference is a two-sample test on a window, and the window length, the binning, and the number of features being tested together determine how often the alarm fires with nothing wrong.
- Jun 22, 2026 Prompt Cache Economics: Hit Rate, Prefix Length, and Break-Even What a cache hit actually saves, how prefix granularity bounds the achievable hit rate, and where cache storage competes with concurrency for the same memory.
- Jun 21, 2026 Affine Quantization: Scale, Zero Point, and the Error They Introduce How a real-valued tensor is mapped onto a small integer grid, why the zero point exists, and where the rounding error lands inside a dot product.
- Jun 18, 2026 Iteration-Level Batching and the Cost of a Fixed Batch Why a batch fixed at admission time strands accelerator capacity, and how re-deciding the batch at every decode step changes the cost of a token.
- Jun 16, 2026 KV Cache Quantization and Per-Token Scale Placement Why the key and value caches need different quantization axes, how per-token scales interact with the attention kernel, and what happens to the softmax when keys lose precision.
- Jun 15, 2026 Span Attribution: Making Generated Claims Point at Real Text How offsets survive chunking, encoding, and decoding; what a citation can and cannot prove; and why quote verification is a string operation rather than a judgement.
- May 30, 2026 Deadline-Aware Batching and Tenant Fairness When a batch is full, admission becomes an arbitration problem. Deadline slack, tenant shares, and starvation bounds each answer a different question about who gets the next slot.
- May 20, 2026 Dynamic Shapes, Guards, and the Cost of Recompilation A compiled kernel needs concrete extents; symbolic shapes force either a general kernel, a bucket, or a recompile, and each choice moves cost to a different place.
- May 14, 2026 Keeping a Vector Index Current Without Rebuilding It How segments, tombstones, and compaction let an index absorb edits, why deletion is more expensive than insertion, and where the visible-lag window comes from.
- Apr 22, 2026 Chunked Prefill and the Interference It Removes Prefill is compute-bound and decode is bandwidth-bound, so running them in the same iteration makes each wait on the other. Splitting prompts into chunks is the standard remedy.
- Apr 22, 2026 Inter-Annotator Agreement as a Ceiling on Measured Quality If two careful human raters disagree on a third of items, no scorer can exceed that agreement. Cohen kappa, Krippendorff alpha, and why the ceiling is not a target.
- Apr 6, 2026 Disaggregated Prefill and Decode Pools Separating prompt processing from token generation into distinct pools, the cache handoff that joins them, and how the capacity ratio between pools is set.
- Mar 25, 2026 Clipped N-Gram Precision and the Brevity Penalty How BLEU converts n-gram overlap into a single number, why counts are clipped against the reference, and where the brevity penalty stops correcting for short output.
- Mar 20, 2026 Operator Fusion and the Conditions That Block It Why merging adjacent operators into one kernel removes memory traffic, how producers and consumers are classified, and which graph shapes make the merge illegal or unprofitable.
- Mar 9, 2026 Rolling Back a Stateful Serving Fleet Under Live Traffic Restoring an earlier model is simple until the serving fleet holds request state, warm caches, and in-flight streams; then the rollback itself becomes a state transition with its own failure modes.
- Feb 26, 2026 Replica Packing and Topology Constraints Tensor parallelism turns a model into a group of devices that must be co-located and upgraded together. That group, not the individual replica, is the unit a scheduler can place.
- Feb 19, 2026 Position Bias in Pairwise Preference Judging When a judge compares two responses, the order they appear in changes the verdict. Measuring the effect requires swapping the pair and reading the disagreement rate.
- Feb 14, 2026 Replaying Logged Traffic to Evaluate a Candidate Offline Recording request and response pairs makes it possible to score a new model against decisions that were already made, but replay is only valid when the candidate sees the same context the incumbent saw.
- Feb 10, 2026 Paged KV Cache Allocation and the Block Table Why reserving contiguous cache for a worst-case output length caps batch size, and how fixed-size blocks with an indirection table remove most of that waste.
- Feb 4, 2026 Prefill and Decode: The Two Cost Curves Behind One Token Price Why the same model spends prefill time on arithmetic and decode time on memory bandwidth, and how that split decides what a token costs to serve.
- Feb 2, 2026 Graph Breaks, Fallback Paths, and Numerical Drift What happens at the boundary where tracing stops, why the eager fallback does not produce bit-identical results, and how a runtime keeps two execution paths consistent.
- Jan 12, 2026 Deadline-Aware Batch Assembly Turning latency targets into per-step scheduling decisions: slack, remaining-work estimates, and why maximising throughput and meeting deadlines diverge.
- Jan 8, 2026 Approximate Nearest Neighbour Indexes and the Recall They Trade Away How graph, cluster, and hash indexes reduce a linear scan to a bounded walk, which knob controls recall, and why recall is a property of the query distribution rather than the index.
- Jan 5, 2026 Preemption by Swap Versus Recomputation When the cache pool cannot serve a running sequence, the scheduler must take memory back; the two recovery strategies have different costs that depend on sequence length.
- Dec 16, 2025 Automatic Rollback Controllers and the Ways They Misfire A controller that reverts a deployment on a metric threshold is a feedback loop wrapped around a feedback loop; its failure modes are oscillation, latency in the evidence, and disagreement about what baseline means.
- Dec 5, 2025 Paged Key-Value Residency and Block Tables How fixed-size cache blocks and per-sequence block tables remove the allocation and fragmentation penalties of contiguous cache extents.
- Nov 19, 2025 Error-Compensated Rounding at Two and Three Bits How sequential rounding with a residual update turns a coarse grid into a usable one, why the column order matters, and when the compensation stops converging.
- Nov 11, 2025 How Key-Value Cache Bytes Set the Batch Ceiling The byte arithmetic that decides how many sequences fit beside the weights, why the ceiling moves with context length, and what remains after it is reached.
- Nov 6, 2025 Reserved Versus On-Demand Capacity: The Break-Even Utilization Comparing a discounted commitment against pay-per-use pricing when demand has a diurnal shape, including the utilization at which the commitment stops saving money.
- Sep 2, 2025 Metadata Filtering Inside Vector Search Why post-filtering a top-k list cannot satisfy a restrictive predicate, how pre-filtering collapses an index walk, and what the middle strategies actually cost.
- Sep 2, 2025 Prefix Caching and Shared Residency Deduplicating key-value blocks across requests that begin with the same tokens, how blocks are matched by hash, and what eviction must preserve.
- Jul 3, 2025 Chunk Boundaries Decide What Can Be Retrieved How fixed-size, sentence, and structural chunking place boundaries, what overlap recovers and what it duplicates, and why the split point is an index-time commitment.
- Jun 26, 2025 Hybrid Lexical and Dense Retrieval: Fusing Two Ranked Lists Why BM25 and embeddings fail on different queries, how reciprocal rank fusion combines their lists without comparable scores, and where the fusion still loses.
- Jun 25, 2025 Canary Promotion as a Repeated Statistical Test A canary compares a small exposed fraction against the incumbent, which means the promotion rule is a hypothesis test run many times, and its false-positive rate depends on how often it is evaluated.
- Jun 23, 2025 Prefill and Decode Disaggregation Prefill saturates arithmetic and decode saturates bandwidth. Putting them on separate machines lets each be sized for its own bottleneck, at the cost of moving the KV cache over a network.
- Jun 18, 2025 Decomposing Generated Text into Atomic Claims for Factuality Scoring Sentence-level factual accuracy conflates several errors into one label. Splitting output into self-contained claims changes both the denominator and the failure modes.
- Jun 12, 2025 Speculative Decoding Cost Accounting: The Acceptance Rate Threshold How draft tokens turn a bandwidth-bound decode step into several tokens, the acceptance rate at which the extra compute pays for itself, and why batch size moves that threshold.
- May 29, 2025 Tensor-Parallel Residency and Interconnect Pressure Sharding a model across devices splits weights and key-value heads, but every decode step becomes a collective; how that changes the memory and latency balance.
- May 24, 2025 Layout Assignment and the Elimination of Transposes A tensor has one logical shape and many physical orders; choosing one per value turns transpose chains into bookkeeping and decides which kernels can read their inputs directly.
- May 13, 2025 Batch Size, Occupancy, and the Marginal Cost of a Token How the cost of one more concurrent request falls as the batch grows, where it stops falling, and why the knee of that curve sets the price floor.
- May 13, 2025 Confidence Intervals for Accuracy on Small Evaluation Sets A score on a few hundred items is a sample from a binomial, and the interval around it is usually wider than the differences being reported. How to compute it and when the independence assumption fails.
- May 5, 2025 Kernel Launch Capture and Replay Recording a launch sequence once and replaying it as a single submission removes per-launch host overhead, at the cost of fixed topology and a separate memory model.
- May 3, 2025 Layer Sensitivity and Bit Allocation Under an Average-Width Budget How per-layer sensitivity is measured, why a greedy allocation on a non-additive error metric misleads, and how to spend a fixed average bit width across a network.
- Apr 20, 2025 Autoscaling a Token Service with Cold Starts and Diurnal Load Why reactive scaling fails for inference replicas, how to derive the lead time a scale-out needs, and what a warm pool costs against the peaks it covers.
- Apr 13, 2025 Splitting Prefill From Decode Inside a Step Prompt processing is compute-bound and token generation is bandwidth-bound; how mixing them in one step costs latency, and how chunked prefill bounds the interference.
- Apr 8, 2025 Requantization Inside the Kernel: From Int32 Accumulator to Int8 Output How a convolution kernel turns a 32-bit integer accumulator into the next layer's 8-bit input, why the multiplier is a fixed-point pair, and where saturation and rounding diverge from the reference.
- Jan 28, 2025 Context Dilution: When More Retrieved Passages Lower Answer Quality Why irrelevant context degrades generation rather than merely failing to help, how position in the prompt changes attention, and what to cut before raising k.
- Dec 9, 2024 Fake Quantization and the Straight-Through Estimator Why a rounding operation has no useful gradient, how training simulates quantization in the forward pass, and what the straight-through substitute does to the scale update.
- Sep 7, 2024 Sizing a Replica Pool from Arrival Rate and Latency Target Queueing arithmetic that converts requests per second and a latency objective into a number of replicas, including the headroom that utilization hides.
- Sep 2, 2024 Where Latency Is Measured, and Why Percentiles Do Not Average A serving latency number is a function of the point where the clock starts, and percentiles computed at different layers cannot be combined; both facts determine whether a monitoring figure means anything.
- Jul 23, 2024 Model Artifact Identity: Why a Checkpoint Hash Is Not a Deployment The behaviour a serving fleet exhibits depends on far more than tensor values, so deployment identity has to cover the tokenizer, the preprocessing path, and the runtime configuration as one unit.
- Jul 18, 2024 The Unbiased Pass@k Estimator for Code Generation Sampling k completions and checking whether any passes is a biased estimate of pass@k. The combinatorial correction, why it is needed, and how variance scales with the sample.
- Jun 19, 2024 Static Memory Planning from Tensor Liveness Intervals Turning a liveness interval per tensor into an offset in one arena, why interval-graph colouring is optimal for monotone lifetimes, and where greedy packing loses bytes.
- Apr 18, 2024 Cross-Encoder Reranking and the Cost of Reading Every Pair Why a reranker sees query and passage together, what that buys over a dual encoder, and how candidate depth turns a quality gain into a latency bill.
- Apr 10, 2024 Prefix-Cache-Aware Routing and Cache Locality A shared system prompt makes the same prefill work repeat on every replica. Routing by longest cached prefix trades load balance for a large reduction in prefill cost.
- Apr 1, 2024 Computational Graphs as an Intermediate Representation Why a model compiles to a list of pure value-producing nodes, how topological order doubles as the schedule, and which properties of a program a graph cannot express.
- Feb 20, 2024 The KV Cache Memory Budget and What It Costs per Token Deriving bytes per cached token from attention shapes, then turning the freed capacity into a hard ceiling on concurrency and a floor on cost per token.
- Feb 1, 2024 Why Perplexity Is Not Comparable Across Tokenizers Perplexity measures average surprise per token, so the vocabulary decides the unit. Converting to bits per byte removes the unit but not every confound.
- Jan 26, 2024 Detecting Benchmark Contamination from the Model Side A score on a public test set measures memorization as well as skill. N-gram overlap against training data, perturbation tests, and the limits of each detector.
- Dec 27, 2023 Tile Selection: Cost Models Against Measurement A fused kernel still needs a tile shape and a loop order, and the compiler must choose them either by estimating cost or by measuring candidates on the device.
- Dec 12, 2023 Recall-Oriented Summary Scoring with ROUGE Why summarization metrics divide by reference length rather than output length, how longest-common-subsequence scoring differs from n-gram counts, and what the recall direction lets a system get away with.
- Dec 11, 2023 Activation Outlier Channels and Mixed-Precision Decomposition Why a few hidden-state channels carry magnitudes far above the rest, how splitting those channels out of the integer path preserves accuracy, and what the split costs per token.
- Oct 17, 2023 Constant Folding and Algebraic Rewrites on a Fixed Graph How a compiler evaluates constant subgraphs at build time, which algebraic identities survive floating-point arithmetic, and why rewrite order determines the result.
- Aug 20, 2023 Autoscaling Model Servers on Queue Depth Rather Than Utilization Utilization-based autoscaling assumes work is interruptible and cheap to queue; for inference it produces oscillation, so the control signal has to be the wait time a request would experience.
- Aug 12, 2023 Point-in-Time Feature Reads and the Skew They Prevent Training a model on features computed after the fact and serving it on features computed at request time yields two different distributions; point-in-time joins are the mechanism that forces them to agree.
- Jul 14, 2023 Amortized Cost per Token: Depreciation, Power, and Idle Capacity Building a total cost of ownership for a token service from hardware life, power draw, and utilization, and showing why idle capacity is the largest term.
- Jun 12, 2023 Per-Channel and Group Scales: Granularity Against Metadata Cost Why one scale per tensor loses to one scale per output channel, how group-wise quantization spends extra bytes to buy resolution, and what the metadata costs at each granularity.
- Apr 28, 2023 Shadow Traffic: Duplicating Requests to a Candidate Without Serving Them How a serving stack runs a candidate model on a copy of live requests, what the duplication costs at the proxy, and why the comparison it produces is weaker evidence than it looks.
- Jan 8, 2023 Why a Larger Context Window Does Not Remove the Need for an Index The arithmetic that separates a corpus from a prompt, what a retrieval step buys in tokens and memory, and the narrow cases where loading everything still wins.
- Oct 31, 2022 Least-Outstanding Routing When Request Costs Vary Round robin assumes equal service cost and equal replica capacity. Token generation breaks both assumptions, so routers score replicas by work in flight instead of counting requests.
- Oct 18, 2022 Admission Control and the Memory Watermark Reserving headroom so that growth does not become preemption: how watermarks, reservation bounds, and head-of-line blocking interact in a cache pool.
- Aug 4, 2022 Range Calibration: Trading Clipping Error Against Rounding Error How the clipping threshold is chosen for a quantized tensor, why the minimum-error range is not the observed range, and how percentile and divergence criteria differ in what they preserve.
- Jun 27, 2022 Why Token Billing Diverges from GPU-Seconds The gap between what a request costs in hardware time and what it costs at a per-token price, and the workload shapes that make the two accounting systems disagree.
- Dec 7, 2021 Answer Normalization and the Fragility of Exact Match Exact-match scoring turns a string comparison into a quality verdict. The normalization pipeline in front of the comparison decides how much of the score is measurement.