Tag
batching
- Sep 20, 2026 Adapter Residency When One Replica Serves Many Fine-Tunes Low-rank adapters let dozens of fine-tunes share a base model, but each token still needs one adapter applied. Residency, batch grouping, and routing decide whether that sharing pays. inference systems
- May 1, 2026 Batching and the Amortization of Per-Item Cost How grouping work converts a fixed per-operation cost into a per-batch cost, where the delay comes from, and why batch size is a throughput constraint rather than a tuning knob. performance engineering
- Jan 12, 2026 Deadline-Aware Batch Assembly Turning latency targets into per-step scheduling decisions: slack, remaining-work estimates, and why maximising throughput and meeting deadlines diverge. inference systems
- Dec 10, 2025 Optimizations That Move Cost Instead of Removing It Batching, precomputation, and compression reduce one metric by increasing another, so measuring only the targeted metric counts a trade as a win. performance engineering
- Nov 21, 2025 Batching and the Amortization of Per-Request Cost How grouping work into larger units trades latency for throughput, which fixed costs actually amortize, and where a batch becomes slower than the sum of its parts. backend systems
- Nov 11, 2025 How Key-Value Cache Bytes Set the Batch Ceiling The byte arithmetic that decides how many sequences fit beside the weights, why the ceiling moves with context length, and what remains after it is reached. inference systems
- Oct 14, 2025 GraphQL Execution: Resolvers, Fan-Out, and Query Cost How a GraphQL document becomes a tree of resolver calls, why list fields produce N+1 queries, and which limits keep a single endpoint from being a denial-of-service vector. backend systems
- Sep 23, 2025 Batching and Flush Policy in Telemetry Agents How size and time triggers interact in an export buffer, what per-request overhead a batch amortizes, and why an unbounded queue is not backpressure. observability
- Aug 27, 2025 Batching a Backfill Without Holding Locks or Losing Restarts Key-range batching, batch sizing, commit frequency, and the idempotence rules that let a long data update be interrupted and resumed safely. databases
- May 13, 2025 Batch Size, Occupancy, and the Marginal Cost of a Token How the cost of one more concurrent request falls as the batch grows, where it stops falling, and why the knee of that curve sets the price floor. inference systems
- Jan 26, 2025 Producer Batching, Compression, and Acknowledgment Pipelining A produce request has a fixed cost per round trip and per replica, and batching, compression, and in-flight limits each trade latency for a different part of that cost. backend systems
- Sep 6, 2024 KV Cache Arithmetic: Bytes per Token, Layers, and the Batch Size Ceiling How the decode-time key and value cache is sized, why it scales with every layer rather than only the last, and how its growth sets the maximum batch on a fixed device. deep learning
- Dec 11, 2023 MutationObserver Records, Batching, and Microtask Delivery Observers accumulate a record per mutation and deliver them once per microtask checkpoint, which coalesces bursts into a single callback and makes delivery timing a question about the event loop. web platform
- Sep 26, 2023 Padding, Packing, and Masking for Variable-Length Sequence Batches What it costs to batch sequences of different lengths behind a rectangular tensor, how masks keep padding out of losses and attention, and how packed sequences avoid the wasted arithmetic. deep learning
- Feb 20, 2023 Radio Tail Energy and the Cost of the Second Request Energy per byte is not a constant on a cellular radio; it depends on how long the radio stays powered, which makes request spacing a first-order energy decision. mobile computing