Tag
request-routing
- Jun 2, 2026 Routing Requests Between Old and New with a Strangler Facade How a routing layer in front of a legacy application lets capabilities move to a replacement one slice at a time, what the routing key must capture, and why session and data ownership decide the order. software architecture
- Apr 10, 2024 Prefix-Cache-Aware Routing and Cache Locality A shared system prompt makes the same prefill work repeat on every replica. Routing by longest cached prefix trades load balance for a large reduction in prefill cost. inference systems
- May 13, 2023 Route Matching Precedence and the Cost of Ambiguity How a dispatcher chooses one handler among overlapping patterns, why specificity beats registration order, and where percent-decoding changes which route wins. backend systems
- Oct 31, 2022 Least-Outstanding Routing When Request Costs Vary Round robin assumes equal service cost and equal replica capacity. Token generation breaks both assumptions, so routers score replicas by work in flight instead of counting requests. inference systems