Backend Systems · Cluster
Message Queues
Delivery semantics and consumer groups.
- 01 Append-Only Logs and the Queue Abstraction Destructive reads and append-only retention produce different broker state, different fan-out costs, and different limits on what a consumer can ask for.
- 02 Delivery Semantics and Where Duplicates Come From At-most-once, at-least-once, and exactly-once describe the disposition of a single message when an acknowledgement is lost, not a property of the network.
- 03 Acknowledgements, Offsets, and Visibility Timeouts A broker can learn about completion only through an explicit signal or an expiring lease, and the timing of both decides how much work is repeated after a crash.
- 04 Consumer Groups and Partition Assignment How a group coordinator keeps one owner per partition, what a generation change does to in-flight work, and why the assignor choice changes rebalance cost.
- 05 Producer Batching, Compression, and Acknowledgment Pipelining A produce request has a fixed cost per round trip and per replica, and batching, compression, and in-flight limits each trade latency for a different part of that cost.
- 06 Credit-Based Flow Control and Consumer Prefetch Limits A broker that pushes messages needs permission to send, and the counter that grants it decides both memory consumption and how evenly work lands across consumers.
- 07 Ordering Guarantees and Head-of-Line Blocking A log guarantees order only within a partition, and that guarantee survives retries, consumer parallelism, and rebalances only under specific conditions.
- 08 Log Compaction, Retention, and Replay Retention deletes from the start of a log and compaction keeps only the newest value per key, which turns the log into a replayable store of current state.
- 09 Dead Letter Queues and Poison Message Quarantine Which failures deserve another attempt, how a broker counts attempts without understanding the error, and why the quarantine queue needs an owner and a bound.
- 10 Exactly-Once Delivery Across System Boundaries The dual-write problem, the outbox that removes it, and the narrow sense in which a message can be applied exactly once when the database and the broker are separate systems.