Hey everyone, welcome to the thirty-third issue of The Main Thread. This piece is about building systems where backpressure is a first-class design concern, not an afterthought discovered during an incident.
Fast producers and slow consumers are a disaster waiting for the right traffic spike.
The scenario plays out the same way every time. A Kafka consumer processes messages fine at baseline load. Then, one day during a sale event, the traffic doubles. The consumer can’t keep up - it reads messages slower than they arrive. Due to this, the consumer lag grows. We add more memory to buffer the backlog. The buffer fills. Now messages are either dropped silently or the producer blocks and everything upstream stalls. Users see timeouts. On-call gets paged. The engineers spend the next three hours debugging a problem that has a name and same well-understood solutions.
The name is backpressure failure. The solutions are backpressure mechanisms.
Backpressure is the ability of a slow consumer to signal to a fast producer: slow down.
In physical systems, it’s literal - water pressure pushes back against a pump that’s moving too fast. In software systems, it’s structural - the consumer’s throughput limit must propagate upstream and constrain the producer’s send rate. When that signal is not present, the mismatch accumulates somewhere: in a buffer, in a queue, in latency, in dropped data, or in a cascading failure that takes down the whole pipeline.
TL;DR
Push models are backpressure-blind by default: the producer sends at its own rate regardless of consumer capacity
Pull models have backpressure built in: the consumer controls the rate by controlling when it asks for more
Buffers buy time but don’t solve the underlying rate mismatch; unbounded buffers are a time bomb
Rate limiting is producer-side backpressure: cap the input rate, so consumers are never overwhelmed
Reactive streams formalize backpressure into a protocol: consumers declare demand, producers send exactly that much
Drop, block, or buffer is the core decision framework: each has correct use cases and incorrect ones.
Push vs Pull: Where The Problem Lives
The root cause of most backpressure failures is using a push model in a context that requires backpressure.
In a push model, the producer decides when to send data. It sends at its own rate - as fast as it can generate events, or as fast as the network allows. The consumer receives whatever arrives. The consumer has no say. It it processes slower than data arrives, something must absorb the difference such as a buffer, a queue, or a dropped connection. The producer doesn’t know the consumer is struggling because nobody told it to care.
In a pull model, the consumer decides when to receive data. It requests the next batch only when it is ready to process it. The rate mismatch cannot accumulate because the producer sends only what the consumer asks. Backpressure is structural: if the consumer is slow, the producer sits idle waiting for the next request. No buffer overflow. No dropped messages. No cascading failure.
Most systems are a mix. HTTP/2 uses a pull model at the flow control layer, Kafka is pull by default, gRPC streaming is push by default (with optional flow control), WebSockets are push, with no built-in backpressure whatsoever.
Understanding which model our communication layer uses tells us where backpressure must be added explicitly.
Buffering: Necessary But Not Sufficient
Buffers are the first instinct when consumers fall behind. Add a buffer between producer and consumer; now the producer can keep sending while the consumer catches up.
Buffers are necessary as they smooth over transient rate spikes. A consumer that’s 20% slower that the producer for 5 seconds during a GC pause is fine if a buffer can absorb the difference. The problem starts when the rate mismatch is not transient but structural, and buffers are sized as if it is.
The mathematics of buffer overflow: If a producer sends at rate P and a consumer processes at rate C, and P > C, the buffer fills at rate P - C. With P = 10,000 msg/s, C = 9,000 msg/s, and a buffer of 100,000 messages, we have 100,000 / (10,000 - 9,000) = 100 seconds before the buffer is full. That's a 100-second delay before we hit the real problem.
Unbounded buffers are more dangerous pattern. They appear to solve the problem - the producer never blocks, the consumer processes at its own rate, everything looks fine in dashboards. Until memory runs out. An unbounded in-memory queue filling with unconsumed messages will eventually OOM the process. We have traded a visible capacity problem for a delayed crash.
Bounded buffers force the decision. When a bounded buffer fills, we must choose: block the producer, drop incoming messages, or reject them with an error. The choice is the backpressure decision, and making it explicitly is the right design. A bounded buffer with a clear overflow policy is better than an unbounded buffer with an implicit “crash when memory runs out” policy.
import asyncio
class BoundedQueue:
def __init__(self, maxsize: int, overflow_policy: str = "block"):
self._queue = asyncio.Queue(maxsize=maxsize)
self._overflow_policy = overflow_policy
self._dropped = 0
async def put(self, item):
if self._overflow_policy == "block":
await self._queue.put(item) # Blocks when full — backpressure propagates upstream
elif self._overflow_policy == "drop_newest":
try:
self._queue.put_nowait(item)
except asyncio.QueueFull:
self._dropped += 1 # Newest item is dropped; oldest is preserved
elif self._overflow_policy == "drop_oldest":
if self._queue.full():
self._queue.get_nowait() # Evict oldest to make room
self._dropped += 1
self._queue.put_nowait(item)
async def get(self):
return await self._queue.get()
@property
def dropped_count(self):
return self._droppedThe overflow_policy parameter forces an explicit decision at design time.
Rate Limiting As Producer-Side Backpressure
Rate limiting at the producer is backpressure applied proactively rather than reactively. Instead of waiting for the consumer to signal distress, we cap the input rate, so it never exceeds what the consumer can handle.
This is the right pattern when:
The consumer throughput ceiling is known and stable
The product is an external actor (user requests, API clients) whose rate we control at the boundary
We want to protect consumer capacity rather than adapt to it dynamically
A token bucket enforces rate limits with burst tolerance. The bucket holds tokens; each request consumes one. Tokens refill at a constant rate. When the bucket is empty, requests either wait or are rejected.
import time
import threading
class TokenBucket:
def __init__(self, rate: float, capacity: int):
self.rate = rate # Tokens added per second
self.capacity = capacity # Maximum tokens (burst ceiling)
self.tokens = capacity # Start full
self.last_refill = time.monotonic()
self._lock = threading.Lock()
def _refill(self):
now = time.monotonic()
elapsed = now - self.last_refill
new_tokens = elapsed * self.rate
self.tokens = min(self.capacity, self.tokens + new_tokens)
self.last_refill = now
def acquire(self, tokens: int = 1) -> bool:
with self._lock:
self._refill()
if self.tokens >= tokens:
self.tokens -= tokens
return True
return False # Reject: bucket empty
# At the API gateway layer: 1000 req/s sustained, burst up to 1500
limiter = TokenBucket(rate=1000, capacity=1500)
def handle_request(request):
if not limiter.acquire():
return Response(429, "Too Many Requests", retry_after=1)
return process(request)Rate limit at the boundary is clean and predictable. The consumer is never exposed to more than its rated capacity.The tradeoff is that rate limiting drops or delays requests regardless of the consumer’s actual current load. If the consumer has spare capacity during an off-peak period, rate limit doesn’t use it - the ceiling is fixed.
For dynamic rate matching, we need feedback from the consumer. This is where reactive streams become relevant.
Reactive Streams: Formalizing the Backpressure Protocol
Reactive Streams is a specification (implemented in Java’s java.util.concurrent.Flow, RxJava, Project Reactor, Akka Streams) that makes backpressure a first-class protocol between producers and consumers.
Here, consumers declare demand explicilty, and producers send at most that much.
The request(N) call is the backpressure signal. The publisher cannot send more items than the subscriber has requested. If the publisher is generating faster than the subscriber requests, it must either buffer, drop, or apply backpressure upstream.
In Project Reactor (Spring WebFlux’s reactive foundation):
Flux.fromIterable(largeDataSet)
.onBackpressureBuffer(1000)
.publishOn(Schedulers.boundedElastic())
.subscribe(
item -> processItem(item),
error -> handleError(error),
() -> log.info("Stream complete")
);The inBackpressureBuffer(1000) call is explicit: buffer upto 1000 items, then apply the overflow policy. We are forced to make the decision at the write time.
TCP is the original reactive stream. The TCP receive window is a demand signal: the receiver advertises how many bytes it can accept, and the sender blocks when the window fills. This is why a fast server sending to a slow client eventually stalls - TCP backpressure propagates through the socket buffer and back to the application layer. HTTP/2 extends this with per-stream flow control. Most higher-level protocols reinvent or disregard this mechanism, often unwisely.
Queue Based Backpressure Patterns
Message queues sit between producers and consumers, decoupling their rates. But a queue is not a backpressure mechanism by itself - it’s a buffer. Backpressure requires a feedback loop from the queue back to the producer.
Consumer Lag Has A Backpressure Signal
In Kafka, consumer lag (the difference between the latest offset and the consumer’s committed offset) is the backpressure signal. When the lag grows, the consumer starts falling behind.
from kafka import KafkaConsumer, KafkaAdminClient
def get_consumer_lag(topic: str, group_id: str) -> dict[int, int]:
admin = KafkaAdminClient(bootstrap_servers="localhost:9092")
consumer = KafkaConsumer(group_id=group_id, bootstrap_servers="localhost:9092")
partitions = consumer.partitions_for_topic(topic)
lag_by_partition = {}
for partition in partitions:
tp = TopicPartition(topic, partition)
end_offset = consumer.end_offsets([tp])[tp]
committed = consumer.committed(tp) or 0
lag_by_partition[partition] = end_offset - committed
return lag_by_partition
# Adaptive producer throttle: slow down when consumer lag exceeds threshold
class LagAwareProducer:
def __init__(self, topic: str, group_id: str, max_lag: int = 10_000):
self.topic = topic
self.group_id = group_id
self.max_lag = max_lag
def send(self, producer, message):
lag = sum(get_consumer_lag(self.topic, self.group_id).values())
if lag > self.max_lag:
# Consumer is behind — pause before sending more
wait_seconds = min(lag / self.max_lag, 5.0) # Cap at 5s backoff
time.sleep(wait_seconds)
producer.send(self.topic, message)This is a feedback control loop: lag measurement → throttle decision → slower production → lag reduction. It is crude but effective for batch producers that can tolerate adaptive throughput.
Dead Letter Queues and Circuit Breakers
When consumers consistently fail to process messages due to exceptions, timeouts, invalid data, messages pile up as retries. Without a dead letter queue (DLQ), poison messages retry indefinitely, consuming consumer capacity and preventing healthy messages from being processed.
A DLQ is backpressure at the failure level: instead of retrying bad messages indefinitely and degrading consumer throughput, bad messages are quarantined so the healthy flow can continue. Combined with circuit breakers on the consumer’s downstream dependencies (covered in the Circuit Breakers piece, this gives us a complete failure isolation strategy for queue-based systems.
Drop, Block, or Buffer: The Decision Framework
When a consumer can’t keep up, we have exactly three options for the excess load. Each is correct in some contexts and catastrophically wrong in others.
Buffer: When We Can Afford to Wait
We should buffer when:
The rate mismatch is temporary (traffic spikes, GC pauses, short-lived slowdowns)
The data has value that justifies the memory cost of holding it
Out-of-order processing is acceptable
We have bounded, monitored buffer capacity
We shouldn’t buffer when:
The rate mismatch is structural: buffering just delays the inevitable crash
Data has a TTL and stale processing is worse than no processing (sensor readings, stock quotes, live metrics)
Memory is constrained and we can't predict buffer growth
Always bound the buffers. Unbounded buffers are a hidden time bomb. We must set a capacity limit, monitor fill level, and alert at 70%. If you're routinely hitting 70%, the capacity problem must be solved, not buffered around.
Block: When We Need Every Message
We should block (applying backpressure to the producer) when:
Data loss is unacceptable - financial transactions, audit logs, user commands
The producer can tolerate latency in case of batch jobs, background processing pipelines
We have end-to-end flow control and blocking won't cause upstream deadlock
We shoundn’t block when:
The producer is a user-facing request - blocking an HTTP handler blocks the connection, degrading user experience
The producer can't safely hold state while waiting - real-time event streams, sensor feeds
Blocking would cascade into upstream resource exhaustion
Blocking is the right answer for durability-critical pipelines. It's the wrong answer for interactive, latency-sensitive systems.
Drop: When Staleness Is Worse Than Loss
We should drop when:
Data has a short validity window - real-time metrics, live video frames, current sensor readings
The consumer's job is to track the current state, not process every event
We would rather have a slightly stale view than a backed-up queue of outdated readings
class LatestValueBuffer:
"""
Keeps only the most recent value per key.
Old values are dropped when a newer one arrives.
Correct for live metric feeds, position tracking, status indicators.
"""
def __init__(self):
self._latest = {}
self._lock = threading.Lock()
def put(self, key: str, value):
with self._lock:
self._latest[key] = value # Old value is silently overwritten
def get(self, key: str):
with self._lock:
return self._latest.pop(key, None)
This is the right data structure for a dashboard consuming device telemetry. We don't need every temperature reading from the last 30 seconds; we need the current temperature. Dropping intermediate readings is correct behavior, not a failure.
Observability for Backpressure
Backpressure failures are silent by default. A buffer that's 80% full doesn't throw an exception. A consumer falling behind doesn't log a warning. We discover the problem when the buffer fills, the OOM happens, or the queue lag is already enormous.
We must instrument these signals explicitly:
Queue depth / buffer fill level: Alert at 70%, treat 90% as an incident. If the system is regularly at 50%, we should recalibrate capacity or producer rate.
Consumer throughput vs producer rate: Track both independently. The gap is the metric. A growing gap means we are heading toward overflow; a stable gap means we are at steady-state (which may still be unsustainable if absolute fill level is high).
Processing latency percentiles: p99 latency climbing while p50 is stable often means a queue is filling. The median consumer is fine; the tail is processing work that's been waiting.
Drop rate: If we are using a drop policy, the drop rate is our primary health signal. A non-zero drop rate during normal operation is a problem; a non-zero drop rate during known traffic spikes may be acceptable - as long as we have decided that.
The Mental Model
Every data pipeline has a throughput ceiling at each stage. Backpressure is the mechanism by which a slow stage signals its ceiling to faster upstream stages. Without backpressure, the mismatch accumulates: in buffers, in queues, in latency, and eventually in failures.
The engineering task is not to eliminate rate mismatches because they're inevitable in any real system with variable load. The real task is to handle them explicitly:
Know the model. Is each link in the pipeline push or pull? Push links need explicit backpressure mechanisms.
Bound the buffers. Every buffer needs a capacity limit and an overflow policy decided at design time.
Choose the policy deliberately. Block, drop, or buffer - each is correct somewhere and wrong everywhere else.
Make backpressure visible. Queue depth, lag, and drop rate are first-class metrics, not afterthoughts.
The systems that survive traffic spikes know what to do when capacity runs out.
Namaste!
— Anirudh
