Why Your Load Balancer is Solving a Market Design Problem

Hey everyone, welcome to the thirty-seventh issue of The Main Thread.

Pick up any systems textbook and load balancing reads like a parade of small algorithmic tricks: round-robin, weighted round-robin, least-connections, consistent hashing. Each one is presented as an engineering choice, determined by tail latency and CPU overhead. That framing is correct but it misses something.

Now, pick up any economics textbook and read about market design: auctions, matching markets, mechanism design, and you will notice the questions are eerily similar.

How do you allocate a scarce resource among competing claimants? How do you elicit honest information from agents who have incentives to lie? How do you balance efficiency against fairness?

Your load balancer is answering exactly these questions. Every request is a buyer. Every backend is a seller of compute capacity. The algorithm is the market mechanism that matches them. You don’t usually think of this way, but the moment you do, a lot of decisions that look like config changes come out as economic policy.

This isn’t a stretch. The two fields converge because they are solving the same underlying problem: given a fixed pool of capacity and a stream of demands, who gets what, and how do we know we picked well? The economists got there first and developed a mature vocab for it. We, as engineers, have never been told this vocab that applies to our job.

This issue talks about what changes when you see load balancing through the lens of market design.

Every load balancing algorithms is an allocation rule, that is, a function from “the current state of the world” to “which seller serves this buyer”.

This is uniform allocation. Each backend gets a 1/N share of incoming demand. It’s the algorithmic equivalent of equal division: simple, fair, in the trivial sense, and indifferent to whether the recipients can actually use what they are given.

It works well when every backend is identical and cost of every request is same. The moment either assumption breaks, round-robin allocates badly, and unlike market, it has no feedback mechanism to notice.

This algorithm introduces property rights over capacity. Each backend has a declared share, let’s say 4:2:1 for three boxes of different size. The allocation respects those weights. This is planned economy with admin-set quotas.

It is better than round-robin for heterogeneous fleets, but the weights are a guess about capacity, not a measurement of it. If the weights are wrong, we might over-allocate to weak servers and starve strong ones.

This is the first algorithm that incorporates observed state. It allocates the next request to the backend with the smallest number of active connections (a proxy for current load).

This is closer to how markets actually work: capacity goes where demand shows it’s needed. The proxy is imperfect (a backend with many idle long-poll connections might look busier than one eating through CPU), but the principle is right: allocate against the observed pressure, not against assumptions.

This is the one that should make economists smile. Pick two backends at random; send the request to whichever has smaller queue. Mathematically, this exponentially improves the maximum load compared to pure random assignment, with O(1) overhead.

It is the stochastic version of comparison-shopping: we don’t have full information about every seller, but a small sample is as good. Real markets work this way too - buyers don’t survey every store, they check a few and pick the better deal. The information cost matters as much as the allocation rule.

This uses a meaningful price signal, that is, actual latency. The backend that has been responding fastest gets the next request. This is closer to a true market because the signal (observed performance) is harder to fake than a self-reported metric.

This is still imperfect (recent latency may not predict future latency under a different load), but it tightens the loop between observation and allocation.

We can rank these algorithms by how much information about the world they incorporate into the allocation:

The pattern is clear: better algorithms use better signals. This is exactly what economists mean when they say markets aggregate information that no central planner could collect.

Let’s consider a different kind of load balancer which focuses on “which backend owns this key” instead of “which backend serves this request“. Caches, sharded databases, distributed object stores all face this problem. The standard answer is consistent hashing which is now baked into Cassandra, DynamoDB, Memcached, and basically every modern partitioned system.

Consistent hashing is, formally, a fair division problem. We have a unit interval (a hash ring). We have N agents (backends). WE need to partition the interval among them in a way that:

Property 2 is very important. In standard fair-division economics, when a participant exits, we typically have to redivide everything. Consistent hashing’s only appeal is that it doesn’t: only O(K/N) keys move when one of N nodes go away, where K is the total keyspace. That’s an unusual and valuable property.

Economists have a name for this: It’s a form of stable matching with low churn cost. The cost of recognizing the allocation when participant change is small. Real world examples are rare - when a doctor leaves a residency match program, the whole match doesn’t redo itself, but only because of careful mechanism design. Consistent hashing achieves the same thing with a beautifully simple data structure.

The vanilla version has a known fairness problem: with random placement on the ring, some nodes get arcs much larger than 1/N. We fix this by using virtual nodes, which is essentially a replication of identity. Each physical node claims many points on the ring, smoothening the per-node-arc length toward the mean. This is the same idea a splitting a single bidder into many small bids to reduce variance in an auction outcome. Small samples are noisy, but many small samples converge.

Google’s Maglev hashing (2016) takes this further. It guarantees near-perfect even distribution and bounded disruption, at the cost of precomputed lookup table. From a mechanism design view, Maglev is choosing a slightly more centralized allocation in exchange for tighter fairness guarantees. That’s a real economic tradeoff, not just an engineering one.

The takeaway is: when we choose a partitioning scheme, we are choosing a fairness criterion. “Consistent hashing with 100 vnodes per server“ is a policy decision about how much variance in per-node load we will tolerate, expressed in code.

This is where the analogy stops being cute and starts being useful.

Most load balancers rely on backends to report their own health. The backend exposes /healthz, returns 200 OK if it considers itself alive, and the load balancer trusts that signal. This is the classic information symmetry problem that George Akerlof described in The Market for Lemons (1970): the seller knows more about the quality of the goods than the buyer does, and has an incentive to misrepresent it.

A backend that’s deeply degraded - failing slowly, leaking memory, processing requests but corrupting data - can absolutely return 200 OK from its health endpoint. The endpoint is a separate code path. It doesn’t exercise the database connection. It doesn’t actually serve a representative request. It just answers, “yes, I am here”.

Akerlof’s argument was when buyers can’t tell good cars from lemons, the market unravels: buyers offer the average price, owners of good cars refuse to sell at that price, the average quality drops, the price drops, and so on. If we translate this to load balancing: when health checks can’t distinguish healthy backends from healthy ones, the load balancer keeps sending traffic to the degraded ones. The worst part is, the degraded ones often have more spare capacity (they are failing requests fast, or rejecting work) and so look more attractive to a least-connections load balancer. Lemons get more traffic, not less.

The economics literature has answers to the lemons problem, and engineering has converged on analogous solutions:

Akerlof’s collaborator Michael Spence proposed that high-quality sellers can credibly distinguish themselves through costly signals (a warranty, a degree, a reputation). The engineering analogue is: deep health checks that exercise real code paths - actually run a query against the database, actually call a downstream service, actually allocate memory. A backend that passes a deep health check is sending a costly signal that a degraded backend can’t fake..

Skip self-reports, and measure outcomes. Passive health monitoring tracks the actual error rate and latency of requests the load balancer sent to each backend, and ejects backends that misbehave. Envoy’s outlier detection works this way. So does AWS’s “least outstanding requests“ algorithm. The principle is same as in mechanism design: don’t trust agent reports when we can observe outcomes directly.

Track behaviour over time, not just current state. A backend that has failed three times in the last minute should be treated with suspicion even if its current health checks is green. Circuit breakers are precisely this - a reputation system for backends. The “open / half open / closed“ states are reputation tiers.

The lesson for us is that a load balancer that trusts self-reports is a load balancer with no defence against the lemons problem. Production-grade systems combine all three: deep health checks (signaling), passive outlier detection (verification), and circuit breakers (reputation). It we are running anything serious on a load balancer that only does shallow /healthz polling, we have designed a market that selects for lemon.

Sticky sessions (pinning a user’s requests to the same backend for the duration of their session) feel like a pure engineering optimization. They preserve in-process state, they make caching effective, they avoid having to rehydrate context on every request.

But economically, session affinity is a switching cost. Once a user is bound to a backend, the cost of moving them is non-zero: we would lose in-memory state, blow the local cache, possibly recompute expensive context. The load balancer’s freedom to reallocate is constrained.

Industrial organization economics have studied switching costs for decades. The findings are uncomfortable:

The pragmatic implication of this is that session affinity is not free, even when it’s “working”. We are trading allocation efficiency for state-locality benefits. That trade is sometimes correct (a chat application where context rebuilding costs seconds) and sometimes deeply wrong (a stateless API where the only “stickiness” is a thin layer of caching). Most productions systems use sticky sessions reflexively, without measuring whether the locality benefit exceeds the misallocation cost.

The cleanest workaround is the same one economists recommend for switching-cost-heavy markets: make switching cheap. If session state lives in a shared cache (Redis, Memcached) rather than in-process memory, any backend can serve any request, and stickiness becomes a hint rather than a binding constraint. We have eliminated the switching cost rather than tolerating it.

This is also why “consistent hashing for cache locality, but with fallback to any backend on miss” is such a robust pattern. We get locality benefit when it’s available, and we don’t pay the misallocation cost when it is not.

Mechanism design asks a sharper question than market design. It doesn’t take the rules as given and study what happens. It asks: if we get to design the rules, can we design them so that participants have an incentive to behave the way we want?

The canonical example is the Vickrey auction. In a sealed-bid second-price auction, each bidder submits a bid; the highest bidder wins but pays the second-highest bid. This rule is incentive-compatible: it’s in every bidder’s interest to bid their true valuation. Lying gains them nothing and may cost them. Truthful revelation is the dominant strategy.

What would an incentive-compatible load balancer look like?

The setting: backends report their current capacity/load to the load balancer. The load balancer wants those reports to be honest. Backends might have an incentive to lie - for instance, to under-report load and claim more traffic (gaming a metered system), or to over-report load and shed traffic to peers (laziness).

The mechanism design move is: design the allocation rule so honest reporting is the dominant strategy. Some sketches:

Outcome-based allocation. Don't allocate based on reported load; allocate based on observed performance from the previous window. If a backend lied about capacity and accepted more traffic than it could handle, its observed latency went up and its share goes down next window. Lying is self-punishing. This is exactly what passive outlier detection does, and it's why it's so robust.

Pricing as a sorting mechanism. In multi-tenant systems where multiple services share a fleet, you can charge tenants for the resources they consume and let backends bid for work based on their marginal cost. Tenants reveal their willingness-to-pay (priority); backends reveal their available capacity (cost). This is the design behind Borg-style cluster managers and behind cloud provider spot markets. Incentive compatibility is approximate, but the system aligns reported state with consequences.

Random audits. Borrow from tax enforcement. Most reports are trusted, but a random sample is verified by sending a synthetic test request and measuring the actual response. Backends that fail audits are penalized in their share allocation. The cost of cheating becomes high enough that honest reporting is the rational choice. Engineering teams already do this informally with synthetic monitoring; few do it as part of the load-balancing decision itself.

The frontier here is interesting. In a single-team, single-fleet setting, backends don't have meaningful preferences and the lemons problem is unintentional - backends "lie" because their health endpoint is a poor proxy for their actual health, not because of strategic intent. But in multi-tenant cloud environments, in service meshes spanning organizational boundaries, in federated systems composed of independently-operated nodes, the strategic incentives become real. The load balancer is no longer a benevolent dictator; it's a market designer with adversarial participants.

If we build infrastructure for that world, the mechanism-design literature has fifty years of mathematical work waiting for us. The question "can I trust this report?" is one that economists have answered carefully. Engineers have mostly punted on it.

Hal Varian - UC Berkeley professor, long time Chief Economist at Google, author of Intermediate Microeconomics (the textbook everyone reads) - built much of his career on the theory and practice of allocation mechanisms. His work on Google’s ad auctions is one of the most famous engineering-economics applications: turning ad slot allocation into a mechanism that approximates a Vickrey-Clarke-Groves outcome at internet scale.

A few of Varian’s points map directly to load balancing:

An efficient allocation maximizes total surplus, not equal shares. In economics, an allocation is Pareto efficient if you can't make anyone better off without making someone worse off. This is not the same as fairness. A round-robin balancer is "fair" in the sense of equal allocation, but it's not Pareto efficient - sending a CPU-bound request to a CPU-bound backend when a memory-bound backend is idle is leaving surplus on the table. Real load balancers should optimize for total throughput / minimum tail latency, not for equal per-backend QPS. Equal QPS is a coincidence of optimal allocation when backends are identical, not a goal.

Prices aggregate information that no planner has. Varian is fond of pointing out that a market price encodes information from millions of agents that no central planner could collect. The load-balancing analogue: a price-like signal - observed latency, observed error rate, observed queue depth - aggregates information about backend state that no health-check protocol could fully describe. Algorithms that listen to this signal (least-response-time, EWMA-based, outlier detection) outperform algorithms that don't.

The right mechanism depends on the structure of preferences and information. There is no universal optimal auction; Vickrey is optimal under certain assumptions, English auctions under others, sealed-bid first-price under yet others. The same is true of load balancing. Round-robin is optimal under "identical backends, identical requests, no observation cost." Least-connections is optimal under "observable load, low-cost monitoring." Consistent hashing is optimal under "stateful workload with locality value." There is no universally best load balancer because there is no universal market structure.

The systems-engineering tendency to look for a "best" load balancing algorithm reflects the same naivety as looking for a "best" auction. The right answer is: characterize the allocation problem (request properties, backend properties, information costs, switching costs), then pick the mechanism whose assumptions match.

When we start treating load balancing as market design, several things become hard to unsee.

We stop treating "the load balancer" as a single decision. The actual mechanism is a stack: an allocation rule, a health-checking subsystem, a circuit breaker, a session affinity policy, a partitioning scheme. Each is a separate market-design choice with its own assumptions and failure modes. Conflating them - "we use AWS ALB" - hides the design space.

We start questioning your defaults. Sticky sessions on by default? We have imposed a switching cost. Shallow health checks only? We have created a lemons market. Round-robin to a heterogeneous fleet? We have implemented central planning with bad quotas. None of these are necessarily wrong, but they should be defended, not defaulted.

We take metrics more seriously as price signals. Latency is not just an SRE dashboard number; it's the price our system is paying for the current allocation. A spike in tail latency on one backend is a market signal that capacity has become scarce there. Algorithms that respond to it allocate efficiently; algorithms that ignore it don't.

We design for adversarial inputs more readily. When we understand the lemons problem, we don't need a security incident to motivate adding outlier detection. When we understand mechanism design, we don't need to be told that self-reported metrics in a multi-tenant system are gameable.

We appreciate why this is a hard, unsolved area. "How do we allocate compute capacity efficiently across heterogeneous workloads with imperfect information and strategic participants?" is not a question with a tidy answer. It's an open problem in two fields at once. The fact that load balancers work as well as they do is a triumph of engineering pragmatism over theoretical impossibility.

The reason load balancing is hard isn't that the algorithms are complicated. The algorithms are simple. It's hard because the problem underneath is the oldest one in economics: how to allocate scarce resources among competing demands when nobody has full information and some participants might lie.

Treating that problem as if it's solely a systems problem leaves real techniques on the table. Mechanism design has decades of mathematics about incentive compatibility. Industrial organization has decades of work on switching costs. Market microstructure has decades of work on information aggregation. Most of this is directly applicable, and almost none of it shows up in systems textbooks.

The reverse is also true: economists who study these problems abstractly rarely get to see them implemented at the scale and speed of a production load balancer. The fact that algorithms like power-of-two-choices and consistent hashing emerged from systems engineering, not from economics, suggests that this conversation should run in both directions.

What you should take from this is a new vocabulary for the trade-offs you are already making. Once you can name them - switching costs, information asymmetry, signaling, mechanism design - you can argue about them. And once you can argue about them, you can do something other than accept the defaults.

Your load balancer is doing market design. The only question is whether it's doing it well.

What's the most economically-loaded engineering decision you have made without realizing it? I am collecting examples of systems decisions that turn out to be market design problems in disguise: pricing, scheduling, rate limiting, queue ordering, anything. The patterns are everywhere once you start looking.

Hit reply. I read everything.

If this clicked, forward it to someone who works on infrastructure or someone who works on incentives: they're solving the same problem from different sides. And if you want more of this kind of cross-domain thinking, subscribe to The Main Thread - practical engineering from unusual angles, one essay per week.