Inference Engineering

Inference Engineering: Part 7/8: How MoE Creates Uneven GPU Work

Inference Engineering: Part 7/8: How MoE Creates Uneven GPU Work

What happens when different tokens need different model components on different GPUs?

Nov 16, 2026

Inference Engineering: Part 6/8: Should Prefill And Decode Run On Different GPUs?

Inference Engineering: Part 6/8: Should Prefill And Decode Run On Different GPUs?

When does separating two phases improve the service, and what new costs does it create?

Nov 9, 2026

Inference Engineering: Part 5/8: When Speculative Decoding Backfires

Inference Engineering: Part 5/8: When Speculative Decoding Backfires

Can spending extra work on a draft reduce the number of slow generation steps?

Nov 2, 2026

Inference Engineering: Part 4/8: Measuring LLM Serving Under Load: Throughput and Latency

Inference Engineering: Part 4/8: Measuring LLM Serving Under Load: Throughput and Latency

How can a system generate more tokens overall while giving users a worse experience?

Oct 26, 2026

Inference Engineering: Part 3/8: GPU Memory, Token By Token

Inference Engineering: Part 3/8: GPU Memory, Token By Token

Why can a model that fits on the GPU still run out of memory?

Oct 19, 2026

Inference Engineering: Part 2/8: One Long Prompt Delays Everyone Else's Tokens

Inference Engineering: Part 2/8: One Long Prompt Delays Everyone Else's Tokens

Why can one request with a long input disrupt users whose answers are already streaming?

Oct 12, 2026

Inference Engineering: Part 1/8: One Request, Many Turns on the GPU

Inference Engineering: Part 1/8: One Request, Many Turns on the GPU

An LLM answer needs repeated computation and growing memory. Follow what happens when many users need both at once.

Oct 5, 2026

The Main Thread

AboutEssaysXGitHubRSS
© 2026 The Main Thread.
beehiivPowered by beehiiv