The Illusion of Scale, Part 4: Latency Is a Design Decision, Not a Measurement

Anusha Mukka Posted on May 31 The Illusion of Scale, Part 4: Latency Is a Design Decision, Not a Measurement # distributedsystems # performance # architecture # backend I need to tell you about the time I confidently presented a latency budget to a stakeholder and then watched it disintegrate in production like wet tissue paper. We had a system with a 200ms latency budget. We'd measured every component. Auth service: 15ms. Business logic: 30ms. Database query: 40ms. Total: well inside budget. I remember feeling good about this. We'd done the work. We had numbers . We shipped it. In production, the auth call that took 15ms in testing was regularly hitting 200ms at peak. So I panicked, ran profiling tools, even on-boarded new profiling tools coz LLM agents did not exist back then and I was desperate and sure that a block of code was causing it, maybe writing to DB, reading from DB, calling an API, something.. something that would explain this, but there it is.. the bitter truth! The auth service was slow. Because it was shared with four other services, all of which peaked at the same time, and nobody, I mean nobody, including me -- had reserved any capacity for ours. We had 15ms allocated. We were spending 200ms. The rest of the budget was irrelevant at that point. That was the moment I stopped treating latency as a measurement exercise and started treating it as a design problem. Measure-and-optimize sounds like engineering rigor. In practice, it's usually "discover your architectural constraints too late to change them cheaply." This is Part 4 of a series on the assumptions that quietly wreck systems at scale. You can't optimize your way out of a bad structure The instinct is totally reasonable. Build the thing, run load tests, measure latency, optimize what's slow. Feels like good engineering discipline. I've given this exact advice to junior engineers. The problem is: by the time you're measuring in production, the decisions that created the latency are three laye