Reducing Tail Latency on High-Throughput RPC

p99 is a different animal

Average latency hides the requests that hurt. On a high-throughput fleet the p99 is dominated by queueing, cold caches, and the occasional slow peer.

The levers

• Connection reuse — TLS and TCP handshakes are pure overhead on a hot path.
• Request coalescing — identical in-flight reads should share one upstream call.
• Locality — route to the node nearest the caller, not the nearest to you.

None of these are exotic. Applied together they routinely halve p99.