Envoy CPU per request: what held up, what we dropped
By Otto van der Schaaf, We-Amp ·
We asked one question of Envoy: where does it spend CPU on each request, and how much of that can be cut? We kept asking it, each time with a sharper instrument, because the earlier answers were less reliable than they looked.
The scope is narrow. One proxy, x86-64 Linux, synthetic traffic. The short answer: a PGO and ThinLTO rebuild cut CPU per request by 20% to 40%, some settings cost more CPU than the code changes we kept saved, most code changes were small or could not be told from noise, and our method changed twice along the way. The code changes in this post are patches on our own fork: none has been proposed to the Envoy project or reviewed by its maintainers. The detailed numbers are in the PGO and ThinLTO post and the price list.
Contents
- How did we measure Envoy’s CPU per request?
- What does a large route table cost per request?
- Can constant headers be cheaper?
- What cut Envoy’s CPU the most?
- What do Envoy settings cost?
- What did not hold up?
- How do we measure now?
- Where is the code?
How did we measure Envoy’s CPU per request?
We count user-space instructions per request first and CPU cycles per request second. An instruction count barely moves between runs, so it shows whether a change removed work. Cycles show whether that work was what the request was waiting on, but they are noisy, and a good part of this post is about how noisy.
Each comparison in the route and header work ran two or three builds against the same fixed, open-loop request rate from Nighthawk, in alternating order, over several rounds, with Envoy’s process counted alone by perf stat. (The author of this post is an emeritus maintainer of Nighthawk.) For those we report the mean and how many rounds went each way.
The route and header work below ran on a bare-metal test machine with two Intel Xeon E5-2673 v4 sockets, clock fixed at 2.3 GHz, Envoy’s 4 workers on one socket and the load generator on the other. The builds are -c opt builds of Envoy main at 8ce942ce5 (13 November 2026), with and without each change.
What does a large route table cost per request?
With a routes list, a request that matches the last of 1,000 routes ran 408.8k user-space instructions, against 75.3k with a single route: about 334 instructions for every route the scan passes over. Checking the path before the other match criteria cut that to 288.6k, 29.4% fewer, without changing which route is selected.
A routes list promises the first match in configured order, and a linear scan is the direct way to keep that promise. For large tables Envoy offers the matcher tree (VirtualHost.matcher), which in Envoy’s own benchmark stays flat at about 1.2 µs per iteration, building the request included, from 1 to 16,384 routes. The question here is what a routes list costs at a size where the matcher tree is the better fit, and how much of that is recoverable without changing its semantics.
The test table is one virtual host of direct responses. The first N−1 routes are drawn at random: 60% distinct prefixes, 20% exact paths and 20% regexes, 30% of them with a header matcher the client never sends. The last route matches every request, so the scan visits the whole table. That is the worst case. An earlier match costs less, and saves less.
| Routes | Unmodified | Path first | Change, 8 rounds |
|---|---|---|---|
| 1 | 75.3k | n/a | |
| 200 | 141.5k | 117.4k | 17.0% fewer, 8 of 8 |
| 1,000 | 408.8k | 288.6k | 29.4% fewer, 8 of 8 |
User-space instructions per request. The counts were the same at a light and a heavier load to within 0.3k, and they move only about 0.2% with how the load is spread over workers. The scan’s cycle and latency figures came from runs under the old connection reuse described below, so we leave them out.
Checking the path first
At that commit, for prefix, exact-path and path-separated-prefix routes, matches() calls matchRoute() first, which runs through the runtime fraction, gRPC, header, query parameter, cookie, TLS, metadata and filter-state criteria and stops at the first that fails, and only then compares the path. In a long table almost every route fails on the path. The change swaps the operands of the conjunction (for path-separated prefixes, the same reorder written as an early return):
// before
if (RouteEntryImplBase::matchRoute(route_match_context, stream_info, random_value) &&
path_matcher_->matchPathWithoutQuery(route_match_context.sanitizedPathWithoutQuery())) {
// after
if (path_matcher_->matchPathWithoutQuery(route_match_context.sanitizedPathWithoutQuery()) &&
RouteEntryImplBase::matchRoute(route_match_context, stream_info, random_value)) {Neither side changes state the other reads: the sanitized path is computed once and cached on the match context. So the selected route and first-match order stay the same. The one case that costs more: a route whose path matches but whose other criteria reject the request now evaluates both conditions.
Leaving regex routes alone
An earlier revision reordered regex routes too. A benchmark we wrote to check it, regex routes that each require a header the request does not send, ran 2.1 to 4.1 times slower than main in every round: with the pattern first, every route runs its regex even when a header check would have rejected it for less. The current change keeps regex and URI-template routes in the original order, which on our table costs 7.6% more instructions than the earlier revision (288.6k against 268.1k). We accepted that: the loss when the order is wrong is much larger than the gain when it is right.
A prototype index
We also built a prototype that indexes a virtual host’s routes when the configuration loads, keeping first-match order. In microbenchmarks it is slower than path first for virtual hosts with 2 to 4 routes (8% to 32%) and for regex routes at every size from 2 routes up (3.6% to 38.8%), and we have not measured its memory or configuration-load cost. Whether a second mechanism beside the matcher tree is wanted is not our call. It stays a prototype, and its code is not among the branches linked below.
Can constant headers be cheaper?
Yes, by about 300 instructions per constant header. On a route that adds 109 constant headers, the change ran 32.8k fewer instructions per request (165.9k to 133.1k), lower in 8 of 8 rounds. In microbenchmarks the constant path is 24% to 32% faster. In cycles the saving was too small for our setup to resolve.
At that commit, Envoy’s header parser, which the router and several other components use to apply configured headers, stages each configured value as an owned string and copies it into the header map. Copying was the safe default under the header map’s lifetime rules. A constant value, though, sits in the same configuration entry as its key, and the key was already passed by reference, so referencing the value adds no new lifetime requirement. Values produced by a formatter keep the copy. The router tests pass under AddressSanitizer; we have not run the other users’ tests under it.
What cut Envoy’s CPU the most?
A rebuild, not a code change. An unmodified Envoy built with clang’s profile-guided optimization and ThinLTO used 20% to 40% less CPU per request than a standard optimized build of the same commit, on HTTP/1 and HTTP/2 proxy benchmarks. The PGO and ThinLTO post has the numbers, the costs and the recipe. Those runs used the old connection reuse described below, but the effect is large and went the same way in every round, so it stands.
On one proxied shape, user-space instructions per request fell by about a quarter and instructions per cycle rose from 0.50 to 1.00. Our reading, which we have not confirmed with a top-down breakdown, is that the standard build stalls on instruction fetch and the profile lets the compiler place hot code together. If so, that is the contrast with the route work: the route change removes instructions, and this one also makes the instructions that remain cheaper to run.
What do Envoy settings cost?
When Envoy’s CPU usage is high, settings can explain more of it than code. On one bare-metal machine, an external authorization check added about 55% CPU per request with an HTTP service and about 81% with gRPC (both lower bounds), and 100% trace sampling added 34%. The price list gives every row with its conditions and 95% interval.
Its harness and raw data are public at github.com/We-Amp/envoy-price-list, and it is where our current method first ran in full.
What did not hold up?
Several results we believed for a while turned out to be wrong, and one flaw sat in our own method. All were caught before publication. The connection-reuse flaw also affected the runs behind the PGO post, which says so. For each: the error, how we caught it, and the rule it left behind.
The connection reuse in our load generator. Nighthawk’s default reuses the most recently used connections, so 2 to 4 of Envoy’s 4 workers carried the load, a different set in every run. Cycles per request followed the busiest worker’s share: the more one worker carried, the cheaper a request looked (correlation −0.80 to −0.87 on two request shapes). Spreading requests over every connection cut the standard deviation of paired differences from 6.9% to 1.3% on one shape and from 6.5% to 1.9% on the other. Large effects and instruction counts stood; smaller cycle results from those runs were set aside. Rule: spread the load, and fail any run that one worker dominates.
The wrong binary. A later build repointed Bazel’s bazel-bin symlink, so three early verdicts rested on builds without optimization. Their apparent gains of 37% to 89% were void. Rule: copy every binary out, prove it is optimized and record its hash.
Stale symbols. Envoy’s build ID is its git commit, so perf served cached symbols from a different build of the same commit. Rule: clear the symbol cache between builds.
An energy saving. We reported a 17% to 25% energy saving from fewer workers, then withdrew it: the load generator’s idle loop was spinning on the same processor package. Rule: a sleeping client, and Envoy alone on its socket.
An A/A offset that was chance. Same-binary runs seemed to show a small bias, at a z of about 2.7. It came from leaving one run out and pooling t values as if they were z values. A run with the labels swapped settled it as chance. Rule: pool only like with like, and check any apparent bias with a label swap.
Our own regression. The regex reorder above, caught by its own benchmark. Rule: benchmark the case a change can make worse, not only the one it improves.
Misrouting in the index. The prototype’s first way of taking a literal prefix from a regex misrouted on alternation and on quantified literals, and our own review found two more defects, in case-insensitive exact routes and in how regex bytes were encoded. The fix takes the prefix from RE2’s PossibleMatchRange() and adds differential tests against the linear scan. Rule: an index is tested against the scan it replaces.
Ideas we dropped. “Not resolved” means the CPU change was inside the noise of those runs, not that there was no effect.
| Idea | The change | Verdict |
|---|---|---|
| TLS write batching | halved writes per request | CPU change not resolved |
| One-pass HTTP/1 header encode | fewer passes over headers | not resolved |
| Deferred flush | no change in writes; Envoy already coalesces | not resolved |
| Stream free lists | reused request objects | not resolved |
| Larger buffer slices | larger slices for bodies | not resolved |
| BOLT after PGO | post-link layout | not resolved in a short test |
| io_uring socket interface | different I/O path | more CPU on both machines tried; few rounds, direction only |
| tcmalloc per-CPU caches | 10% to 19% fewer instructions | no measurable change in CPU |
How do we measure now?
With the load spread over every worker, paired runs, an A/A floor and a gate that voids any run it cannot trust. Each rule is a lesson from the section above:
- Spread the load. Requests rotate over every connection, and a run that one worker dominates fails a validity check.
- Paired, balanced and repeated. 12 rounds or more in balanced order unless stated otherwise, with a 95% interval on the mean of paired ratios.
- An A/A floor. The same binary against itself in the same session sets the smallest difference worth reporting. Across our A/A runs a bias above about 0.2% is excluded, and sessions differ by about 0.34% on top of each run’s own interval.
- A gate that fails closed. Missed rates, client-side queuing, busy cores or shared hardware counters void the run.
- Instructions first. Cycles confirm; under about 1% from one session we claim nothing.
- Build proof. Optimized binaries only, with their hashes recorded.
The route and header counts above predate the full method: 8 rounds, instructions only.
Where is the code?
The changes are patches on our Envoy fork, on the branches listed below. They have not been proposed to the Envoy project and no Envoy maintainer has reviewed them. Any error in them is ours.
patches/route-match-path-first-v4: path first, against8ce942ce5, with a semantic test and three benchmarks.patches/router-header-reference-v3: constant headers by reference, against8ce942ce5, with a unit test of value lifetime that also passes without the change.pgo-recipe-v1: the PGO and ThinLTO recipe in one commit, also split aspgo-recipe-1-bazelrc,pgo-recipe-2-docsandpgo-recipe-3-trainer.exp/upstream-pool-lru: an experiment against v1.40.2, switched by an environment variable, priced in the price list, with no test.
Envoy’s route matching, header code, matcher tree and benchmarks are the work of its contributors; PGO and ThinLTO are LLVM’s. Issues are off on the fork; corrections go to the harness’s issue tracker.