Envoy with PGO and ThinLTO: 20% to 40% less CPU per request

Large C++ servers often gain a lot from profile-guided optimization (PGO), and Envoy is one of them. We rebuilt an unmodified Envoy with clang PGO and ThinLTO (LLVM’s scalable link-time optimization) and compared it with a standard optimized build of the same commit. On our HTTP/1 and HTTP/2 proxy benchmarks, CPU per request fell by 20% to 40%. Across 12 workload shapes at 2,000 requests per second it fell by 27% to 40%.

The official Envoy build is a sound default. PGO adds a training step, a profile to maintain and a binary that is harder to reproduce, and those are real costs for a release process. This post is for people who already build their own Envoy and pay for its CPU.

What we measured is narrow: a router-only proxy on three x86-64 Linux test machines, synthetic traffic, long-lived connections. We did not measure gRPC, HTTP filters other than the router, connection-heavy traffic or arm64. Expect a smaller gain where more of the time goes to the kernel or to cryptography.

The numbers

Each figure compares the PGO + ThinLTO build with bazel build -c opt of the same commit (Envoy main at 24a950b6, 1 November 2026). As a control we ran the official 1.40.2 release binary on three shape and load combinations. It matched our baseline within the noise of those runs (from 0.6% higher to 3.7% lower), and the PGO + ThinLTO build used 33% to 39% less CPU per request than the official binary.

Measure	Result	Machine	Rounds
CPU per request, 12 shapes, 2,000 requests/s	27% to 40% lower	AMD VM	4, all won
CPU per request, 8,000 requests/s, two shapes, 4 or 16 workers (4 comparisons)	31.6% to 40.4% lower	AMD VM	4 each
CPU per request, 16 workers, 32,000 requests/s	20% lower	AMD VM	4, all won
CPU per request at the highest sustained load	about 26% to 30% lower	AMD VM	2 per sweep
Maximum throughput, direct responses	33% to 39% higher	AMD VM	2
Maximum throughput, proxied	at least 4% higher; rig-limited, ceiling not measured	AMD VM	2
CPU cycles per request, client on the same host	37% to 43% lower	Core i9	3
CPU cycles per request, client on a second machine	33% to 35% lower	Core i9	3
Envoy’s energy per request above idle	3.9% to 12.9% lower	Xeon	4 to 6

The fixed-load rows ran 4 rounds each, and the PGO + ThinLTO build used less CPU in every round on record. One of the four 8,000 requests/s comparisons was a control run that kept only its average, so it has no per-round count. The Core i9 rows ran 2 workers over 3 rotated rounds. The saturation and throughput rows rest on 2 rounds per sweep, so read their sizes as approximate.

In the AMD VM runs p99 latency was also lower, by 2% to 16%. These runs were not designed to measure latency, so take that as a direction rather than a size. Time to ready with 1,000 routes was about 10% shorter with a warm page cache.

The 12 shapes, at 4 workers and 2,000 requests/s in total:

Shape	CPU per request
HTTP/1, 4 KB proxied response	40.1% lower
HTTP/1, header-heavy direct response	35.3% lower
HTTP/1, 16 KB proxied response	31.3% lower
HTTP/2, 4 KB proxied response	40.2% lower
HTTP/2, header-heavy direct response	26.8% lower
JSON access log, direct response *	31.6% lower
Text access log, proxied *	36.6% lower
1,000-route table *	39.7% lower
Response headers modelled on a real site, HTTP/2 (two sets) *	27.5% and 29.8% lower
TLS, header-heavy direct response *	31.9% lower
TLS, 4 KB proxied response *	39.9% lower

Shapes marked * were not in the training traffic. The two header sets use the header names and value lengths of two real sites, with random values. Read each row to about 3 points: one shape measured in two sessions gave 35.3% and 32.8%. Differences between rows of that size are not findings.

All of these figures come from a six-leg training profile (a leg is one training run with one protocol and traffic shape). The nine-leg mix we now recommend was checked on the Xeon, in cycles per request: 38.8% and 33.4% lower on the two HTTP/1 shapes, in 6 of 6 pairs, against 38.6% and 33.7% for the six-leg profile on that machine.

How we measured

Fixed load. Nighthawk sent a fixed, open-loop request rate with a pool of 16 connections per Envoy worker, so both builds served the same traffic. The pool size is a limit: see “Connection reuse” below for how the requests were spread over it. CPU per request is Envoy’s user plus system time divided by successful responses. (The author of this post is an emeritus maintainer of Nighthawk and its largest contributor, with more than 300 pull requests opened and about 280 merged.)
Separate cores. Envoy, Nighthawk and the origin were pinned to disjoint cores. In the AMD VM, Envoy had one core per worker as the VM presents it, both hardware threads. On same-host runs the three still share the host’s memory bandwidth.
Randomized or alternating order. AMD VM comparisons ran 4 rounds in shuffled order. Xeon comparisons ran 6 rounds in A-B-B-A order (4 for the two higher-load energy checks), and the PGO + ThinLTO build won 6 of 6 pairs for HTTP/3, churn and stale profiles. Order matters: in a different experiment, a change looked 3.6% worse when the baseline always ran first, and 0.7% the other way, inside the noise, when rerun in alternating order.
Connection reuse. These runs used Nighthawk’s default connection reuse, which sends most requests over the most recently used connections. We later found that this concentrates the load on some of Envoy’s workers, a different set in each run, and that CPU per request depends on how many workers are busy. Both builds ran under the same conditions, and the main effects are large and favour the PGO + ThinLTO build in every round, so they stand. Smaller figures in this post are less certain than their precision suggests, and the text says where. Our current method spreads the load over every connection and every worker; it is described in the price list post, and the harness is public. How we found the effect is in our account of the whole Envoy CPU effort.
Cycles where time misleads. On the Core i9, with power management active, a leaner binary lets the CPU frequency governor lower the clock, and CPU time barely moves. There we count CPU cycles per request.

Test machines

All three are x86-64 Linux.

Label	CPU	Setup	Used for
AMD VM	AMD Zen 2 workstation processor, one socket, 64 cores, 128 threads	Envoy, load generator and origin in one Hyper-V Linux virtual machine, Ubuntu 24.04, kernel 6.6	The 12 shapes, load and saturation sweeps, controls, tests, startup
Xeon	Two Intel Xeon E5-2673 v4 sockets, 20 cores and 40 threads each	Bare metal, Ubuntu 24.04, kernel 6.8, clock fixed at 2.3 GHz with turbo off; Envoy on one socket, load generator and origin on the other	HTTP/3, configuration churn, stale profiles, energy
Core i9	Intel Core i9-9880H, 8 cores, 16 threads, a 2019 laptop	Bare metal, Ubuntu 24.04, kernel 7.3; 1 GbE USB Ethernet (Realtek r8152). Load generator on the same host, or load generator and origin on a second machine: Intel Core i7-8750H, 6 cores, 12 threads, Ubuntu 22.04, 1 GbE (Realtek r8169), wired through a switch	Cycle counts at fixed load, with and without a real network card in the path

The USB Ethernet adapter on the Core i9 adds kernel network work that a rebuild of Envoy does not reduce, so the over-the-network figures are conservative.

Why a rebuild helps

Two hardware counters show where the gain comes from. On a 4 KB proxied response in the AMD VM, user-space instructions per request fell from 172k to 125k, and instructions per cycle rose from 0.50 to 1.00. On the Core i9, instructions per cycle rose from 0.39 to 0.68.

Our interpretation, which we have not confirmed with a top-down breakdown or instruction-cache counters: the standard binary has 68.6 MB of code, the hot paths are scattered through it, and the processor stalls fetching instructions. A profile lets the compiler place hot code together, move cold paths away and inline across translation units where it pays. The optimized binary has 65.3 MB of code.

User-space cycles per request fell from between 341k and 349k to between 121k and 124k on that shape. Kernel cycles per request did not change. That is why the total falls 40% and not 64%.

PGO alone gave between a little over half and three quarters of the combined gain on most shapes. ThinLTO without a profile moved CPU per request between 8% lower and 1% higher, which on 4 rounds is not clearly separated from noise.

Where the gain holds and where it shrinks

Under load. The saving was smaller at the highest load. With 16 workers configured and 32,000 requests/s in total, near the rig’s limit with p99 around 14 ms, it was 20%. With the connection reuse described above, not every worker carried load in those runs, so how much of the drop comes from load and how much from that concentration is not resolved. In two saturation sweeps on 4 worker cores, with 32 or 64 connections per worker, it was about 26% to 30%. Direct responses topped out at about 52,500 requests/s on the standard build and 70,000 to 73,000 on the optimized one. On the proxied path the standard build was CPU-bound at about 39,400. The optimized build reached about 41,000 using 3.1 of its 4 worker cores, because the rig’s upstream path became the limit, so the proxied CPU per request compares two slightly different throughputs.

On untrained shapes that use trained code. The starred shapes above improved by 27% to 40%. They are new configurations, but they run through the codec and router code the profile covered.

On an untrained protocol. HTTP/3 was not in the first training run. Cycles per request still fell: 24% on small proxied responses and 9% to 10% on header-heavy ones. That is about a quarter to two thirds of the HTTP/1 gain on the same machine. An earlier nine-leg profile, the six legs plus two HTTP/3 legs and one under configuration churn, cut HTTP/3 cycles a further 27% to 28%, and left HTTP/1 with no resolved change (within 1.5%). We did not run that build against the standard build directly. Chaining the two paired comparisons gives about 35% to 45% for trained HTTP/3.

Under configuration churn. Routes and clusters came from file-based xDS: about three route updates per second, each a full 205-route configuration, and three cluster updates per second. Cycles per request were 32% to 33% lower, against 34% to 39% without churn.

With a profile from another commit. We applied one profile to newer and older code, and compared each with a standard build of that code:

Code built	4 KB proxied	Header-heavy direct
Same commit as the profile	39.3% lower	33.7% lower
11 days and 219 commits newer than the profile	31.0% lower	26.3% lower
4 weeks older than the profile	32.0% lower	30.9% lower

Figures are total cycles per request on the Xeon; the PGO + ThinLTO build won 6 of 6 pairs in every cell. That keeps roughly four fifths or more of the same-commit gain (78% to 92%). The share is a ratio of two measured savings, so it is less precise than either. We have not tested a gap as large as a minor release.

Energy

Energy is a smaller saving than CPU, and it depends on load. On the dual-socket Xeon E5-2673 v4, at a fixed 2.3 GHz with turbo off, Envoy ran alone on one socket, so that socket’s RAPL package energy counter (Intel’s on-chip energy meter) measured Envoy’s load and little else.

Load	Energy per request above idle	CPU time per request
4,000 requests/s, four shape and worker combinations	3.9% to 9.0% lower	30% to 37% lower
12,000 requests/s, 4 KB proxied	12.2% lower	39% lower
20,000 requests/s, 4 KB proxied	12.9% lower	38% lower

The 4,000 requests/s figures are the least certain: at low load the number of awake workers also moves energy, and the connection reuse described above changed that from run to run. Counting idle draw and memory power, energy per request fell 3.4% to 3.9% at the two higher loads. At 4,000 requests/s the measured change was 0.8% to 2.1%, which one session cannot separate from noise. At that load the cores wake every few hundred microseconds and rarely reach deep sleep (on this machine one intermediate sleep state was disabled in firmware), so power appears to follow wakeups more than busy time. If CPU is what you provision for, the same traffic fits on fewer cores. Results vary by CPU.

What we tested

We did not run Envoy’s full test suite. The tests below ran at the measured commit on a PGO + ThinLTO build made with raw -flto=thin compiler flags. We ran the HTTP, router and TLS unit tests and three integration suites: 118 of 119 targets pass. The failure, MultipleRequestsWithTrailersDecodeHeadersEndsRequest in http2_integration_test, also fails on the standard build, in 8 of 40 runs. Eight suites pass under AddressSanitizer with ODR-violation detection plus UBSan’s vptr and function checks.

Costs and caveats

Build time. On Envoy v1.40.2, with 24 parallel jobs on a 128-thread host with other work running, a standard -c opt build took about 24 minutes. The instrumented build took about 45 minutes from a cold start, dependency downloads included. The optimized build took about 25 minutes. Treat these as indicative.
Training time. A smoke run takes under a minute. Our nine-leg mix takes about ten minutes. The instrumented binary is slow and is not for production.
Memory. The instrumented build peaked at about 21 GiB across all build processes. The optimized build peaked at about 20 GiB while compiling and 9 GiB in the ThinLTO link stage. This is with Bazel’s thin_lto feature. With raw -flto=thin compiler flags, ThinLTO runs inside one linker process, which peaked at 33 GiB.
Binary size. Stripped, the optimized binary is about 103 MB, against 107 MB for the standard build. Unstripped it is 206 MB with Envoy’s default split debug info. A 2.5 GB binary appears only if you combine -flto=thin compiler flags with --fission=no, which puts all debug info inline. An early build of ours did that, and its first start from a cold page cache took 344 ms, against about 225 ms for the standard build.
Debuggability. Expect deeper inlined frames and cold-split function fragments in stack traces. Keep the profile with the build. Profiles differ from run to run, so two people following the recipe get different binaries.
Retraining. The stale-profile data above covers code 11 days newer than the profile and code 4 weeks older, on two shapes. Plan to retrain on a schedule and when your traffic mix changes.
Untrained code. Code the training run never executes has zero counts, and the compiler may treat it as cold: optimized for size and moved away from hot code. Such code can get slower than in a standard build. We did not measure a filter left out of training. Train with every protocol and filter you run.
Stale profiles and correctness. By design, a mismatched profile changes optimization decisions only. We ran no tests on a build with a stale profile.

LLVM’s BOLT post-link optimizer on top of this showed no clear gain in a short test that could not resolve differences below about 5% to 8%. We ran it in instrumentation mode in the AMD VM; branch-record sampling on bare metal may do better.

Whether a project should ship binaries built this way is a separate question. Reproducible builds, turnaround for security releases, CI cost, per-architecture profiles and crash triage all weigh against it, and that decision belongs to Envoy’s maintainers.

Try it yourself

These commands were verified on Envoy v1.40.2 with its default hermetic clang 22, on a fresh clone with no local changes. Verified means the four steps build and the profile is used; we did not benchmark this exact build. In a separate comparison, the thin_lto feature form executed the same instructions per request as raw -flto=thin flags, within 0.2%. You need bazelisk and the host packages listed in Envoy’s bazel/README.md.

git clone --depth 1 --branch v1.40.2 https://github.com/envoyproxy/envoy
cd envoy

# 1. Instrumented build, then copy the binary out: step 4 overwrites bazel-bin.
bazel build -c opt //source/exe:envoy-static \
  --copt=-fprofile-generate --linkopt=-fprofile-generate
cp bazel-bin/source/exe/envoy-static ./envoy-instrumented

# 2. Train: run the instrumented binary under traffic that looks like yours,
#    then stop it cleanly so it writes the profile.
mkdir -p prof
LLVM_PROFILE_FILE="$PWD/prof/envoy-%p-%m.profraw" ./envoy-instrumented -c your-config.yaml &
#    ... send traffic ...
ADMIN_PORT=9901   # the admin port in your-config.yaml
curl -X POST "http://127.0.0.1:${ADMIN_PORT}/quitquitquit"
wait   # Envoy writes the profile as it exits; do not merge before it has

# 3. Merge with the llvm-profdata from Envoy's own hermetic toolchain (same LLVM as the compiler).
LLVM_PROFDATA=$(ls "$(bazel info output_base)"/external/*llvm_toolchain_llvm/bin/llvm-profdata)
"$LLVM_PROFDATA" merge -output="$PWD/envoy.profdata" prof/*.profraw

# 4. Optimized build: Bazel's thin_lto feature, default split debug info.
bazel build -c opt //source/exe:envoy-static \
  --fdo_optimize="$PWD/envoy.profdata" \
  --features=thin_lto \
  --ltobackendopt=-Wno-unused-command-line-argument
# Result: bazel-bin/source/exe/envoy-static

Keep the instrumented binary. Step 4 rebuilds the same path under bazel-bin, so copy the binary out after step 1.
Stop cleanly. The profile is written only on a clean exit. The profile holds function and branch counts, not request data.
Use the toolchain’s llvm-profdata. Envoy compiles with a clang that Bazel downloads, and the matching llvm-profdata sits in Bazel’s output base. A distro llvm-profdata from LLVM 18 rejects the raw profile with a format version error.
--fdo_optimize makes the profile a tracked build input, so Bazel’s caches stay correct when the profile changes. It takes an absolute path or a Bazel label. For a PGO-only build we checked that the binary is byte-identical to one built with raw -fprofile-use flags. Step 1 uses raw flags so the output path comes from LLVM_PROFILE_FILE.
--features=thin_lto needs --ltobackendopt=-Wno-unused-command-line-argument. Without it the build fails: the ThinLTO backend actions inherit two compile flags that clang reports as unused, and Envoy builds with -Werror. The option applies to the backend actions only.
A profile from another commit needs three more flags: --copt=-Wno-profile-instr-out-of-date --copt=-Wno-profile-instr-unprofiled --copt=-Wno-backend-plugin. Bazel’s FDO support adds only -fprofile-use, so Envoy’s -Werror still applies to profile mismatch warnings. -Wno-backend-plugin silences every backend diagnostic; there is no narrower flag.

The recipe is also on our Envoy fork, on the commit we measured, as three branches:

build presets: --config=pgo-instrument and --config=thinlto in .bazelrc, with the flags above;
a documentation page on building with PGO and ThinLTO;
a training kit under tools/pgo/ that drives the nine-leg mix with Nighthawk and merges the profile.

pgo-recipe-v1 has all three in one commit.

Credit

None of the machinery is ours. PGO and ThinLTO are LLVM’s work, and --fdo_optimize and the thin_lto feature are Bazel’s. We measured what they do for Envoy. We used instrumented training runs; sampled profiles from production traffic are the common alternative, and we have not tried them.

Reproductions

If you reproduce this on your own traffic, run the two builds in alternating order and post what you get as an issue on the A/B harness. Results on arm64, gRPC, filter-heavy configurations and production profiles would fill the gaps listed at the top, and a result that comes out worse is as useful as one that comes out better.