What Envoy settings and releases cost in CPU: a measured price list
By Otto van der Schaaf, We-Amp ·
Turning on tracing, adding an authorization check or switching certificate type all change what a proxy costs to run. The size of that change is rarely written down, and when it is, the conditions behind it are missing. This is a price list for Envoy: what a setting, a feature or a new release costs in CPU per request, measured the same way every time, with the conditions next to every number.
This list prices nine common settings and a second proxy hop, compares three official releases, measures what surplus worker threads cost in a container pod, and prices one experimental change to how the upstream connection pool picks a connection. It is one machine, one load level per row and loopback networking. Read every figure as a price on that rig, not as a promise for yours.
Among the larger prices on this rig: an external authorization check adds about 55% CPU per request with an HTTP service and about 81% with a gRPC service (both lower bounds), 100% trace sampling adds 34%, and an ECDSA certificate or session resumption saves 17% to 18% when clients reconnect every 10 requests.
How every number is measured
- Paired A/B, balanced order. The two configurations (the baseline and one changed setting) run alternately in balanced order, 12 rounds or more per comparison unless the row says otherwise: A-B-B-A for two arms, rotated for three or more. Each round gives one paired ratio. The figure is the mean of those ratios as a percent change, with its 95% interval. Where a table compares several rows at once, the significance test is corrected for the number of rows.
- Fixed load. An open-loop load generator sends the same request rate to both arms, so both serve the same traffic. Rates were chosen so the baseline proxy is about 25% to 33% busy on its four workers; a row measured at another load level says so.
- CPU is cycles on the proxy process. Hardware counters count user plus kernel cycles on the proxy alone and divide them by the responses it served. Client, origin and side services are not included. The worker-thread section is the one exception: the proxy runs in a pod there, and CPU is the pod’s CPU time from the kernel’s accounting, divided by the responses it served.
- A validity gate that fails closed. Every run must reach its rate within 1%, have no request rejected by the load generator’s own queue limits (client overflow), and keep at least 25% headroom in the load generator. The proxy’s cores must be free of other work before and during the measurement window, the load must be spread across workers, and each hardware counter must run for the whole window rather than share time with others. From the settings runs onward the processor’s idle-state settings must also match at the start and the end of every run; the release comparison was run before that check existed and records the idle states once. The worker-thread section has its own checks, listed there. A run that fails any check is not counted. Where a check was switched off for a row, the row says so.
- An A/A floor. The same binary and configuration run against themselves on the same machine set the size of difference the setup cannot tell from noise: about 1.2% on HTTP/1 proxied responses in the settings session (about 1.7% in the earlier release session) and about 1.9% on HTTP/2. Across all A/A runs pooled, a systematic bias larger than about 0.2% is excluded. What remains is a spread between sessions of about 0.34% (one standard deviation) on top of each run’s own interval.
- What that means for small numbers. An effect under about 1% from one run is not a finding, whatever its own interval says. An effect between 1% and 2% needs a second session, or an interval that still excludes zero once the between-session spread is added. Every line below says which case it is.
The harness and the raw data behind every row are public: github.com/We-Amp/envoy-price-list (code under Apache 2.0, data under CC BY 4.0). Why the method looks like this, and the flaw in our earlier runs that led to it, is in our account of the whole Envoy CPU effort.
Settings
Common conditions for every line in this section:
- the official Envoy v1.40.2 release binary, 4 workers;
- a bare-metal dual-socket Intel Xeon E5-2673 v4 server, fixed 2.3 GHz, turbo off, Ubuntu 24.04 with kernel 6.8. The proxy is pinned to four cores of one socket with their hyperthread siblings left idle, and the load generator and origin run on the other socket;
- the processor’s default idle states;
- client and origin on loopback, HTTP/1.1 unless the line says otherwise; the origin is an nginx server;
- the baseline arm about 30% busy. Costlier settings ran at up to 55% busy at the same request rate. On this machine busier cores cost less per request, so large costs are slightly understated;
- latency is not part of this list unless a line says so.
Kernel cycles on loopback include part of the client’s and origin’s receive work. That is fair for comparing arms. It is not what a deployment behind a network card pays.
| Setting (against the baseline) | Traffic | CPU per request [95% interval] |
|---|---|---|
| ECDSA P-256 certificate instead of RSA-2048 | TLS 1.3, reconnect every 10 requests, 2,800 req/s | −17.9% [−18.1, −17.7] |
| ECDSA P-256 instead of RSA-2048 | TLS 1.3, keep-alive, 4,800 req/s | no resolved difference |
| TLS session resumption on, RSA-2048 | TLS 1.3, reconnect every 10 requests, 2,800 req/s | −17.4% [−17.7, −17.1] |
| HPACK dynamic table size 0 instead of 4,096 | HTTP/2, header-heavy direct response, 8,000 req/s | about +7% [+5, +9] |
| OpenTelemetry tracer, 0% sampling | 4 KB proxied, 5,200 req/s | +6.7% [+6.2, +7.3] |
| OpenTelemetry tracer, 1% sampling | same | +7.0% [+6.5, +7.5] |
| OpenTelemetry tracer, 100% sampling | same | +33.8% [+33.2, +34.3] |
| File access log, default text format | same | +3.6% [+2.7, +4.5] |
| File access log, JSON | same | +4.5% [+3.8, +5.2] |
| External authorization, HTTP service | same | +54.5% [+53.4, +55.7], a lower bound |
| External authorization, gRPC service | same | +80.7% [+79.7, +81.7], a lower bound |
| Retry policy configured, no retries triggered | same | +1.5% [+0.9, +2.2], near the floor |
| Per-connection buffer limit of 4 KiB | 16 KB proxied, 5,440 req/s | +20.6% [+20.0, +21.1] |
| tcmalloc per-CPU caches on | 4 KB proxied and header-heavy direct | no measurable change |
| A second proxy hop, both proxies evenly loaded | 4 KB proxied, 5,200 req/s | +99.6% [+98.9, +100.3] for the pair |
The details and the conditions specific to each line follow.
TLS certificate: ECDSA P-256 instead of RSA-2048
About 1.7M cycles (≈0.78 ms of CPU on this machine) less per full TLS 1.3 handshake. With a new connection every 10 requests, reconnects spread out in time (about 16 client connections at a time) and no session resumption, that is −17.9% CPU per request [−18.1, −17.7] (2,800 req/s, 4 KB responses). An earlier run in which all connections reconnected at the same moment gave −18.0%. There is no resolved difference on keep-alive connections (−0.7% [−1.5, +0.2], 4,800 req/s). Negotiated on every handshake: TLS 1.3, AES-128-GCM, key exchange X25519MLKEM768.
TLS session resumption on (against off)
About 1.7M cycles (≈0.76 ms of CPU on this machine) less per resumed handshake with an RSA-2048 certificate. With a new TLS 1.3 connection every 10 requests, reconnects spread out in time (about 16 client connections at a time) and every reconnect resumed, that is −17.4% CPU per request [−17.7, −17.1] (2,800 req/s, 4 KB responses). An earlier run with reconnects at the same moment gave −18.6%. A resumed handshake with an RSA-2048 certificate costs about the same as a full handshake with an ECDSA P-256 certificate. Negotiated parameters as for the certificate line.
For the TLS lines the client’s queue checks were relaxed. In the reconnecting shape, every request that opens a new connection waits for its handshake, so the count of waiting requests equals the count of new connections and that check cannot fail; client overflow stayed checked and was zero in every run. In the keep-alive shape every client connection does a full handshake at once at start-up, before the measurement window. Requests then queue in the client: with RSA certificates about 900 to 1,200 overflow events and up to about 1,000 waiting requests per run, with ECDSA no overflow and up to about 300 waiting requests. That start-up overflow was not counted, and the waiting count was bounded instead. No request was dropped: the counters that mean a drop stayed checked and at zero, the rate at the proxy met its target in every run, and the client kept 75% to 77% headroom. The certificates are self-signed leaves with no intermediate, and every handshake negotiated TLS 1.3.
HTTP/2 HPACK dynamic table off (size 0, against the default of 4,096)
About +7% CPU per request [+5, +9] on header-heavy HTTP/2 direct responses (no upstream): 16 request headers from the client and 52 response headers, identical on every request, 16 connections, 8,000 req/s. Identical headers are the case where the dynamic table helps most. The HTTP/2 A/A floor, measured in a later session on the same machine, binary and configuration, is about 1.9%. Larger table sizes were not measured.
Tracing: OpenTelemetry tracer, gRPC export to a local collector
Against no tracer: +6.7% [+6.2, +7.3] at 0% sampling, +7.0% [+6.5, +7.5] at 1%, +33.8% [+33.2, +34.3] at 100%. At 0% the proxy still builds a span and propagates trace context for every request; only the export is skipped. No sampler extension of this tracer in this release avoids building the span: the sampler runs after the span has been created. Measured on 4 KB proxied responses at 5,200 req/s. The collector’s own CPU is not included.
File access log
+3.6% CPU per request [+2.7, +4.5] in the default text format and +4.5% [+3.8, +5.2] as JSON with the 15 standard fields. Measured on 4 KB proxied responses at 5,200 req/s, one line per request, including the log flush thread.
External authorization: one check per request, always allowed
With an HTTP service, about 285k cycles (≈130 µs of CPU on this machine) per check, +54.5% CPU per request [+53.4, +55.7]. With a gRPC service, called through the proxy’s own gRPC client over HTTP/2, about 424k cycles (≈190 µs) per check, +80.7% [+79.7, +81.7]. Both on a 4 KB proxied response at 5,200 req/s, with connections to the service kept alive and the request body not sent to the service. About four fifths of the added cycles are user-space work in the proxy. Both ran busier than the reference at the same request rate (47% and 55% against 31%), and on this machine busier cores cost less per request, so read both figures as lower bounds. The service’s own CPU is not included.
Retry policy configured, no retries triggered
About +1% to 2% CPU per request (+1.5% [+0.9, +2.2]; instructions +1.2%), 4 KB proxied responses, 5,200 req/s. One session. With the between-session spread added the interval is [+0.6, +2.4], and cycles were higher in all 12 rounds, so the direction holds. The size is close to the smallest effect this setup resolves.
Per-connection buffer limit of 4 KiB on the listener and the upstream cluster (against the default of 1 MiB)
+20.6% CPU per request [+20.0, +21.1] on 16 KB proxied responses, 5,440 req/s. The cost is mostly kernel time and comes from flow control on bodies larger than the limit: the proxy pauses reading from the upstream about twice per response. There is no measurable cost on 2 KB responses, which fit within the limit (−0.5% [−1.3, +0.3], 5,200 req/s). One limit and one body size were measured.
tcmalloc per-CPU caches
With the official binary on glibc 2.35 or later they are off. The caches need tcmalloc to register restartable sequences (rseq) for each thread, and glibc registers rseq for every thread first, so tcmalloc’s registration fails. Setting GLIBC_TUNABLES=glibc.pthread.rseq=0 stops glibc’s registration and turns the caches on (measured on glibc 2.39). Instructions per request fall 10% (4 KB proxied) and 19% (header-heavy direct response). CPU per request does not change measurably: −0.9% [−1.7, −0.1] and +1.6% [+0.3, +2.9], neither significant once corrected for the number of rows. The setting disables glibc’s own use of restartable sequences for the process and its children.
A second proxy hop (client → proxy → proxy → origin, plaintext HTTP/1.1)
About double. With both proxies evenly loaded at about 30% busy, the pair costs +99.6% proxy CPU per request [+98.9, +100.3] against one proxy. Per hop there is no measurable difference from a single proxy (−0.2% [−0.6, +0.1]). 4 KB proxied responses at 5,200 req/s. Both proxies ran on the same socket.
To load the second proxy evenly, every proxy in this measurement, the single-proxy reference included, used one upstream connection pool per downstream connection (connection_pool_per_downstream_connection). That setting costs nothing measurable on one hop (−0.1% [−0.6, +0.4]). It is a measurement device here, not a recommendation: it stops upstream connections from being shared and keeps them open after the downstream connection has closed, until the idle timeout.
With the default shared pool and a fast origin, few requests are in flight at once. The first proxy then reuses a few connections towards the second, and one worker of the second proxy carries most of its load (40% to 88% of it here, at 52% to 94% of a core, against 32% when evenly loaded). That chain cost less than the evenly loaded pair, +84% to +90% against +99.6%, in two runs. Its headroom is set by that one worker instead of four. For that chain the check for evenly spread load was off on the second proxy.
Releases
Three official release binaries, each checked against its published digest: v1.32.13, v1.36.13 and v1.40.2. Same machine, pinning and idle states as the settings above; 4 workers evenly loaded, about 26% to 33% busy; HTTP/1.1 over loopback, no TLS. Three shapes: 4 KB proxied at 5,200 req/s, 16 KB proxied at 5,440 req/s, and a 128-byte direct response with 57 request and 52 response headers added by the route at 8,000 req/s. 12 rounds in rotated order, with the test corrected across the 9 comparisons.
| Shape | v1.36.13 against v1.32.13 | v1.40.2 against v1.32.13 |
|---|---|---|
| Header-heavy direct response, CPU per request | −4.6% [−5.6, −3.7] | −6.3% [−7.4, −5.2] |
| 4 KB proxied, CPU per request | about equal | +1.2% [+0.6, +1.8] |
| 16 KB proxied, CPU per request | not resolved | +1.3% [+0.9, +1.7] |
| Proxied, user-space instructions per request | about −9.5% | about −12% |
| Header-heavy, user-space instructions per request | −17.9% | −28.7% |
- The header-heavy direct response got 6.3% cheaper by v1.40.2, in all 12 rounds, more than four times the half-width of the 95% interval that session’s A/A run gave for this shape (1.4%). About three quarters of that was already in v1.36.13.
- Instructions per request fell a lot, cycles much less. From v1.32.13 to v1.40.2 the proxied shapes run 12% fewer user-space instructions per request, measured to a few hundredths of a percent. User-space instructions per cycle fell with them (0.55 to 0.48 on 4 KB proxied), so fewer instructions did not mean fewer cycles. Across releases, an instruction count is not a stand-in for cost.
- On proxied requests v1.40.2 cost about 1.2% to 1.3% more than v1.32.13. That is from one session on one machine at one load level, and it sits in the 1% to 2% range that needs a second session. The 16 KB figure is three times the half-width of the 95% interval that session’s A/A run gave for that shape (0.4%). The 4 KB figure on its own is no larger than the offset that session’s A/A could not exclude (its interval reached 1.7%); it is listed because the 16 KB row agrees with it. Each release is one binary, and a difference of this size at equal work is what code layout alone can produce. The comparison is between the official binaries, not between the sources. The cause was not measured. The binaries grew from 82.6 to 106.9 MB between these releases.
On this evidence the list says nothing about whether proxied traffic got cheaper across releases. That would take a second load level and a second machine.
Surplus worker threads in a container pod
Conditions: a single-node Kubernetes cluster on the same bare-metal machine, the node confined to one socket (20 cores, 40 hardware threads), the official v1.40.2 container image, a pod requesting 2 CPUs with no CPU limit. Traffic is HTTP/1.1 over 512 client connections to a 2-byte direct response, with no upstream. The client reaches the pod through the container bridge and a host port, so the absolute CPU per request includes that kernel work. CPU in this section is the pod’s CPU time from the kernel’s accounting, not cycles. Each run had to reach its rate within 1% at the client and at the proxy, with zero client overflow, at least 25% client headroom and no other work on the client’s CPUs. The two-worker pod’s workers were about 17% busy at 2,000 requests per second and about 40% busy at 8,000. The kernel spread the connections over the workers; no connection balancer was configured. 12 paired rounds in balanced order unless stated.
A proxy pod that requests CPU but sets no CPU limit starts one worker thread per CPU the node lets it use, not one per requested CPU. On a 40-CPU node, a pod requesting 2 CPUs started 40 workers.
Each surplus worker costs CPU while idle (about 0.4 ms of CPU per second with the machine’s default idle settings) and about 70 KB of memory. At low load that idle cost is the whole difference: +5% CPU per request at 2,000 requests per second (95% interval +3% to +8%), with no latency difference.
At higher load, most of what the surplus workers cost on this machine came from its idle-state settings. At 8,000 requests per second over 512 connections, so that every worker took traffic, CPU per request was about 44% higher and median latency 43% to 46% higher than with two workers, in two sessions, with the machine’s default idle settings. Both pods ran almost the same instructions per request (the 40-worker pod about 1% more). The 40-worker pod ran them more slowly: between requests its cores entered the deepest idle state several times as often. With every idle state deeper than the shallowest one disabled on the pod’s CPUs (4 paired rounds per condition), the difference fell to about 8% (95% interval +3% to +13%) in CPU per request and about 3% (+0.5% to +6%) in median latency. Disabling those states also keeps the processor package out of its deep idle states, so the two are not separated here. Power was not measured, and this is not advice to disable idle states. Measured on one machine with one Envoy release.
The percentages are against a near-empty request. The absolute penalty on this machine, about 45 µs of CPU per request with default idle settings and about 7 µs with deep idle states off, would be a smaller share of a request that does TLS, routes to an upstream and runs real filters.
Setting --concurrency to the CPU request, setting a CPU limit, or giving the pod dedicated CPUs brings the count down.
Upstream connection reuse order (experimental)
This row prices a code change, not a setting: an experimental patch to v1.40.2 (branch exp/upstream-pool-lru of We-Amp’s Envoy fork) that makes the upstream HTTP/1 connection pool reuse the least recently freed connection instead of the most recently freed one.
Conditions: the patched v1.40.2 build (one binary, the order chosen per process); 4 workers per proxy; the same bare-metal machine, clock and default idle states as above; loopback; plaintext HTTP/1.1; 4 KB proxied responses; CPU as total cycles of the proxy processes; exact connection balancing (exact_balance) on both listeners, so each new connection goes to the worker with the fewest. The order was switched on the first proxy only. Nothing closed a connection during the main runs, so a peer that closes idle connections, the usual risk of reusing the least recently used connection, was not exercised. The check for evenly spread load was off for the second proxy, whose spread is the result. The patched build has no A/A run of its own.
Measured with one proxy forwarding HTTP/1.1 to a second proxy. The effect depends on how much larger the connection pool between them is than the load needs.
Pool much larger than the load needs (512 connections left by one traffic spike, a handful in use). With the default order the first proxy sent nearly all requests over a few connections, and the second proxy’s workers shared the load unevenly: the busiest worker did 36% to 100% of the work, differently on every start, and in some runs reached a full core while that proxy as a whole was 30% to 55% busy. With least-recently-freed reuse the busiest worker did 25% to 27% of the work in every run. The price was +3.4% [+2.0, +4.8] CPU per request for the two proxies together at about 30% busy and +1.4% [+0.2, +2.6] at about 42% busy (repeat run: +3.6% [+2.2, +5.0] and +1.7% [+0.5, +3.0]). The two figures at 42% busy are in the 1% to 2% range: they have the same sign in both runs, and with the between-session spread added their intervals are [+0.1, +2.7] and [+0.4, +3.1].
Pool grown with the load (18 to 48 connections). The default order was already close to even: the busiest worker did 26% to 39% of the work and none came near a full core. Least-recently-freed reuse narrowed that to 25% to 29% and made latency a few percent worse. It did not change CPU per request by an amount this setup resolves: +0.6% [+0.1, +1.2] at about 30% busy and +0.1% [−0.4, +0.6] at about 42% busy, both under 1%, and with the between-session spread added both intervals include zero ([−0.2, +1.5] and [−0.7, +0.9]). With a limit of 1,000 requests per connection on the second proxy, the two orders spread the load equally (busiest worker 26% to 28%).
On a proxy whose upstream is a web server rather than another proxy, the change cost nothing measurable in CPU (+0.1% [−0.5, +0.7] at about 30% busy and +0.3% [−0.5, +1.1] at about 42% busy). All figures were measured on loopback; their size on a real network is not known. Least-recently-freed reuse keeps every pooled connection in use, so the pool does not shrink after a spike.
After a brief stall of the second proxy under steady traffic. The second proxy was paused for 40 ms, and the first proxy’s pool grew from about 30 to about 500 connections. Additional conditions: one rate (about 30% busy), one stall, two minutes observed after it, one session.
With the default order, the second proxy’s busiest worker then did 54% of the work (34% to 66% across runs, against 34% before the stall) for the full two minutes, and the pool stayed at its peak. Least-recently-freed reuse kept the load even throughout (busiest worker 25% to 26%). It cost about +1.2% CPU per request for the two proxies together [+0.3, +2.2], with the interval widened by the between-session spread. That is less than with the pool left by a start-up spike above.
Three existing settings on the first proxy compare as follows:
max_connection_durationof 30 s brought the load back to the level of a pool that fits the load (busiest worker 28% to 33%) 30 s after the stall in every run, and shrank the pool. Over the same two minutes the two proxies used about 2.5% more CPU per request than with the default order [+1.5, +3.5], with the interval widened by the between-session spread. That is a comparison over this window, not a steady price of the setting.max_requests_per_connectionof 1,000 took 80 to 100 s, with one worker briefly near a full core in some runs.idle_timeoutof 5 s shrank the pool within seconds, but the load evened out only slowly, and not in every run.
All figures are from one stall size, one rate and one value per setting, measured on loopback.
Measured but not published
These were measured, but the measurements do not hold up well enough to publish, so they carry no number here:
- tail latency on the TLS rows, which measured the client’s reconnect waves rather than the proxy;
- latency through two proxy hops, and how the cost of a second hop splits between the two proxies;
- maximum sustained rates for the two-hop chains and for the connection reuse order;
- latency for the external authorization rows, which includes the service’s response time;
- CPU figures for the request-cap and idle-timeout settings after a stall, and latency figures for the connection reuse order beyond the direction given above.
What this does not cover
- One machine, a 2016 server CPU. One load level per row, one release for the settings, 4 workers and loopback only, except in the worker section.
- Client-side CPU, and the CPU of the collector and the authorization service.
- With deep idle states this machine charges more per request when load is spread thin (see the worker section), so every line names the idle-state setting. No settings row was repeated with idle states capped.
- TLS 1.2, other ciphers and certificate chains with an intermediate.
- gRPC proxying, HTTP/3, arm64, and connection-heavy traffic beyond the two TLS reconnect shapes.
If one of these figures does not match what you see, compare the conditions first.