The microbenchmark told me to do the wrong thing

I have been building a proof-of-concept plugin system on the WebAssembly Component Model: plugins compile to .wasm once, an Axum host loads them at startup, and each one declares its HTTP, WebSocket and SSE routes through an OpenAPI spec. They run in a sandboxed Wasmtime store with fuel limits, and they can query their own tables through Diesel across the WASM boundary.

The interesting part was not getting it working. It was a performance change I made, measured, and got backwards.

The setting

Wasmtime's pooling allocator can keep a slab of a plugin's linear memory resident between instantiations, instead of returning it to the OS. The knob is linear_memory_keep_resident. Keeping memory warm sounds obviously good: the next instantiation reuses a page that is already mapped and already zeroed.

I set it to 2 MiB and measured a 23% improvement. That number was real. It was also useless, because of how I measured it.

What the number was actually measuring

A single-threaded loop instantiating one plugin over and over is close to the best case for keeping memory resident. Each teardown pays a memset to clear the slab, and each instantiation immediately reuses that same warm slab. In a loop, the cost is paid once and collected once, right away.

Under real concurrency, the trade reverses. The memset on every teardown is work on the hot path, and with several instances alive at once, the slab you warmed is frequently not the one the next request gets. You have replaced a single madvise — which the kernel handles lazily and cheaply — with two megabytes of memory writes per request.

The sweep at 16 concurrent connections, against one plugin route:

`keep_resident`	CPU per request	Throughput
0	370 µs	~10k RPS
64 KiB	379 µs	~10k RPS
256 KiB	431 µs	9.2k RPS
1 MiB	810 µs	5.5k RPS

At 2 MiB it cost roughly half the throughput. The setting that was 23% faster on one thread was a 2× regression under the load the thing actually gets.

The bug hiding behind it

While sweeping, I found something worse. The pool's total_memories and total_tables were left at 64, and that number caps concurrent instances — not total ones. Past about 32 concurrent connections with keep-alive, requests started failing.

They failed as 502 Bad Gateway, from the passthrough handler, with no log line. So the symptom was: throughput stops scaling, and the only evidence is in the load generator's output, not in mine. I had been reading the RPS numbers as saturation when they were partly errors.

Two lessons, and the second is the one I keep re-learning:

A capacity limit is also a failure mode. If it can be hit, it needs a log line at the point it is hit, not just documentation of the limit.
An error path that returns a status without logging is a hole you will fall into precisely when you are measuring something else.

The change that measured nothing

The same pass removed a static ROUTER: LazyLock<_> inside the plugins. It looked like a cache. It could not have been one: every HTTP call gets a fresh instance, so the lazy initializer ran once per request anyway, and the lock never cached anything across calls.

Replacing it with a plain function measured identical, to the microsecond. I kept the change anyway. Code that claims a cache which cannot exist is worse than code that makes no claim — the next person to read it, including me in six months, will reason about the system based on that claim.

What I would tell myself

Benchmark under the concurrency you expect, not the concurrency that is easy to set up. Single-threaded loops are excellent for finding algorithmic waste and actively misleading for anything involving memory reuse, allocator behaviour, or contention — which is most of what a host process does.

And when a result looks like a clean win, check what it costs at the other end of the parameter you were not varying.