Benchmarking PHP without lying to yourself

I wanted to know what exponentiation costs in PHP, so I wrote the loop anyone would write: ten million iterations of $result = 2 ** 16;, timed with hrtime(). It came back at 18.92 ms, about 1.9 ns an iteration. Exponentiation is free, then.

The number was real. The conclusion was wrong, because there is no exponentiation in that program. Both operands are literals, so the compiler folds the expression while compiling and the loop body it emits is an assignment of the integer 65536. I measured ten million assignments and read the result as a fact about **.

Move the base out of the compiler’s reach and the operation comes back:

<?php

declare(strict_types=1);

$iterations = 10_000_000;
$base       = (int) ($argv[1] ?? '2');

$start = hrtime(true);

for ($i = 0; $i < $iterations; $i++) {
    $result = $base ** 16;
}

printf("%.3f ms\n", (hrtime(true) - $start) / 1_000_000);

Same loop, same iteration count, 55.63 ms instead of 18.92 ms. The opcode dumps say why: the first version’s loop body is a single ASSIGN of a constant, and the second’s is a POW instruction feeding an assignment. The benchmark was honest about what it timed. I was the one who mislabeled it.

What varied that I did not intend

A microbenchmark is a controlled experiment with an unflattering number of uncontrolled variables.

Frequency scaling and thermal limits mean a laptop does not run at one speed. It shows up as drift: later runs slower than earlier ones, in a series that looks otherwise clean. Interleaving the variants, rather than running all of A then all of B, turns that drift into noise hitting both sides instead of a difference that flatters whichever ran first.

Other processes on the machine show up differently — as outliers, a handful of runs far above the rest while the median barely moves. That is the argument for reporting a median and a range rather than a mean.

OPcache state decides whether a run paid to turn source into opcodes at all, which is a different order of cost from anything happening inside the loop; the path a request takes from source to opcodes has the figures. Benchmarking with the cache off measures a configuration you do not deploy. The JIT is the same kind of variable and a more treacherous one, because it does not shift cost uniformly — what the JIT reaches and what it does not records a loop that ran sixteen times slower with it enabled at a hundred iterations and six times faster at two million.

Dataset size decides which memory you are measuring. Ten elements live in L1 and report the speed of L1. The production array has fifty thousand elements and does not.

The disclosure the number needs

Here is the opening measurement reported the way I would want to read it.

PHP 8.5.9 CLI, NTS, arm64, Homebrew build. opcache.enable_cli=1, and opcache.jit=disable, which is this build’s default — no machine code was generated for either variant. Apple M4 Pro laptop running macOS, not quiesced; load average sat near 1.7 throughout. Ten million iterations per run, hrtime(true) read once before and once after the loop in the same process, 15 runs per variant, variants interleaved, one warmup pair discarded. Held constant: iteration count, loop structure, ini settings. Varied: whether the exponent’s base was a literal or arrived through $argv.

The folded variant’s median was 18.92 ms across a range of 18.52 to 19.68 ms, 6.1% of the median. The runtime-base variant’s median was 55.63 ms across 55.39 to 56.15 ms, 1.4%. The ratio of the medians is 2.94.

The variance line is the one that gets dropped, and it is the one that decides whether the ratio means anything. Here the two ranges — 18.52 to 19.68 and 55.39 to 56.15 — do not come near each other, so the 2.94x is a real separation rather than an artifact of run ordering. Had they overlapped, the honest report would have been that this method could not tell the two variants apart, whatever the medians said. Quoting a ratio whose ranges overlap is how a benchmark becomes an opinion with digits.

Relative numbers travel, absolute ones do not

The 18.92 ms figure is close to useless to you. Your CPU is different, your PHP build has different extensions loaded, and your machine is busy with something mine was not. Quote it in a pull request and someone will compare it against a number from their laptop, and the comparison will be meaningless in a way that is hard to see.

The ratio survives the trip. So does the spread, which tells a reader how much to trust the ratio. Report the absolute figures too, but attached to the machine that produced them, and expect them to be wrong somewhere else.

There is a second reason to distrust the absolutes. The 3.7 ns per iteration separating my two variants is real, and it is also irrelevant to a request that spends most of its time waiting on a database. Scaling a microbenchmark up to a request means knowing how many times the operation runs in that request, which the microbenchmark cannot tell you.

Benchmark and profiler answer different questions

A profiler answers “where did the time go” for one execution of the real path. It finds the function worth caring about, and it distorts absolute costs while doing it, because the instrumentation is part of what you are timing — far more for an instrumenting profiler than a sampling one, by a margin I have not measured.

A benchmark answers “did this change help” for one operation under conditions you chose. It is precise about a narrow thing and silent about whether the thing matters.

Both instruments run inside the process, which bounds what either can see. Time a request spends queued at the socket waiting for a free worker is spent before any of your code starts, so it appears in neither, and sizing a PHP-FPM pool has to be measured from outside the worker to be measured at all.

Reaching for the wrong one is cheap to do and expensive to recover from. An afternoon spent microbenchmarking two implementations of a function that a profiler would have shown as a rounding error in the request is an afternoon spent well by every measure except the one that counts.

A result that contradicted the expectation

I expected a warm cache of compiled code to beat compiling from source, so I set up 300 class files and compared requiring all of them with OPcache disabled against requiring them under opcache.file_cache_only=1 with the cache directory already holding 301 entries. Same machine and method as above, 15 interleaved runs each. This is a different series from the one behind the compile figure in what happens when PHP runs your code, taken at a different time, so both its median and its range come out a little different from the 7.69 ms spanning 7.61 to 7.89 quoted there. That gap is the run-to-run variation this article is about, turning up in its own numbers.

Compiling: median 7.678 ms, range 7.567 to 7.938. Warm file cache: median 8.024 ms, range 7.962 to 8.202. The ranges do not overlap, so the cache really was slower here — by 4.5%, a margin I would not have trusted from three runs.

I have not chased that down, and I am reporting it rather than dropping it. What I can say is what the experiment was not: a CLI process cannot inherit another process’s shared memory cache, so this compared deserializing opcodes from disk against compiling small files, which is not the comparison an FPM pool makes. The result is real and the question I thought I was asking was not the one I asked.

A method that survives contact with production

Measure the real path first, with a profiler, under something resembling real input. That produces a ranked list of where the time goes and rules out most of what you were about to optimize.

Form a hypothesis that names an operation. “The container resolves this service on every request and it costs more than it looks” is a hypothesis. “Arrays are slow” is not. The first one has an answer — what container compilation removes and what it does not — precisely because it names something a measurement can isolate.

Microbenchmark that operation, and only it, with the variants interleaved and the variance reported. Check the opcodes if the result looks too good — a difference that large usually means the two variants are not doing the same work.

Then re-measure the real path and confirm the effect survived. It often does not, and finding that out costs one profiler run, which is cheaper than maintaining an optimization that never helped.