# Benchmarking PHP without lying to yourself

> Microbenchmarks measure exactly what you tell them to, which is rarely what you meant. Method, noise, and the numbers worth reporting.

- Published: 2026-07-16
- Tags: benchmarking, profiling, opcache
- Source: https://elephantphp.com/blog/benchmarking-php-without-lying-to-yourself/
- Language: en-US
- Author: Alden Pike

---
I wanted to know what exponentiation costs in PHP, so I wrote the loop anyone
would write: ten million iterations of `$result = 2 ** 16;`, timed with
`hrtime()`. It came back at 18.92 ms, about 1.9 ns an iteration. Exponentiation
is free, then.

The number was real. The conclusion was wrong, because there is no
exponentiation in that program. Both operands are literals, so the compiler
folds the expression while compiling and the loop body it emits is an assignment
of the integer 65536. I measured ten million assignments and read the result as
a fact about `**`.

Move the base out of the compiler's reach and the operation comes back:

```php
<?php

declare(strict_types=1);

$iterations = 10_000_000;
$base       = (int) ($argv[1] ?? '2');

$start = hrtime(true);

for ($i = 0; $i < $iterations; $i++) {
    $result = $base ** 16;
}

printf("%.3f ms\n", (hrtime(true) - $start) / 1_000_000);
```

Same loop, same iteration count, 55.63 ms instead of 18.92 ms. The opcode dumps
say why: the first version's loop body is a single `ASSIGN` of a constant, and
the second's is a `POW` instruction feeding an assignment. The benchmark was
honest about what it timed. I was the one who mislabeled it.

## What varied that I did not intend

A microbenchmark is a controlled experiment with an unflattering number of
uncontrolled variables.

Frequency scaling and thermal limits mean a laptop does not run at one speed. It
shows up as drift: later runs slower than earlier ones, in a series that looks
otherwise clean. Interleaving the variants, rather than running all of A then
all of B, turns that drift into noise hitting both sides instead of a difference
that flatters whichever ran first.

Other processes on the machine show up differently — as outliers, a handful of
runs far above the rest while the median barely moves. That is the argument for
reporting a median and a range rather than a mean.

OPcache state decides whether a run paid to turn source into opcodes at all,
which is a different order of cost from anything happening inside the loop; [the
path a request takes from source to
opcodes](/blog/what-happens-when-php-runs-your-code/) has the figures.
Benchmarking with the cache off measures a configuration you do not deploy. The
JIT is the same kind of variable and a more treacherous one, because it does not
shift cost uniformly — [what the JIT reaches and what it does
not](/blog/opcache-jit-and-where-time-actually-goes/) records a loop that ran
sixteen times slower with it enabled at a hundred iterations and six times
faster at two million.

Dataset size decides which memory you are measuring. Ten elements live in L1 and
report the speed of L1. The production array has fifty thousand elements and
does not.

## The disclosure the number needs

Here is the opening measurement reported the way I would want to read it.

PHP 8.5.9 CLI, NTS, arm64, Homebrew build. `opcache.enable_cli=1`, and
`opcache.jit=disable`, which is this build's default — no machine code was
generated for either variant. Apple M4 Pro laptop running macOS, not quiesced;
load average sat near 1.7 throughout. Ten million iterations per run,
`hrtime(true)` read once before and once after the loop in the same process, 15
runs per variant, variants interleaved, one warmup pair discarded. Held
constant: iteration count, loop structure, ini settings. Varied: whether the
exponent's base was a literal or arrived through `$argv`.

The folded variant's median was 18.92 ms across a range of 18.52 to 19.68 ms,
6.1% of the median. The runtime-base variant's median was 55.63 ms across 55.39
to 56.15 ms, 1.4%. The ratio of the medians is 2.94.

The variance line is the one that gets dropped, and it is the one that decides
whether the ratio means anything. Here the two ranges — 18.52 to 19.68 and 55.39
to 56.15 — do not come near each other, so the 2.94x is a real separation rather
than an artifact of run ordering. Had they overlapped, the honest report would
have been that this method could not tell the two variants apart, whatever the
medians said. Quoting a ratio whose ranges overlap is how a benchmark becomes an
opinion with digits.

## Relative numbers travel, absolute ones do not

The 18.92 ms figure is close to useless to you. Your CPU is different, your PHP
build has different extensions loaded, and your machine is busy with something
mine was not. Quote it in a pull request and someone will compare it against a
number from their laptop, and the comparison will be meaningless in a way that
is hard to see.

The ratio survives the trip. So does the spread, which tells a reader how much
to trust the ratio. Report the absolute figures too, but attached to the machine
that produced them, and expect them to be wrong somewhere else.

There is a second reason to distrust the absolutes. The 3.7 ns per iteration
separating my two variants is real, and it is also irrelevant to a request that
spends most of its time waiting on a database. Scaling a microbenchmark up to a
request means knowing how many times the operation runs in that request, which
the microbenchmark cannot tell you.

## Benchmark and profiler answer different questions

A profiler answers "where did the time go" for one execution of the real path.
It finds the function worth caring about, and it distorts absolute costs while
doing it, because the instrumentation is part of what you are timing — far more
for an instrumenting profiler than a sampling one, by a margin I have not
measured.

A benchmark answers "did this change help" for one operation under conditions
you chose. It is precise about a narrow thing and silent about whether the thing
matters.

Both instruments run inside the process, which bounds what either can see. Time
a request spends queued at the socket waiting for a free worker is spent before
any of your code starts, so it appears in neither, and [sizing a PHP-FPM
pool](/blog/php-fpm-process-management-in-production/) has to be measured from
outside the worker to be measured at all.

Reaching for the wrong one is cheap to do and expensive to recover from. An
afternoon spent microbenchmarking two implementations of a function that a
profiler would have shown as a rounding error in the request is an afternoon
spent well by every measure except the one that counts.

## A result that contradicted the expectation

I expected a warm cache of compiled code to beat compiling from source, so I set
up 300 class files and compared requiring all of them with OPcache disabled
against requiring them under `opcache.file_cache_only=1` with the cache
directory already holding 301 entries. Same machine and method as above, 15
interleaved runs each. This is a different series from the one behind the
compile figure in [what happens when PHP runs your
code](/blog/what-happens-when-php-runs-your-code/), taken at a different time,
so both its median and its range come out a little different from the 7.69 ms
spanning 7.61 to 7.89 quoted there. That gap is the run-to-run variation this
article is about, turning up in its own numbers.

Compiling: median 7.678 ms, range 7.567 to 7.938. Warm file cache: median 8.024
ms, range 7.962 to 8.202. The ranges do not overlap, so the cache really was
slower here — by 4.5%, a margin I would not have trusted from three runs.

I have not chased that down, and I am reporting it rather than dropping it. What
I can say is what the experiment was not: a CLI process cannot inherit another
process's shared memory cache, so this compared deserializing opcodes from disk
against compiling small files, which is not the comparison an FPM pool makes.
The result is real and the question I thought I was asking was not the one I
asked.

## A method that survives contact with production

Measure the real path first, with a profiler, under something resembling real
input. That produces a ranked list of where the time goes and rules out most of
what you were about to optimize.

Form a hypothesis that names an operation. "The container resolves this service
on every request and it costs more than it looks" is a hypothesis. "Arrays are
slow" is not. The first one has an answer — [what container compilation removes
and what it does not](/blog/what-compiled-di-containers-actually-buy-you/) —
precisely because it names something a measurement can isolate.

Microbenchmark that operation, and only it, with the variants interleaved and
the variance reported. Check the opcodes if the result looks too good — a
difference that large usually means the two variants are not doing the same
work.

Then re-measure the real path and confirm the effect survived. It often does
not, and finding that out costs one profiler run, which is cheaper than
maintaining an optimization that never helped.