Benchmarking 2D renderers without fooling yourself

Renderer benchmarks are easy to rig, and usually nobody means to. A library is left on a slow default. A scene happens to suit one design. Vsync hides the real cost. A number comes from one lucky run. The result looks rigorous and means very little.

We build spryte, so we have every reason to want a flattering benchmark, and every reason to distrust one we wrote ourselves. 2dbench is our attempt to make rigging hard to do and easy to check. This post walks through its rules, why each exists, and what you can take from them for your own measurements. The full reasoning is in METHOD.md.

1. Make the work identical by construction

The first way a benchmark goes wrong is that the libraries aren't doing the same work. One adapter updates objects differently, another skips a property, and you end up timing the setup, not the renderer.

In 2dbench, scenes own a seeded simulation. Each frame, the scene writes structure-of-arrays state: x, y, scale, rotation, color (0xRRGGBBAA) and texture. Adapters only turn those arrays into draw calls. They don't simulate anything.

That has three consequences:

• Simulation cost is identical for everyone. It's still included in the frame time, as it would be in a real game.
• Adapters can't depend on knowing the simulation. They read the per-frame arrays; they don't precompute the scene.
• More complex scenes keep all game logic in the shared code. In the raycaster scene, the simulation casts rays, shades and depth-clips enemies, and adapters only draw the resulting slices.

The rest of the setup is fixed too: same textures, a 960×540 canvas at device pixel ratio 1, nearest filtering, no antialiasing, unrotated sprites on whole pixels. Every device draws the same number of pixels.

2. Make dynamic scenes actually dynamic

An early version of the benchmark had a flaw: some properties a scene claimed to animate didn't change often enough. A library that skips unchanged work could win by caching values that never moved. That's a real optimisation, but it isn't what the scene was meant to measure.

The rule now, from the agent guide: every property a scene declares must keep changing during the run, so nothing can win by skipping unchanged work. In the dynamic scenes rotation changes every frame, alpha every 2 frames, scale every 4 and tint every 8 (METHOD.md). Adapters set only the properties a scene uses, and a bulk API that can mark properties static is configured to match what the scene really changes.

If a scene is meant to measure position-only drawing, call it that (2dbench's bunny is exactly this). If it's meant to measure per-sprite tint and alpha, those values have to move.

3. Check the output before you time it

The fastest renderer is the one that draws nothing. Missing sprites, a wrong blend mode or culling that kicks in at high counts all make a library look faster. So nothing is timed until a pixel-parity gate passes.

Before timing, each adapter draws the same seeded frame, 10 steps in, at 2,000 and at 50,000 sprites. Its pixels are compared with a reference library's. Two numbers decide pass or fail:

Metric	What it is	Catches	Fails when
Coverage	The adapter's drawn area divided by the reference's	Skipped, culled or missing sprites	Outside 99.5–100.5%
Interior differences	Pixels differing by more than 8 levels in any channel, where the 3×3 neighbourhood is one flat colour in both frames	Wrong tint, alpha, blend or texture inside a sprite	Above 0.5%

The flat-neighbourhood rule is the important part. Libraries legitimately round the corners of rotated sprites differently; PixiJS's roundPixels, for example, snaps rotated corners. Any edge, even one sprite drawn over another, breaks flatness, so edge rounding doesn't count against anyone. A wrong colour in the middle of a sprite does.

We validated the metric by deliberately giving one library a wrong tint. Interior differences rose from 0% to 22%. A check you haven't seen fail isn't a check yet.

Overall differing pixels are still reported, for transparency. In the published desktop results, coverage was 99.8–100% and 0% of pixels differed inside sprites; the rotated scenes differed at sprite edges on 8–13% of drawn pixels.

Choosing the reference

The reference is PixiJS Sprite. It's the most widely used 2D web renderer, it supports every scene, and it isn't made by us.

That wasn't the first choice. An earlier version used a hand-written WebGL2 reference. We dropped it because it was close to spryte's own design. As a referee it wasn't independent, and as a contestant it added no information. If your reference shares assumptions with one contestant, it quietly favours that contestant.

4. Time to GPU completion, not to submission

WebGL calls return as soon as the work is queued. If you stop the clock there, you're measuring how fast the CPU can queue commands, and the GPU's share disappears.

2dbench times each frame from simulation start to GPU completion. The sync is a 1-pixel readPixels, which waits for all submitted GPU work (Canvas 2D uses createImageBitmap). Frames are timed directly, not through requestAnimationFrame, so vsync and background-tab throttling don't apply.

Each run does 20 warm-up frames first, covering JIT, texture upload, buffer growth and any objects a retained-mode library creates. Then it measures 100 frames and reports, per sprite count, the median frame, the p95 frame, and the median main-thread time before the sync. That last number is useful on its own: in How spryte got fast, splitting main-thread time from total time is what showed the gap was JavaScript per sprite.

The sync has a cost, and we list it as a threat to validity. A real game never stalls on a readback; it lets frames overlap. So absolute times include one full CPU–GPU round trip, and renderers that lean on that overlap would do relatively better in a game than here. Every adapter pays the same stall, so ratios stay fair. The alternative, timing without a sync, is far worse.

5. Interleave rounds, and pick the count from data

Machines heat up, background tasks wake, and whatever runs first gets a cooler chip. Running library A fully, then library B, bakes all of that into the comparison.

2dbench runs several rounds. Each round starts every adapter on a fresh page, in a rotated order, so thermal throttling and ordering spread evenly. It reports the median across rounds with min–max, and keeps every round's value in the results file as roundsFrameMs.

The number of rounds came from a variance study: the bunny and particles scenes, 7 rounds each, on one RTX 5060 laptop. It found:

• At the published counts (10k, 50k, 200k), one round's median frame differed from another's with a coefficient of variation of 9.2% in the median cell, and up to 35.8%.
• Taking every window of 3 consecutive rounds, the 3-round median missed the 7-round median by 9.8% in the median cell, and up to 28.6%.
• Ratios between libraries, which is what any claim is about, missed by up to 15% at 50k and 200k with 3 rounds.

So the default is 5 rounds, not 3. Medians are used throughout because frame times have a long right tail (GC pauses, OS interrupts): the median of 100 frames ignores the tail, which p95 reports separately, and the median of per-round medians resists one bad round.

6. Know your timer's resolution

The noisiest cells in the study were the fastest ones. Frames under about 2 ms varied most, because the browser's timer resolution and scheduling jitter are a large share of the frame. At 10k sprites, the bulk APIs, at about 0.5–1.5 ms a frame, varied by 24–36% between rounds.

Browsers coarsen performance.now() against fingerprinting. The phone runner measures the smallest step it can see: about 0.1 ms in desktop Chrome, and 1 ms in WebKit, which every iOS browser uses. A 2 ms frame measured with a 1 ms timer is barely a measurement.

2dbench flags cells under 2 ms with †. They aren't wrong, but they're low precision, and ratios are only published where frames are 2 ms or more. One example from the study shows why: a bulk-API ratio at 10k had a per-round range of 1.00–4.00×, with frames under 1 ms. The same pair at 200k stayed within 2.55–3.38×. The iPhone results in the README are labelled indicative for the same reason.

The fix, if you need precision at small numbers, is more work per frame, not more decimal places.

7. Refuse to time a software GPU

On Linux, headless Chrome falls back to SwiftShader, a CPU emulation of a GPU, whatever the GPU flags say. Timings from it measure the CPU pretending to be a GPU, and they look perfectly plausible.

2dbench reads the GPU string from WEBGL_debug_renderer_info and refuses a timed run when it matches a software renderer (SwiftShader, llvmpipe, softpipe or Microsoft Basic Render). Parity still checks fine on software rendering, and CI does exactly that on every pull request, for correctness only. To time on Linux, run with a visible window (HEADED=1), which gets the real GPU. An explicit override exists, and it marks the result as software.

Headless Chrome uses the real GPU on Windows and macOS, and the GPU string is recorded either way. The cost of headless runs is that compositing and presentation aren't measured, which we list as a residual risk.

8. Record the environment, including power

A laptop on battery, or on a power-saving plan, throttles. Without a record of that, two honest runs can disagree and nobody knows why.

Every result records the GPU, browser version, OS, CPU, memory, library versions, the 2dbench commit, and the power source and plan. On Windows the runner reads these from the OS; elsewhere you give them in a NOTE. The agent guide is blunt: don't report timings from a laptop on battery, and don't run two timed runs at once.

Even then, absolute times vary between days on the same machine. The signal is the ratio between libraries within one run. Results from different machines aren't directly comparable, and neither are ratios from different benchmarks. 2dbench's structure-of-arrays simulation is lighter than the one in debuff's own bench, which leaves more of each frame to the renderer, so the same two libraries come out further apart: about 4.7× for particles at 200k here, against 3.8× there. Both are true of their benchmark.

9. Configure every library the way its authors would

A badly configured library looks slow, and it's the easiest unfairness to miss when you don't know that library well.

Each 2dbench adapter uses its library the way its own documentation recommends for performance, and describes its setup in a setup field copied into every result. Where a library has two reasonable paths, both are included: PixiJS Sprite and ParticleContainer are separate adapters, so general compares with general and specialised with specialised. Alternatives we measured are written down: PixiJS runs on its WebGL renderer because it measured faster than its WebGPU renderer on these scenes, and roundPixels made no measurable difference to speed. When an adapter can't draw a scene identically, it declines through supports() and says why, rather than faking it. Canvas 2D has no per-draw tint, so it only runs bunny.

10. Handle the conflict of interest in the open

2dbench was started by the debuff project, and spryte is a contestant. We can't remove that conflict, so the rules work around it:

• The parity reference is PixiJS Sprite, not spryte.
• spryte's adapters get no special treatment.
• Any change that helps spryte's results specifically needs review from someone outside the debuff project before it's merged.
• Library maintainers are invited to review their own adapter. Corrections that keep parity and are idiomatic are merged, and results are rerun.
• Changes to timing, parity or statistics need a rationale, an update to METHOD.md in the same PR, and a changelog note, because results before and after aren't comparable.
• The method, adapters and raw per-round data are public, and results data is CC0.

That doesn't make our numbers trustworthy by declaration. Trust has to be earned over time, and the best check is to re-run it yourself.

Takeaways for your own benchmarks

1. Share the simulation. Put all non-drawing work in common code, so the only thing that differs is the renderer.
2. Gate on output. Compare pixels with an independent reference before timing anything. Ignore edges, check interiors, and prove the check fails on a deliberate bug.
3. Pick a reference you don't control, and that doesn't share one contestant's design.
4. Sync to the GPU. A 1-pixel readback is crude, but timing without one measures command queuing.
5. Make animated properties really animate throughout the run, so caching unchanged values can't win.
6. Interleave and rotate, run enough rounds to make your ratios stable, and choose that number from a variance study, not habit.
7. Report medians with spread, keep every round's raw value, and read p95 for stutter.
8. Flag results near your timer's resolution. Under about 2 ms, and anywhere WebKit's 1 ms timer applies, treat numbers as indicative.
9. Check the GPU string and refuse software rendering.
10. Record the environment, power included, and compare ratios within one run, not absolute times across machines or benchmarks.
11. Write down your threats to validity. METHOD.md has a table of them, each with its residual risk. Writing one is the quickest way to find what you haven't handled.

No synthetic benchmark predicts a specific game. Once you've chosen a renderer, profile your own game on your own target devices.