What 94% line coverage missed

@debuffjp/juice, debuff's tweens, easing and particles, has 94% line coverage from nine tests. We ran mutation testing on it: Henka made 266 small changes to the source, one at a time, and ran the tests after each. 132 changes passed every test. We read each of those. 69 are real gaps in code the tests run, such as particles moving the wrong way. The others are in code that never runs (35), change nothing a test could see (18), or are default tuning values that a test shouldn't pin (10). The script, both raw results and the triage are in bench/mutation.

File	Lines covered	Mutants	Missed	Gaps in code that ran
particles.ts	96%	78	59	46
ease.ts	81%	105	46	11
tween.ts	93%	51	13	5
trail.ts	100%	24	9	4
recipes.ts	100%	8	5	3
All	94%	266	132	69

Windows 11, Node 24.15.0, Vitest 5.0.5, juice 0.30.0 (360 lines of source). Coverage from Vitest's V8 provider (81% of branches); mutants from Henka 0.1.0.

1. What a mutant is

A mutant is the code with one small change, the kind of slip a person or an agent could make: a + that should be -, a < that should be <=, a number set to 0. If no test fails, nothing checks what that code does. This one passed all nine tests:

// particles.ts, Particles.update()
- p.x += p.vx * dt;
+ p.x -= p.vx * dt;

Every particle now flies the opposite way to its velocity. The line runs in the tests, so coverage counts it. The tests check that particles are emitted, expire and go back to the pool, and none of them checks where a particle ends up.

2. Gaps in code that ran

The 69 gaps are lines the tests execute without checking what they do. Most of them are in the particle system, where 46 changes passed:

• Position along either axis reversed, gravity's sign flipped, and velocity divided by dt instead of multiplied.
• p.drag === 1 flipped to !==, which ignores drag for every particle. No test sets drag.
• Size and alpha over a particle's life: inverting the size curve, dividing alpha instead of multiplying it and drawing each circle at four times its radius all passed.
• Colour picking divided by the list length instead of multiplied, so every particle takes the first colour.

The rest are spread across the other files. squash() widens a sprite and makes it shorter, then springs back; flipping the height to 1 + amount, so the sprite grows taller, passed, because the test checks only the width. In tweens.kill(target), || turned into && makes the call kill the running tweens of every other object too, and passed. In the easing curves, outQuad's t * (2 - t) became t / (2 - t): the only easing test checks that each curve maps 0 to 0 and 1 to 1, and this one still does.

None of these is a bug in juice today. Each is a change that would ship without a red test, on a line coverage counts as tested.

3. What the coverage number hid

35 misses are in code no test runs. 20 are on lines V8 reports as never run: the second and third pieces of outBounce, tween repeat, and Particles.clear(). The other 15 are in outElastic's middle branch. Its line runs, because the tests call it at 0 and 1, so line coverage counts it, but the formula between the two ends never runs.

Branch coverage (81%) would have pointed at most of those 35. It wouldn't have pointed at the 69: every one of them is on a line and a branch the tests run.

4. Misses that aren't gaps

18 changes make no difference a test could observe. Five are < to <= where two pieces of an easing curve meet at the same value, so either side gives the same answer. Four are ?? 0 changed to ?? 1 where the index is always in range, so the fallback never applies. The other nine include a loop that starts past the end of an array and skips the extra indices, a mirrored spread that gives the same distribution of directions, and shortcuts that only save time. // henka-skip on such a line stops it showing up again.

10 are default tuning values: a particle burst's count, speed, lifetime and size, and squash()'s amount and duration. A test that fails when the default speed range changes from [40, 120] pins the number, and someone tuning the game's feel then has to edit a test to do it. For values like these, test the behaviour they control: that a faster burst spreads further, for example. Henka mutates every number literal, so it lists these with the rest; marking them is the tester's call.

5. Checking Henka against Stryker

Henka swaps each mutant in as Vite loads the file, inside a Vitest process kept open between mutants. That could go wrong in two ways: a stale module could leak one mutant into the next run, or the swap could break something unrelated and fail a test the mutant didn't touch. To check for both, we ran StrykerJS 10.1.1 on the same package and compared verdicts.

The same edit run against the same deterministic tests should get the same verdict, so any disagreement would point at Henka's harness. Matching mutants by file, line and replacement text gives 152 that both tools make. 8 of those match more than one of Stryker's (two identical 2 * t edits on one line can't be told apart by text), and we count those as agreeing when either matches. On all 152 the verdicts agree.

The two tools mutate different things, so their scores don't compare directly: Henka catches 134 of 266 (50%), Stryker 232 of 362 (64%, or 71% of the 326 on code that ran). 106 of Henka's mutants change number literals, which Stryker doesn't mutate, and they account for 58 of its 132 misses. Stryker has mutators Henka lacks. On code that ran, 43 of Stryker's surviving mutants have no Henka counterpart: 20 conditions forced true or false, 8 logical operators (several of them ??) and the rest strings, arrays and method calls. Those are gaps Henka can't report yet.

Method

Stryker needed three changes to run on this repository. tsconfigFile had to point at a file that doesn't exist, because its tsconfig preprocessor calls a function TypeScript 7 no longer exports. It needed a Vitest config of its own, since it doesn't read test.projects. And its sandbox had to keep every tsconfig the root references. The configs are in bench/mutation.

Henka took 10 min 38 s at 2 jobs, a median 3.4 s per mutant, while other work held the CPU at 97%. Stryker took 5 min 17 s at concurrency 2 on a quieter machine. Those times don't compare, and we haven't measured both under the same load. Most of Henka's time per mutant is probably Vitest starting a worker for each test file; we haven't isolated it.

To keep runs small, npx henka --since main mutates only the lines a branch changed. Google's mutation testing goes further: besides mutating changed code only, it reports at most one mutant per line and drops kinds of mutant its developers found unproductive (Petrović et al., 2021). Henka has the first of those three.

What this doesn't show

• One small package. Packages with slow tests weren't tried.
• No speed comparison between the two tools.
• The triage is our reading of each miss against the source. Another reader might move a few between equivalent, tuning and gap; each call is in triage.json with its reason.
• Whether the gaps matter to players. Reversed particles would be noticed in seconds; a wrong easing curve might ship for months.

Try it

In a project that tests with Vitest 5:

npx henka --since main