Henka: tests for your tests

A green test suite tells you the tests passed. It doesn't tell you they'd fail if the code were wrong. Henka (変化, "change") checks that: it breaks your code on purpose, one small change at a time, runs your tests after each, and lists every change no test caught. On debuff's own effects package, with 94% line coverage, 132 of Henka's 266 changes passed every test, including one that sends every particle the wrong way (what it found).

npm install -D henka
npx henka

✔ caught   src/score.ts:3:15 hits * 2 → hits / 2
✖ missed   src/score.ts:2:12 hits < 3 → hits <= 3

missed: no test failed, so nothing tests what these change. Add a test of the behaviour they control.
  src/score.ts:2:12 hits < 3 → hits <= 3

That miss means no test calls combo(3), the one input where < and <= differ. The fix is a test for it.

Why we built it

Agents write most of the tests in debuff games now, and an agent asked for tests writes tests that pass. They run every line and often check little: a test that the player "reaches top speed" passes whether top speed is 140 or 600. Coverage can't tell those tests from good ones. Mutation testing can, and its output suits agents: each miss is a file, a line and a change, which says what the next test should check. Researchers have found the same. Feeding surviving mutants back to a model writes better tests than asking it again (MutGen), and Meta's engineers accepted 73% of the tests its mutant-guided generator wrote (ACH).

Henka writes its results to mutants.out/: outcomes.json and one text file per outcome, a mutant per line. Give missed.txt to your agent and ask for tests of the behaviour behind each line.

One command, no config

Henka runs in any project that tests with Vitest 5 and Vite 8:

• It runs your tests once unchanged, and notes which test files load each source file.
• A Vite plugin swaps in the mutated code as the file loads. Nothing on disk changes, so your editor and watcher never see a half-broken file.
• Each mutant runs only the test files that load its file, in a Vitest process kept open between mutants, and stops at the first failure.
• A mutant that hangs the tests is killed and counts as caught.
• --since main mutates only the lines your branch changed, so a run on a pull request stays small.

It mutates arithmetic, comparisons, logic, signs, number literals and whole function bodies, and leaves strings, types and imports alone. The Henka page has the full list.

Mutants for games

Game tests have a problem other tests don't: the numbers are meant to change. A designer tunes jump height for weeks, and a test that pins it to 140 gets in the way. So Keiko ships a Henka adapter that changes the game's tuning, not its code:

npx keiko mutate

It pushes each tuning param to the ends of its range and expects a test to fail. For games on other engines that use Math.random() and the browser's frame rate, it also runs them with a different seed and at 30 fps and expects every test to pass. A test that fails there is brittle: it depends on luck or on the frame rate. debuff games seed their own randomness and run on a fixed step, so those two don't apply to them. On the platformer template, 8 of 16 param mutants went unnoticed, and the player's air acceleration wasn't tested at either end.

Any test runner

Henka's core runs nothing itself. An adapter runs the tests and applies the mutants, so the same reports and outcomes work for any runner:

import { mutate, writeOut } from "henka";

const report = await mutate({
  name: "my-runner",
  baseline: () => runMyTests(),
  mutants: () => [{ id: "timeout=0", label: "request timeout 30 s → 0", expect: "noticed" }],
  run: (mutant) => runMyTests({ apply: mutant.id }),
});
writeOut(report, "mutants.out");

Limits

• npx henka needs Vitest. debuff games test with Keiko, so for now they get param mutants only; source mutants through Keiko come next.
• A run makes a few hundred mutants for a small package and takes minutes. We haven't benchmarked it against other tools under the same load yet, so we make no speed claim.
• Some mutants change nothing a test could see. Mark those lines // henka-skip.

Henka is MIT-licensed and has no runtime dependencies. Try it on your next branch:

npx henka --since main