What a coding agent needs from a CLI failure

A coding agent runs your CLI and reads two things back: the exit code and stdout. We measured how 12 popular JavaScript CLIs fail, the same way for each. 10 of them reject an option they don't have, but only 2 give that mistake a different exit code from a real failure. Of the 7 with a JSON mode, 6 print parseable JSON when the run fails, and 1 when the tool can't start at all. Nothing in the output says whether trying again would help. The script and the raw results are in bench/cli-failures.

Tool	Version	Unknown option	Usage error vs real failure (exit)	JSON on a failure	JSON when it can't run
ESLint	10.13.0	rejected	2 vs 1	JSON	nothing on stdout
webpack	5.112.2	rejected	2 vs 1	JSON	nothing on stdout
npm	12.2.0	rejected	1 vs 1	JSON	JSON with a code
Jest	30.5.2	rejected	1 vs 1	JSON	nothing on stdout
Biome	2.6.3	rejected	1 vs 1	JSON	nothing on stdout
oxlint	1.90.0	rejected	1 vs 1	JSON	text
Vitest	5.0.5	rejected, stack trace	1 vs 1	written to a file	nothing on stdout
TypeScript (tsc)	7.0.2	rejected	1 vs 1	no JSON mode	no JSON mode
Vite	8.3.5	rejected, stack trace	1 vs 1	no JSON mode	no JSON mode
esbuild	0.28.2	rejected, stack trace (Windows)	1 vs 1	no JSON mode	no JSON mode
Prettier	3.9.10	ignored, with a warning	0 vs 1	no JSON mode	no JSON mode
Rollup	4.65.1	ignored, with a warning	0 vs 1	no JSON mode	no JSON mode

Windows 11, Node 24.15.0, every tool at its current release on 2026-10-28. "Can't run" means the config file passed to it doesn't exist (for npm, a package that doesn't exist).

Why an agent cares

A person reads a failure and knows what kind it is. An agent has to decide from the exit code and stdout, and the decision is one of four:

1. It worked: carry on.
2. The thing it checked failed (a type error, a failing test): read the report and fix the code.
3. The command itself was wrong: fix the command line.
4. The machine or project isn't set up: fix that, and don't retry the same command.

Mixing these up is expensive. An agent that reads a typo'd flag as failing tests goes looking for a bug in code that's fine. One that retries a missing config file loops. Most of the measurements below are about whether a tool lets the agent tell these apart without parsing prose.

What we found

1. Most tools reject an option they don't have. 10 of 12 exit non-zero for --frobnicate. ESLint is the best at it: Invalid option '--frobnicate' - perhaps you meant '--format'?. Prettier and Rollup run anyway and print a warning, which a person sees and an agent piping stdout may not.
2. Three rejections come as stack traces. Vitest and Vite print the message, then the stack of the argument parser that threw it. esbuild's own error is clean, but its Node launcher on Windows adds a stack on top. None of these is wrong, but an agent reading the first lines of output sees parser internals before the problem.
3. Exit codes rarely separate "you typed it wrong" from "your code failed". Only ESLint and webpack exit 2 for a usage error and 1 for a real failure. For the other 8, tsc --noEmit --frobnicate good.ts and tsc --noEmit bad.ts both exit 1, so the exit code alone reads a bad flag as type errors.
4. JSON modes cover failures, not setup. When the run fails for a real reason, 6 of 7 JSON modes give one parseable document on stdout. Vitest writes its JSON report to a file and prints the path, a fair choice that a pipe has to know about. When the tool can't start because its config is missing, 5 of the 7 print nothing on stdout and put a text error on stderr. oxlint prints text on stdout. Only npm answers in JSON, with an error code (E404 for a package that doesn't exist).
5. No tool says whether to retry. npm's codes come closest: E404 and a network failure such as ENOTFOUND are different codes, so an agent that knows npm's codes can tell them apart. Nothing we captured says so in a field an agent could act on without that knowledge.

A contract that answers all four

debuff's CLI has one failure contract, from 0.20:

Exit	Meaning	What to do
0	Success	Carry on
1	A run reported a failure (a test, scenario or fuzz finding), or something unexpected broke	Read the report
2	The project or machine needs fixing first: no browser, invalid config, missing dev hooks, a busy port	Run npx debuff doctor; a retry won't help
3	Bad input: an unknown command or option, a folder that isn't there	Fix the command line

With --json, stdout carries exactly one JSON document, and a command that can't run prints this instead of its result:

{ "error": { "code": "browser_not_found", "message": "…", "fix": "Install Chrome or Edge, or set DEBUFF_BROWSER…", "transient": false } }

code comes from a fixed list and keeps its meaning between releases. transient: true means the same command may work if run again. The same code and fix appear in the errors of debuff's MCP tools.

The bench runs debuff too, before and after:

debuff	Unknown option	Usage error vs real failure (exit)	JSON on a failure	JSON when it can't run
0.19	ignored silently	0 vs 1	text instead of JSON	text
0.20	rejected	3 vs 1	JSON	JSON with a code

0.19 was worse than most of the tools above: it ignored unknown options without a word, and when a --json run failed it printed a human error instead of JSON. Running the bench against 0.20 also caught one more gap: a project with no scenarios came back as unexpected, where it now gets its own code, nothing_to_run, from 0.21.

For tool authors

What we'd ask of any CLI that agents run:

1. Reject unknown options, and name the closest real one, as ESLint does.
2. Give usage errors their own exit code. ESLint and webpack use 2.
3. In JSON mode, keep stdout to one document, including when the tool can't start. Logs go to stderr.
4. Give errors a stable code, and say whether retrying can help.
5. Print a message, not a stack trace, for a mistake the user made.

What this doesn't show

• One platform (Windows) and one run. esbuild's stack trace comes from its Windows launcher; on Linux and macOS npm links its native binary directly.
• One command per tool, picked to be representative. A tool can behave better in another subcommand.
• One way to fail to start: a missing config file. Bad config contents, missing dependencies or a dead network can come out differently.
• The labels for warnings and stack traces are pattern matches on the output, so each row keeps the raw exit codes and output to check them against.

Run it yourself, or add a tool and send a PR:

node bench/cli-failures/run.mjs