# When the LLM is wrong: training a small model on its answers

A small local model that asks an LLM when it isn't sure, and trains on the answers, should need the LLM less and less. With a perfect LLM it does: on BANKING77 the share sent fell to 23%. With an LLM that is wrong 15% of the time, it stayed at 64%. The wrong answers barely moved accuracy (90.2% against 92.0%); they broke confidence. Fitting confidence on a different set of examples brought sends back to 24 to 31%. On real Llama 3.1 8B answers the share sent fell from 30% to 19%.

Every number here comes from a script in [decidekit/eval](https://github.com/debuffjp/decidekit/tree/main/eval) and its committed results file.

| BANKING77, 77 labels, 15% of LLM answers wrong | Sent to the LLM | Head accuracy on test |
|---|---|---|
| Perfect LLM | 23% | 92.0% |
| Calibrate on the LLM's answers | 64% | 90.2% |
| Calibrate on 2 checked examples per label | 24% | 90.1% |
| Calibrate on zero-shot's confident answers (no labels) | 31% | |

Last round of a 10,003-message stream, [eval/long-stream.ts](https://github.com/debuffjp/decidekit/blob/main/eval/long-stream.ts).

## The setup

[Decide](/decide/) turns a sentence encoder (bge-small-en-v1.5, 34 MB) into a classifier with one softmax layer. `withFallback()` answers locally when its confidence is at least 0.9 and calls an LLM otherwise. Every answer becomes a training example, and the head retrains every few hundred answers.

Confidence comes from a temperature fitted on held-out training examples. That works when the labels are right. Here the labels are the LLM's answers.

To control the error rate we replaced the LLM with a stand-in that answers correctly, except on a set share of messages, where it returns zero-shot's best wrong label. That's the plausible kind of mistake: the neighbouring intent.

## 1. Wrong labels break confidence, not accuracy

A softmax head averages over its examples, so 15% wrong labels cost only a couple of points of accuracy. The temperature has no such luck. It's fitted to make confidence match how often held-out labels agree with the head, and 15% of those labels are wrong. The fitted temperature rises until few answers reach 0.9, and nearly everything is sent again. At 30% noise, 86% was sent.

## 2. Fit confidence on examples you trust

`train()` takes `calibrateOn`: examples used only to fit the temperature. With 2 hand-checked examples per label (154 in all), sends at 15% noise fell from 64% to 24%, the perfect-LLM rate.

Most apps won't hand-check examples, so `withFallback` makes its own set. Starting from label names, it keeps texts that zero-shot answered at 0.9 or more (up to 4 per label), labelled with zero-shot's answer. About 7% of those are wrong, but their errors don't track the LLM's. At 15% noise that sent 31%; at 30% noise, 36% against 86%.

When mistakes cluster on hard texts, as real ones do, the result held on both datasets. On CLINC150 (150 labels) the zero-shot set matched hand-checked examples: 21% sent against 20%, and 43% with the LLM's own answers.

## 3. Learn "none of these"

Real traffic includes text that fits no label. An LLM answers `null`, and those answers were thrown away. Learning them as an extra label lets the head turn away off-topic text itself. On CLINC150 with 4.5% off-topic traffic, the share of held-out off-topic queries given a confident wrong label halved, from 7.6% to 3.8%.

But LLMs also say `null` for ambiguous text that does belong to a label. Llama 3.1 8B did on 6.6% of in-scope messages. Those sit close to the labels: median similarity 0.77 to the nearest label, against 0.46 for off-topic text. So Decide learns a `null` only when the text's similarity to every label is below 0.6.

## 4. With a real LLM

We recorded Llama 3.1 8B's answers (Workers AI) for 1,732 messages: 12 BANKING77 intents plus 70 off-topic queries. 8B agreed with the dataset on 89% of them. Replayed through `withFallback` in 5 orders:

| | Sent, second half | In-scope answered right locally | Off-topic given a wrong label |
|---|---|---|---|
| Zero-shot only | 30.1% | | |
| Calibrate on the LLM's answers, drop nulls | 24.6% | 78.3% | 6.7% |
| Decide defaults | 19.1% | 83.3% | 2.4% |

Script: [eval/real-stream.ts](https://github.com/debuffjp/decidekit/blob/main/eval/real-stream.ts). The [two lanes demo](https://debuff.jp/decide/lanes/) replays the same answers.

## What this doesn't show

- The real-LLM run is one 12-label task with one model. The large-label results use a stand-in LLM.
- The 0.6 similarity gate is tuned for bge-small. On CLINC150 it keeps the safety gain (4.4% wrong labels) but answers fewer off-topic texts locally: 24% against 56% with no gate. `multilingual-e5-small` puts even unrelated text above 0.6, so with it the gate never fires.
- Zero-shot's confident answers are only a good calibration set when zero-shot is decent on your labels. With labels whose names say little, write a one-line description for each.

Try it on your own labels: `npm install decidekit`, then follow the [fallback guide](/decide/docs/guide-fallback).