When the LLM is wrong: training a small model on its answers
A small local model that asks an LLM when it isn't sure, and trains on the answers, should need the LLM less and less. With a perfect LLM it does: on BANKING77 the share sent fell to 23%. With an LLM that is wrong 15% of the time, it stayed at 64%. The wrong answers barely moved accuracy (90.2% against 92.0%); they broke confidence. Fitting confidence on a different set of examples brought sends back to 24 to 31%. On real Llama 3.1 8B answers the share sent fell from 30% to 19%.
Every number here comes from a script in decidekit/eval and its committed results file.
| BANKING77, 77 labels, 15% of LLM answers wrong | Sent to the LLM | Head accuracy on test |
|---|---|---|
| Perfect LLM | 23% | 92.0% |
| Calibrate on the LLM's answers | 64% | 90.2% |
| Calibrate on 2 checked examples per label | 24% | 90.1% |
| Calibrate on zero-shot's confident answers (no labels) | 31% |
Last round of a 10,003-message stream, eval/long-stream.ts.
The setup
Decide turns a sentence encoder (bge-small-en-v1.5, 34 MB) into a classifier with one softmax layer. withFallback() answers locally when its confidence is at least 0.9 and calls an LLM otherwise. Every answer becomes a training example, and the head retrains every few hundred answers.
Confidence comes from a temperature fitted on held-out training examples. That works when the labels are right. Here the labels are the LLM's answers.
To control the error rate we replaced the LLM with a stand-in that answers correctly, except on a set share of messages, where it returns zero-shot's best wrong label. That's the plausible kind of mistake: the neighbouring intent.
1. Wrong labels break confidence, not accuracy
A softmax head averages over its examples, so 15% wrong labels cost only a couple of points of accuracy. The temperature has no such luck. It's fitted to make confidence match how often held-out labels agree with the head, and 15% of those labels are wrong. The fitted temperature rises until few answers reach 0.9, and nearly everything is sent again. At 30% noise, 86% was sent.
2. Fit confidence on examples you trust
train() takes calibrateOn: examples used only to fit the temperature. With 2 hand-checked examples per label (154 in all), sends at 15% noise fell from 64% to 24%, the perfect-LLM rate.
Most apps won't hand-check examples, so withFallback makes its own set. Starting from label names, it keeps texts that zero-shot answered at 0.9 or more (up to 4 per label), labelled with zero-shot's answer. About 7% of those are wrong, but their errors don't track the LLM's. At 15% noise that sent 31%; at 30% noise, 36% against 86%.
When mistakes cluster on hard texts, as real ones do, the result held on both datasets. On CLINC150 (150 labels) the zero-shot set matched hand-checked examples: 21% sent against 20%, and 43% with the LLM's own answers.
3. Learn "none of these"
Real traffic includes text that fits no label. An LLM answers null, and those answers were thrown away. Learning them as an extra label lets the head turn away off-topic text itself. On CLINC150 with 4.5% off-topic traffic, the share of held-out off-topic queries given a confident wrong label halved, from 7.6% to 3.8%.
But LLMs also say null for ambiguous text that does belong to a label. Llama 3.1 8B did on 6.6% of in-scope messages. Those sit close to the labels: median similarity 0.77 to the nearest label, against 0.46 for off-topic text. So Decide learns a null only when the text's similarity to every label is below 0.6.
4. With a real LLM
We recorded Llama 3.1 8B's answers (Workers AI) for 1,732 messages: 12 BANKING77 intents plus 70 off-topic queries. 8B agreed with the dataset on 89% of them. Replayed through withFallback in 5 orders:
| Sent, second half | In-scope answered right locally | Off-topic given a wrong label | |
|---|---|---|---|
| Zero-shot only | 30.1% | ||
| Calibrate on the LLM's answers, drop nulls | 24.6% | 78.3% | 6.7% |
| Decide defaults | 19.1% | 83.3% | 2.4% |
Script: eval/real-stream.ts. The two lanes demo replays the same answers.
What this doesn't show
- The real-LLM run is one 12-label task with one model. The large-label results use a stand-in LLM.
- The 0.6 similarity gate is tuned for bge-small. On CLINC150 it keeps the safety gain (4.4% wrong labels) but answers fewer off-topic texts locally: 24% against 56% with no gate.
multilingual-e5-smallputs even unrelated text above 0.6, so with it the gate never fires. - Zero-shot's confident answers are only a good calibration set when zero-shot is decent on your labels. With labels whose names say little, write a one-line description for each.
Try it on your own labels: npm install decidekit, then follow the fallback guide.