Better head training beat fine-tuning: 93.1% on BANKING77
decidekit 0.2 reaches 93.1% on BANKING77's test split, up from 90.6%, and 96.1% on CLINC150, up from 94.6%. The encoder didn't change. We fine-tuned it first, and that lost 2.5 points on BANKING77 and 1.2 on CLINC150 on our dev splits. What worked was replacing the optimiser that fits the small head on top of it with full-batch Adam, using settings from jimothy. npm install decidekit gets you the new training.
| Test accuracy, all training data | 0.1 | 0.2 | Change | Answers only 0.1 got right | Answers only 0.2 got right |
|---|---|---|---|---|---|
| BANKING77 (77 labels) | 90.62% | 93.12% | +2.50 | 41 | 118 |
| CLINC150 (150 labels) | 94.62% | 96.13% | +1.51 | 39 | 107 |
| HWU64 (64 labels, held out from all tuning) | 90.15% | 91.17% | +1.02 | 18 | 29 |
One seed, on cached q8 embeddings from the default model (bge-small-en-v1.5). The paired test (McNemar chi-squared) gives 36.3 and 30.7 for the first two rows. HWU64's 2.1 isn't significant at this size, so that row is a good sign and no more. Through train() end to end, eval/results-h14.json has BANKING77 at 93.05% and CLINC150 at 96.29%, and 0.1 measured that way reported 90.7% and 94.4%.
Every run, test look and script is in eval/spike. The fine-tuning scripts weren't kept, only their results.
The setup
A decision has two parts: a 34 MB encoder turns text into a vector, and a head, a single softmax layer, turns the vector into label probabilities. Every head shares one cached encoder, so a head is small (118,580 bytes for 77 labels) and trains in a browser tab.
Before the work started we set a pass mark: at least +0.5 points on average across BANKING77 and CLINC150, no more than 0.2 lost on either or at 5 and 10 examples per label, HWU64 not falling, calibration error at or below 0.03 from 10 examples per label. Tuning used a seeded 20% dev split. Each final candidate got one look at the test splits, and nothing changed afterwards.
1. Fine-tuning went backwards
Fine-tuning needs intent data that isn't a benchmark we report. That ruled out more than we expected: 5,133 of SLURP's 13,514 texts appear verbatim in HWU64. We used SNIPS (CC0, 13,576 texts, 7 intents) and PAWS paraphrases, 41,119 pairs in all, and trained for 2 epochs with MultipleNegativesRanking, which took 271 s on an RTX 5060.
On the training task it looked like a win: accuracy@1 on held-out SNIPS went from 0.949 to 0.985. On the data we care about it lost at both sizes we ran.
| Dev accuracy | Per label | Plain encoder | Fine-tuned |
|---|---|---|---|
| BANKING77 | all | 92.45% | 89.91% |
| BANKING77 | 10 | 85.92% | 81.48% |
| CLINC150 | all | 96.63% | 95.40% |
| CLINC150 | 10 | 93.22% | 90.20% |
Both columns went through the same ONNX export and q8 quantisation, so the export isn't the difference. Our guess, which we didn't test, is that 7 broad intents and Wikipedia paraphrases pull the encoder away from fine distinctions such as card_arrival against card_delivery_estimate. This rules out one data mix with one set of settings, and no more than that.
2. Most of the gain was in the head
jimothy, another small intent classifier, reports 92.4% on BANKING77 with an older encoder, all-MiniLM-L6-v2. We ran it and got 92.66% from its report on its own split. That pointed at its optimiser rather than its encoder, so we tried its settings on ours.
Each row is one run on the dev splits, with all the data.
| Change to the head | BANKING77 | CLINC150 |
|---|---|---|
| 0.1: mini-batch SGD, weight decay 1e-4 | 88.46% | 95.43% |
| SGD, no weight decay | 90.55% | 96.30% |
| Adam from zero, lr 0.05, no weight decay, 100 epochs | 92.35% | 96.67% |
| The same, starting from label centroids and label names | 92.85% | 96.90% |
| Adam from zero, 400 epochs | 92.95% | 96.57% |
| L-BFGS, best weight decay we tried (1e-5) | 92.75% | 96.67% |
Dropping weight decay was about half of the gain on BANKING77, and Adam gave most of the rest. L-BFGS, 400 epochs and label smoothing landed within 12 dev answers of each other. On one seed those gaps are too small to rank, so we kept the 100-epoch run. Starting from the centroids decidekit already computed kept its head start on small data, which Adam from zero lost on CLINC150 at 5 per label (91.36% to 90.12%).
3. Small data keeps SGD
Even with that start, Adam fell 0.29 points on CLINC150 at 5 per label, over our limit. So 0.2 uses Adam from a mean of 8 examples per label and keeps SGD below that. We measured 5 and 10 and never the cutoff itself, so 8 is a judgement call.
| Test accuracy, mean of 3 seeds | Per label | 0.1 | 0.2 |
|---|---|---|---|
| BANKING77 | 10 | 84.43% | 87.52% |
| CLINC150 | 10 | 92.16% | 92.50% |
| HWU64 | 10 | 85.50% | 85.38% |
| All three | 5 | unchanged | unchanged |
What it costs
- Training time: in Node on cached embeddings, BANKING77 went from 84 s to 123 s and CLINC150 from 167 s to 332 s, on a machine that was busy with other work.
- Page stalls: none that matter. In Chrome on a Ryzen 7 260 laptop, training BANKING77 end to end took 208 s, with a longest stall of 19 ms.
- Size: none. The head has the same shape.
- Calibration: no cost with all the data. Error stays at 0.019 or below on all three datasets.
- Thresholds: retrained heads score differently, so a
minConfidencechosen on 0.1 won't carry over. The CLINC150 out-of-scope threshold moved from 0.9 to 0.98. Saved heads keep working unchanged.
What this doesn't show
- HWU64 fell at 10 per label. The loss is 0.12 points, 4 answers across 3 seeds of 1,076. Our pass mark said "doesn't fall" without saying whether that covered small data, so we read this as a miss we accepted. One HWU64 seed at 10 per label also had a calibration error of 0.034, over the 0.03 limit. 0.1's worst seed there was 0.048.
- All-data rows are one seed. The paired test covers the test set, not the variance between training runs.
- We tuned on the benchmarks we report. Dev splits were separate from test, but BANKING77 and CLINC150 still shaped the choice. HWU64 is the only dataset that played no part, and its gain is the smallest.
- It isn't a win over jimothy. Our 93.1% and its 92.66% use different splits and encoders, and the settings are its settings. It trains in Node and suits you if you train ahead of time.
- 5 per label is unchanged. A contrastive layer might help there. We didn't run it, because it would add about 590 KB per head (an estimate) and the head change had already cleared the bar.
Run it on your own labels and tell us what moved, in Discussions:
npm install decidekit@latest