Skip to content

An LLM decision policy

QLoRA SFT, DPO and a win-rate gate, 2026-10-08 → 10-10

Can a fine-tuned language model, trained on search-teacher labels, pick better than the heuristic bot on the decisions where search disagrees with it? And does blending its preference into the search win games?

Verdict

Not adopted. Per-decision gains were small, and the blend's win-rate gate is null: +0.48 points against a pre-registered +3 point bar. Nothing landed on main.

  • +0.0019 value per decision for DPO over the bot, ±0.0021
  • 50.48% for search + LLM blend against plain search, over 7,100 games
  • 87% of held-out decisions where SFT matched the bot

Setup

  • Base model Qwen/Qwen3.5-9B, a LoRA fine-tune with answer-only loss.
  • Labels from gorge's search teacher (label-v1 records). An unmerged Go branch added world rescoring, a deck catalog and holdout flags to it.
  • Data: 11,192 decisions from 3,000 games over 10 mono pairs.
  • Preference pairs for DPO: value = (8×Q8 + 64×Q64)/72; candidates ≥ 0.06 below the best are rejected (at most 2 per decision); 8,634 pairs.
  • Fresh-seed test set t1: seed 60,000,000, 2,370 decisions. SpellBench split: a catalog over 261 pairings; held-out decks fdn07-ur, fdn12-ugw, faeries, wildfire, uw-control, the-epic-storm.

Results

Units are value per decision (the 64-world Q of the picked candidate minus the bot's), with paired 95% intervals. The write-up warns that these intervals are too narrow, because decisions within one game are correlated.

How much is there to gain. On c1 (11,129 rescored decisions) the 8-world search's overrides of the bot gain +0.042 ±0.005 where they occur (2,501 decisions). Over all decisions the 8-world search gains +0.0094 ±0.0011, and a 64-world oracle ceiling is +0.0283 ±0.0012.

SFT on 539 held-out decisions matched the bot 87% of the time and the search pick 71.2% (always-bot: 77.7%).

2,370 decisions:

Kind n search − bot DPO − bot
casting 1,271 +0.0104 ±0.0038 +0.0065 ±0.0027
attackers 521 +0.0034 ±0.0029 −0.0046 ±0.0044
blockers 160 −0.0004 ±0.0071 −0.0132 ±0.0113
targets 418 +0.0053 ±0.0053 +0.0016 ±0.0044
all 2,370 +0.0072 ±0.0024 +0.0019 ±0.0021

DPO − SFT: +0.0036 ±0.0016. Blending, Q8 + λ(log π_cand − log π_bot): λ 0.02 (fixed in advance) +0.0006 ±0.0017; λ 0.01 (chosen after) +0.0016 ±0.0015; λ 0.05 −0.0016 ±0.0020.

34,625 training decisions; sbt = 1,242 rescored held-out-deck decisions:

Policy sbt all sbt casting t1 all t1 casting
SFT +0.0001 ±0.0016 +0.0008 ±0.0020 −0.0010 ±0.0013 +0.0016 ±0.0012
DPO +0.0018 ±0.0020 +0.0029 ±0.0027 +0.0045 ±0.0022 +0.0094 ±0.0031
search +0.0007 ±0.0028 — +0.0072 ±0.0024 —
blend 0.02 − search +0.0045 ±0.0024 +0.0064 ±0.0036 +0.0020 ±0.0017 +0.0037 ±0.0027

Win-rate gate. Pre-registered: λ 0.02, 7,100 games on the 71 held-out pairings, base seed 90,000,000, bar +3 points. Search + LLM blend against plain search: 3584-3516 of 7100 = 50.48%, Wilson 95% [49.32, 51.64], +0.48 points; seed-clustered 50.48% ±0.34 over 3,550 seeds. The blend changed the search's pick on 1,426 decisions. 926 games had a seat whose observation the bridge did not support.

The gate's result survives only in the run's output in a working log; the write-up and the pre-registration report predate it.

Data

The working directory and the bridge directory are gone. Surviving: the write-up (kept privately), two unmerged gorge branches (the rescoring flags and the bridge), and the gate's output.