Skip to content

The policy target, and learning instant timing

Two A/B tests, 2026-10-24 → 10-25

Two pre-registered A/Bs on the from-scratch MLP self-play loop (sploop, the f84x line). The first asked whether the policy target was the bottleneck. The second asked whether self-play could learn when to cast instants without losing strength. Neither changed the network; both changed what the network is trained on.

Every gate is 4,000 games (2,000 pairs, seats swapped), az:sims=100 on both sides, seed 78, the eval split of pool17k. The 95% half-width at 4,000 games is ±0.0155.

  • not adopted Policy target

    cqraw is level with cq at g9 (0.50475), and PCR is level (0.5016) for about 1.45× the self-play compute.

  • adopted React + held casts

    Strength unchanged (0.4965); opponent-turn share 20.9% → 35.2%, against the human 49.6%.

Policy-target quality

Question

Every sploop generation trained with the cq policy target and a value target blended from outcome and root value. On the g8 pack (40k states) the trained prior already agreed with the visit counts: KL(visits‖prior) 0.076, and the top-1 visit matched the prior's top-1 87% of the time. The cq target rescales completed Q min-max per state, which makes it about 0.93 one-hot. 39% of states have a root Q span below 0.02; there the target's KL is 0.62 and its top-1 flips 39% of the time, so the net is trained hard on noise. Two fixes were tried:

  • Q, cqraw: completed Q without the per-state rescale.
  • P, PCR: playout-cap randomisation, which existed in the kernel's search (FullProb/CheapSims) and had never run in a loop: sims=600:pcr=0.25:cheap=100.

Setup

  • Every arm used the same pinned play and pack binaries; arm Q ran on a later trainer build.
  • Parent: sploop g8, warm start, 6 generations with equal self-play games per generation across arms, and 1,000-pair gates.
  • Arms:
    • C: 200 sims.
    • P: 600 sims with :pcr=0.25:cheap=100, recording every decision. The plan expected ~3% more wall time; measured, P's self-play ran at 4.2 games/s against C's 6.1, about 1.45× the compute.
    • Q: 200 sims, cqraw.
  • Rule, fixed before launch: adopt X only if X6 vs C6 ≥ 0.5155, X6 vs g8 ≥ C6 vs g8, and X's vs-bot at g5 and g6 is no more than 0.02 below C's. Reject at X6 vs C6 ≤ 0.4845. Otherwise extend X and C to 9 generations and gate X9 vs C9 once, adopting at ≥ 0.5155. Loss and KL are logged and never decide.

Results

Gate Score (4,000 games, ±0.0155)
C6 vs g8 0.51625
Q6 vs C6 0.497375
Q6 vs g8 0.504625
P6 vs C6 0.50725
P6 vs g8 0.514875
Q9 vs C9 0.50475
P9 vs C9 0.501625

Both g6 gates fell in the inconclusive band, so both arms ran to g9. vs bot stayed between 0.92 and 0.94 in every arm (P: g7 0.932, g8 0.928, g9 0.935).

Verdict

Neither adopted. The loop keeps cq and no PCR. The calibration fact stands, since the 39% of near-tie states do get the largest cq updates, but fixing it does not move play. The plan names auxiliary heads and a TD(λ) value target as the next levers.

The react rule and held-cast recording

Question

DraftZero's experiment 48 ("learning when to cast instants") found that the rule at play alone does nothing: here, the instwin head-to-head gave 0.5035 / 0.498 (instant timing); for DraftZero, 49.1%. Its gain came from recording held casts as zero-visit candidates. The test asked whether self-play with :instwin=react, with held casts recorded, moves instant timing toward human play without losing strength.

Setup

  • Code: the react rule and held recording in the self-play driver, a +held pack format (a held-candidate column), and a trainer change: held candidates get 0 in the visits, cq and cqraw targets, and sims-0 records skip the value blend.
  • Warm start from the policy-target test's winner. Neither P nor Q won, so the winner is arm C, starting from its g6.
  • Arms: C2 is the winner's recipe exactly; R2 is the same recipe with :instwin=react in the self-play seats only. No rule is active in any gate. 6 generations, equal games per generation, 1,000-pair gates.
  • The first launch failed because the react binaries lacked the v2 entity width; it was relaunched.
  • Litmus: each net against itself with no rule at play, az:sims=100, 1,000 pairs, seed 78, eval split, with the instant census on.
    • L1: the opponent-turn share of instant-speed casts.
    • L2: own upkeep and draw-step instant-speed casts per 100 seat-games.
    • Human reference: 49.6% overall.
  • Rule, fixed before launch: adopt only if (S) R6 vs C6 ≥ 0.4845, (T1) L1(R6) ≥ L1(C6) + 5.0 points, and (T2) L2(R6) ≤ 0.5 × L2(C6).

Results

Measure Winner g6 C6 R6
R6 vs C6 (4,000 games) 0.4965
vs the winner's g6 (4,000 games) 0.511625 0.516875
L1 opponent-turn share (human 49.6%) 19.7% 20.9% 35.2%
L2 own upkeep + draw casts per 100 seat-games 95.65 87.20 4.10
Casts on the opponent's turn: control C6 → react R6 C6 (control) R6 (react + held) 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 17Lands FDN humans: 49.6% of instant-speed casts on the opponent's turn humans 49.6% all instant-speed casts all instant-speed casts, C6: 20.9% all instant-speed casts, R6: 35.2% Stab Stab, C6: 30.1% Stab, R6: 35.2% Luminous Rebuke Luminous Rebuke, C6: 15.7% Luminous Rebuke, R6: 35.0% Faebloom Trick Faebloom Trick, C6: 5.1% Faebloom Trick, R6: 29.4% Goblin Surprise Goblin Surprise, C6: 7.1% Goblin Surprise, R6: 22.0% Giant Growth Giant Growth, C6: 25.6% Giant Growth, R6: 32.5% Refute Refute, C6: 96.6% Refute, R6: 89.2% share of the card's instant-speed casts made on the opponent's turn
Card C6 R6
Stab 30.1% 35.2%
Luminous Rebuke 15.7% 35.0%
Faebloom Trick 5.1% 29.4%
Goblin Surprise 7.1% 22.0%
Giant Growth 25.6% 32.5%
Refute 96.6% 89.2%

Verdict

Adopted (react + held). S passes (0.4965 ≥ 0.4845), T1 passes (35.2% ≥ 20.9% + 5.0), and T2 passes (4.10 ≤ 43.6). Strength is unchanged and the timing moves well toward humans, though opponent-turn share is still 35.2% against the human 49.6%.

Caveat

The react test ran with the v2 entity block zeroed in both arms, because the react binaries' kernel feature width was 84. The relative result stands; absolute strength is not comparable with the policy-target test or sploop.

The next step this leads to, separate policy and value windows and a league buffer, is in the open questions.