Engine comparison: gorge-compliance¶
The comparison against other engines lives in a separate public repository,
gorge-compliance. It pins one
gorge commit (GORGE_PIN), depends on gorge's public packages only, and plays
its corpora through the oracle runner engine/rules exports
(RunOracleScenarioJSON, RunOracleAuditScenarioJSON,
CheckOracleScenarioJSON). gorge never imports it. Its
oracles page
says what each check sees and cannot see.
flowchart LR
subgraph tiny["Short scripted scenarios"]
hs["Hand scenarios"]
xp["XMage ports"]
xv["XMage verdicts"]
fa["Forge adjudication"]
end
subgraph full["Full games"]
xe["xengine sweep"]
end
mg["mtgish<br/>definition cross-check"]
tiny --> g["gorge at the pin"]
full --> g
mg --> g
xe -.->|"XENGINE_DIRAC=1"| d["dirac"]
What agreement means¶
Two engines agreeing is evidence, not proof. gorge compiles Forge's card scripts, so gorge and Forge can share a script bug. dirac lowers gorge's own compiled IR, so it shares gorge's reading of the rules. XMage is written by hand and shares neither. A ruling that gorge is wrong rests on the printed card or the Comprehensive Rules, never on a vote of engines.
The checks¶
| Check | Compares gorge with | Blocks CI? |
|---|---|---|
| Hand scenarios | Oracle text and the CR, as an author read them | Yes |
| XMage ports | XMage's own test assertions | Yes |
| XMage verdicts | XMage's engine, one generated scenario per card or ability | Yes, on committed verdicts |
| Forge adjudication | Forge's engine, on the same scenarios | No |
| xengine | XMage and Forge, on windows and prefixes of full games | No |
| mtgish | An independent encoding of each card's Oracle text | Yes |
Scenarios and verdicts¶
A hand scenario is one JSON file per card. It sets up a board, takes steps through gorge's runner, and asserts what a player can observe: zones, life, P/T, counters, the stack and the options offered. It is written from the printed card, not from gorge's script.
For XMage, oraclediff gen writes scenarios from a set's manifest. Level A is
one per card. Level B adds one per activated ability, trigger, static and
attack or block. A Java driver (oracle/xmageoracle) replays each in XMage,
and oraclediff diff compares the two engines' snapshots at each checkpoint.
A verdict's status is one of agree, gorge_wrong, xmage_wrong (with a
ruling that cites the rules), diverge (not triaged), harness (the driver
could not express it) or xmage_lacks. Only agree and xmage_wrong count
toward a set's claim.
Each set is declared at level A or B in oracle/compliance/declared.json, and
ratchet.json holds each set's outstanding count so it can only fall. Run
make compliance-status for the declared sets and their level counts.
Current per-set numbers are on the support page.
Neither level says gorge is right about a whole game. Scenarios are short, and interactions between cards go untested.
The four-way full-game harness (xengine)¶
xengine plays whole games, then asks the other engines whether they agree.
- Play. gorge plays games with an exploring bot between two-colour decks drawn from a set's supported cards, weighted toward the cards least exercised so far. A coverage ledger records which cards each sweep played and compared.
- Lower. Each game is cut into windows (a position plus a few decisions) and prefixes (the game from the start to turn N). Each becomes a tape that a follower can replay. Gorge's own runner must first reproduce the lowering. Windows it cannot lower are refused, with a reason.
- Follow. XMage and Forge each replay the tapes. XMage is driven by the Java driver. Forge is driven by a harness in our Forge fork. Their output is never committed, because Forge is GPL-3.0. Any engine can join as a follower: Bring your engine is written for their authors.
- Diff and classify.
xengine diffcompares each follower's snapshots with gorge's stored result and gives each item a class, written as a repro directory:agree,diverge,rng,projection_gap,decision_shape,payment_choice,forge_skips_damage_step,harness,not_sent,turn_cap,xmage_lacksandlower_gap. A follower that cannot express a position (decision_shape) is not an engine bug. - Adjudicate.
xengine adjudicatecombines the two followers' diffs into a three-way pattern per item. - dirac as a fourth side. With
XENGINE_DIRAC=1, decks hold only cards dirac plays, and each game first runs in lockstep between gorge and dirac. Each item then also gets dirac's answer:D=G,D!=G, orD-where dirac has none.
Disagreement classes¶
Letters name the engines: G gorge, F Forge, X XMage. A bar separates groups that disagree. Below, a filled dot sides with at least one other engine and a ring stands alone.
| Pattern | gorge | Forge | XMage | Label | Usual meaning |
|---|---|---|---|---|---|
AGREE3 |
agrees | agrees | agrees | nothing to do | |
FX\|G |
alone | agrees | agrees | gorge_suspect |
a likely gorge bug |
GF\|X |
agrees | agrees | alone | xmage_suspect |
an XMage bug, or a script bug gorge and Forge share |
GX\|F |
agrees | alone | agrees | forge_suspect |
a Forge bug |
G\|F\|X |
alone | alone | alone | three_way |
read the rules by hand |
Combined with dirac: AGREE3 D!=G points at dirac. FX|G D=G is a gorge suspect
that dirac shares.
Nothing here rules. A GF|X item yields a note, never an xmage_wrong ruling.
A sweep commits nothing: its findings are leads for a person.
From a finding to a fix¶
- Triage. The adjudicator writes a ticket draft per
FX|Gcluster. A person reads the printed card and the CR and decides who is wrong. - Corroborate. A gorge suspect is checked against both other engines before anyone touches gorge. If two engines agree against gorge, a reading of the rules still decides.
- Fix gorge first. If gorge is wrong, the fix lands in gorge. dirac follows in the same change, or the card is refused by name.
- Pin. gorge-compliance moves to the fixed commit by a pin-bump pull request, and the stale ratchet rows show what the fix closed. See Pinning.
- Upstream. If the other engine is wrong, a person confirms three things before it is listed as a bug: the printed card or the CR says what should happen; the cause is read in that engine's source at its current upstream branch; and a replay reproduces it with the harness ruled out. It is then reported to the upstream project, and a Forge script that disagrees with the printed card is also fixed on our fork. The confirmed list, with upstream links and states, is in upstream-bugs.md.
A pattern such as GF|X is a suspect, and does not go on that list by itself.
Known blind spots¶
- Windows the harness cannot lower. A sweep of two FDN games on 2026-11-02 refused 25 of 46 windows (from the repository's oracles page).
- Forge skips the second damage step when no creature is left in combat. Those
items get the class
forge_skips_damage_stepand are not compared past that checkpoint. - Rare cases where XMage and Forge agree with each other against the rules, which are read from their code and not fixed in gorge.
- XMage answers an unexpected choice with its AI, so a scenario can flip on a rerun. Replay twice before believing a flip.
Running it¶
See the repository README for setup. The commands you need most: