Harness
The harness lied, and every early result was an artefact
The first loop produced a model at 82% frame accuracy that played badly, and a teacher worse than random. An
audit by three separate agents (emulator, data, code) found four bugs:
- x was read from the camera scroll, so distance meant nothing;
- facing flipped mid-round, so "forward" walked away;
- time-overs were scored as draws with made-up damage;
- parallel workers replayed identical matches.
Fix Delete the data. Replay recorded ROM traces in tests, and run a 30-check
acceptance suite on the real ROM that writes a stamp. Collection and play refuse to start without a stamp
matching the ROM, the RAM map and the harness code. Every later fix came with a test that failed first.
Timing
The emulator and the model disagreed about time
Headless Mesen skips rendering frames, so a screenshot could lag the RAM by 0 to 3 frames, differently every
run. And a slow model call could change the fight.
Fix --snes.disableFrameSkipping=true, and lockstep: the Lua bridge
blocks at every input poll until Python sends the next command.
Scoring
"Who won" is harder than it looks
Life bars drain for frames after a hit, a KO blow wraps the life byte below zero, and a time-over zeroes both
bars about 480 frames later.
Fix Stop inferring. Read the ROM's own round-result byte (0x1ACF),
and book damage from the true-life byte on the frame the hit lands.
Data
Not every frame is a decision
While she is hit, thrown or knocked down the stick does nothing, so labels there are noise. Block stun is the
exception: holding down still works.
Fix A controllable flag keeps those frames in the score and out of
training, with a reaction byte to tell block stun from hit stun.
Measurement
Noise hides real progress
Per-round damage has a standard deviation near 30 points. Frame accuracy and validation loss did not predict
play strength, as the laya-vision authors had also found on Atari.
Fix One gate for everything: 20 paired matches (the same openings for every
policy), net damage per round, and "better" only past 2 combined standard errors.
Teacher
The teacher was the bottleneck
Rules went in one at a time, and almost every one had to pass the gate to stay. Rules for fancier moves
(sweep, Lightning Legs) failed and were left out.
| Rule | What it did |
| Jump in from mid range | First teacher better than random |
| Kick near the top of the jump | Hits 69% of the time just before the apex, against 11% on the way up |
| Crouch-guard his attacks | Teacher at +62.0 |
| Walk in and throw | Teacher at +104.5, 40 / 40 rounds |
| Fierce punch for close anti-airs | +105.5, kept although the gain was below the bar |
Surprise How the teacher plays while collecting matters as much as its rules.
Sampling its soft distribution played at about +12; its top choice plus 10% random played at +91 and made far
better data.
Imitation
Imitation copies the wrong thing
r0 learned "repeat my last move": 94% of its disagreements with the teacher repeated its own previous
action.
Fix One round of DAgger cut those mistakes from 17% of decisions to 5%, because
the teacher labelled the situations laya itself got into.
Speed
Measured, not assumed
- Cached vision features: 2.4× faster training steps.
- 256 px instead of 512: 1.34× faster training, 1.9× faster play.
- bf16 on MPS rejected: 10% faster play, only 69% agreement with fp32.
- 4 play workers: ~20 decisions/s on an M4 Pro, 27 on an M4 Max.
Portability
Moving machines exposed hidden assumptions
The feature cache indexed images by absolute path, so data copied to a new Mac crashed training. Mesen died
silently when started while the screen was locked: macOS reports zero displays.
Fix Key the cache relative to the data folder, with a test that moves a data
folder first. Re-run the speed benchmarks on each machine before choosing worker counts.
Arcade
Playing the real game, not a savestate
The video runs arcade mode from power-on: intro, menus, next opponent, bonus stages, continues. The ROM is
deterministic, so a fixed input script always reaches the same fight, and screen bytes in RAM say when a fight
is ready.
Fix A model that always takes its top move plays identical fights in every
window, so each game starts each fight after a different idle delay.