Lessons learned · a hobby project

laya-vision vs Street Fighter II

A 256M-parameter vision model looks at two screenshots and picks Chun-Li's next move, once every 4 frames. After imitation and two rounds of DAgger it wins 40 of 41 rounds against Dhalsim.

40 / 41rounds won, laya r2
+93.1net damage per round
14moves to choose from
Chun-Li's standing kick catches Dhalsim's drill in mid-air on the elephant stage Watch on YouTube
Why

Two things I wanted to learn by doing them

A fast visual decision

Fine-tune a small model on a problem it has never seen

A "System 1" decision: one look at the screen, one choice, no reasoning chain. A fighting game is a good test because the right answer changes every quarter second and you can't fake it.

The harness

Build what makes tuning possible

Understand the problem domain well enough to measure it, then gate every change on real play, like a regression test. This turned out to be most of the work, and all of the value.

The model

What laya-vision is

laya-vision is a research fork of Laya's typed-decision model. It puts a SmolVLM-256M image backbone in front of a decision head that answers predict(state, questions) with a probability for every option. It is PyTorch on Apple MPS, and the base checkpoint was trained on photo questions. It knew nothing about games.

Here the question is "which of these 14 moves now?". The state is two frames, 4 frames ago and now, plus one line of text read from the game's RAM:

me=chunli stand hp=100 opp=dhalsim attack hp=100 dist=mid facing=right corner=none time=early last=forward fireball=none
Mesen 2 · SF2 ROM Screen frame t−4 and frame t Work RAM life, x / y, states, clock laya-vision · SmolVLM-256M + decision head Vision tower frozen · features cached Trained Text layers + LoRA adapters, rank 16 question: which move now? 14 options, each described Decision head P(move) × 14 Action macro hk = R ×2, release ×2 forward/back from facing sf2_bridge.lua applies input frame by frame 2 images image features the one-line RAM note hk 62% per-frame buttons lockstep: the game waits for Python at every decision, so model speed never changes the fight
One decision, from pixels to buttons. The only part that learns is the LoRA on the text layers (blue). The vision tower is frozen, which is why its features can be computed once per image and cached.
Training

A teacher, a student, and one gate

A scripted teacher reads RAM and labels every frame with a soft distribution over the 14 moves, whoever is playing. laya first copies the teacher's own play. Then laya plays, and the teacher labels laya's own frames (DAgger). Every round is judged by play, never by training loss.

ROM harness 30 checks on the real ROM write a stamp Teacher plays top choice + 10% random 37,879 decisions Dataset 2 frames + note + teacher's soft label Train LoRA 256 px, 2 epochs cached vision features laya plays 20 paired matches vs Dhalsim Gate net damage / round up by 2 SE? DAgger teacher labels laya's own frames stop, keep the best no stamp, no run better, or the first miss rows join the data two misses in a row
The loop that ran on the Mac mini: r0 took 135 minutes to train, and each DAgger round about 100 minutes plus 12 to 15 minutes of gate play. The dashed box runs first: collection and play refuse to start on an unverified harness.
−50 0 +50 +100 random: −56.0 ± 7.4 net damage per round, 3 of 42 rounds won random3 / 42 rounds −56.0 laya r0, imitation only: +67.7 ± 12.1, 39 of 49 rounds won r0 · imitation39 / 49 rounds +67.7 laya r1, DAgger round 1: +80.6 ± 9.6, 39 of 46 rounds won r1 · DAgger 139 / 46 rounds +80.6 laya r2, DAgger round 2: +93.1 ± 8.3, 40 of 41 rounds won r2 · DAgger 240 / 41 rounds +93.1 scripted teacher: +105.5 ± 6.8, 40 of 40 rounds won scripted teacher40 / 40 rounds +105.5
Net damage per round against Dhalsim (damage dealt minus taken; whiskers are ± 1 standard error, from 20 paired matches per policy). laya is blue, the teacher it copies is gold. r2 is within noise of the teacher.
What went wrong

Problems I hit, and what solved them

I don't expect anyone to run this: it needs the ROM, Mesen and an Apple-silicon Mac. This part is for anyone pointing a model at a similar problem.

Harness

The harness lied, and every early result was an artefact

The first loop produced a model at 82% frame accuracy that played badly, and a teacher worse than random. An audit by three separate agents (emulator, data, code) found four bugs:

  • x was read from the camera scroll, so distance meant nothing;
  • facing flipped mid-round, so "forward" walked away;
  • time-overs were scored as draws with made-up damage;
  • parallel workers replayed identical matches.

Fix Delete the data. Replay recorded ROM traces in tests, and run a 30-check acceptance suite on the real ROM that writes a stamp. Collection and play refuse to start without a stamp matching the ROM, the RAM map and the harness code. Every later fix came with a test that failed first.

Timing

The emulator and the model disagreed about time

Headless Mesen skips rendering frames, so a screenshot could lag the RAM by 0 to 3 frames, differently every run. And a slow model call could change the fight.

Fix --snes.disableFrameSkipping=true, and lockstep: the Lua bridge blocks at every input poll until Python sends the next command.

Scoring

"Who won" is harder than it looks

Life bars drain for frames after a hit, a KO blow wraps the life byte below zero, and a time-over zeroes both bars about 480 frames later.

Fix Stop inferring. Read the ROM's own round-result byte (0x1ACF), and book damage from the true-life byte on the frame the hit lands.

Data

Not every frame is a decision

While she is hit, thrown or knocked down the stick does nothing, so labels there are noise. Block stun is the exception: holding down still works.

Fix A controllable flag keeps those frames in the score and out of training, with a reaction byte to tell block stun from hit stun.

Measurement

Noise hides real progress

Per-round damage has a standard deviation near 30 points. Frame accuracy and validation loss did not predict play strength, as the laya-vision authors had also found on Atari.

Fix One gate for everything: 20 paired matches (the same openings for every policy), net damage per round, and "better" only past 2 combined standard errors.

Teacher

The teacher was the bottleneck

Rules went in one at a time, and almost every one had to pass the gate to stay. Rules for fancier moves (sweep, Lightning Legs) failed and were left out.

RuleWhat it did
Jump in from mid rangeFirst teacher better than random
Kick near the top of the jumpHits 69% of the time just before the apex, against 11% on the way up
Crouch-guard his attacksTeacher at +62.0
Walk in and throwTeacher at +104.5, 40 / 40 rounds
Fierce punch for close anti-airs+105.5, kept although the gain was below the bar

Surprise How the teacher plays while collecting matters as much as its rules. Sampling its soft distribution played at about +12; its top choice plus 10% random played at +91 and made far better data.

Imitation

Imitation copies the wrong thing

r0 learned "repeat my last move": 94% of its disagreements with the teacher repeated its own previous action.

Fix One round of DAgger cut those mistakes from 17% of decisions to 5%, because the teacher labelled the situations laya itself got into.

Speed

Measured, not assumed

  • Cached vision features: 2.4× faster training steps.
  • 256 px instead of 512: 1.34× faster training, 1.9× faster play.
  • bf16 on MPS rejected: 10% faster play, only 69% agreement with fp32.
  • 4 play workers: ~20 decisions/s on an M4 Pro, 27 on an M4 Max.
Portability

Moving machines exposed hidden assumptions

The feature cache indexed images by absolute path, so data copied to a new Mac crashed training. Mesen died silently when started while the screen was locked: macOS reports zero displays.

Fix Key the cache relative to the data folder, with a test that moves a data folder first. Re-run the speed benchmarks on each machine before choosing worker counts.

Arcade

Playing the real game, not a savestate

The video runs arcade mode from power-on: intro, menus, next opponent, bonus stages, continues. The ROM is deterministic, so a fixed input script always reaches the same fight, and screen bytes in RAM say when a fight is ready.

Fix A model that always takes its top move plays identical fights in every window, so each game starts each fight after a different idle delay.

If you try something similar

What I'd tell you

Still open

Where it stops

It only knows Dhalsim

Every gate so far is against Dhalsim. In short arcade runs it has both beaten and lost to Ryu. Next: gate it against each opponent, then train across all of them.

Not quite real time

About 72 ms per decision against a 67 ms budget on the M4 Pro. Lockstep keeps training exact; live play runs slightly slow.