Lessons learned · part 2

Two systems vs Street Fighter II

laya-vision + text laya + Qwen

Part 1 ended with a small vision model that only knew Dhalsim. Part 2 adds a second system that learns between rounds: a big language model reads her games and writes advice, and a small model follows it. Some of this worked. The part I expected to be smartest did not.

+29hit points / round: a short memory counted from her own games
+32a playbook from other opponents, even ones she never met
0.40 → 0.99text laya learns to follow advice
Part 2 video: two systems vs Street Fighter II Part 2 on YouTube
Why

Can a big model coach a small one?

The idea is the old one about fast and slow thinking. A fighter doesn't reason about every punch. She reacts. Between rounds she thinks: what hurt me, what worked, what to try next.

So I split the player in two and asked one question: can a language model, reading her game logs, write advice that makes her win more?

System 1 · fast

Sees and acts, every decision

laya-vision looks at the screen and rates every move. text laya reads the situation and the current advice, and picks one. No reasoning chain, about 12 ms.

System 2 · slow

Reflects, between rounds

Qwen reads the game log and rewrites her memory: a short memory for this opponent and a long-term playbook. It runs in the background, so the game never waits for it.

The systems

Three models, one loop

System 1 · every decision Screen 2 frames, 4 apart HUD blanked sees laya-vision rates every move: likely works · may work · likely fails Shortlist top 3 + moves the memory names + forward decides text laya reads the moment + short memory, picks one ~12 ms Game Mesen 2, SNES Chun-Li vs CPU System 2 · between rounds, background thread Game log every decision and round reflects Qwen lost round → revise short memory game over → rewrite playbook async: the game never waits Memory playbook (long-term) short memory (this opponent) only Qwen writes it names moves short memory
Blue boxes are the two models I fine-tuned; gold is the one that writes advice. laya-vision: SmolVLM-256M with my LoRA, 256x256 native pixels, PyTorch on Apple MPS. text laya: ModernBERT-large (421M) via laya-mlx, with a LoRA trained only to follow advice. Qwen: Qwen3.8 27B, 4-bit oQ4e with MTP, on the omlx server, thinking off.

laya-vision stays general on purpose. It only says what a move would do right now, from the picture and a one-line note. It gets no opponent name and no advice. Everything opponent-specific lives in the memory, and only Qwen writes the memory.

Getting advice to her hands

First: can she even read advice?

My first loop let Qwen write advice for 37 games. She lost every one. When I looked, the advice never reached her hands: "use lp up close" was in most versions of her memory, and she chose lp 0 times. laya-vision had never been trained with advice text.

So before judging the coach, I tested the channel: the same moment with different advice. Does the choice move?

ModelWhat it did with advice
laya-vision, baseAnswered "b" 30/30 whatever the advice said. It just prefers an option position.
laya-vision, my LoRAReacts to extra text, not its meaning: "more a" and "avoid a" both lower a.
text laya, untrainedA keyword matcher: "avoid sweep" → picks sweep 95%. Ignores conditions.
text laya, fine-tunedFollows it: 99.2% on the test set, "avoid" 98%, unseen wordings 98.7%.

The fix: let laya-vision just watch, and let a text model do the choosing. text laya was fine-tuned on 16k rows from all 8 characters' logs, with real and templated Qwen lessons. The labels come from a rule in code, not from outcomes, so it learns to follow advice faithfully. Whether the advice is any good is the coach's problem.

untrained fine-tuned 0%50%100% All test rows2,000 rows 39.8% 99.2% "avoid …" advicenegations 3.0% 98.0% Condition not met"when he jumps", he isn't 49.7% 99.5% Held-out wordingsnever seen in training 40.6% 98.7%
How often text laya picks the move the advice calls for, on 2,000 held-out test rows. Untrained, "avoid sweep" made it pick sweep. Fine-tuned, it reads polarity words (more / avoid / never / always / drop) and conditions (up close, when he jumps).

Wired in, she attacked 30–40% of the time instead of 6%, and in one session won her first rounds ever against Dhalsim, Ryu, Zangief and Ken. That felt like progress. It was not yet proof.

The fair test

Same rounds, only the advice differs

Watching her play and feeling good about it proves nothing. So the test is headless and paired. A savestate locks the opponent. Every arm plays the same rounds with the same start delays. The only thing that changes is the advice.

The score is hit points per round: damage dealt minus damage taken. Each arm is compared with "Advice: none" round by round. I wrote the verdict rule into the script before running it: an arm helps when the 95% confidence interval of its pooled difference is above 0.

Same start savestate locks the opponent same start delay per round 30 rounds per opponent none"Advice: none", the control code_shortcounted from this opponent code_playbookcounted from other opponents qwenQwen's current short memory Score each round hit points = damage dealt − taken Fixed before the run arm − none, per round helps if pooled 95% CI is above 0
One round, four arms. Pairing removes most of the luck: every arm faces the same opening, so a difference comes from the advice. An arm with no advice replays the control exactly; I checked that in a smoke run.

The two code_ arms are the baseline I should have built first. No language model at all: code sorts her moves by net hit points from her logs, keeps the top 3 as "use" and the bottom 2 as "avoid". code_short counts only rounds against this opponent. code_playbook counts only rounds against the other opponents, so it can also play Blanka and Guile, whom she has never met.

Results

Counting helps. Qwen doesn't, yet.

hit points per round vs no advice · mean and 95% CI −40 −20 +20 +40 0 code_short150 paired rounds · helps+28.8 code_playbook210 paired rounds · helps+31.8 qwen150 paired rounds · not shown+1.1 Qwen − countingsame 150 rounds · worse−27.7
Pooled over 5 opponents (7 for the playbook, which also played Blanka and Guile), run rollouts/ab/20260928-091745. Whiskers are 95% confidence intervals. The last row is Qwen's advice minus code's advice on the same rounds: −27.7 [−42.0, −13.5].
Won of 30nonecode_shortcode_playbookqwen
Ken317164
Zangief66144
Ryu610100
Dhalsim89413
E. Honda1120
Blanka (never met)3–11–
Guile (never met)16–16–
A short memory worksAdvice counted from her own rounds against this opponent: +28.8 hit points per round. Against Ken she goes from 3 wins to 17.
A playbook carries overKnowing her own strengths helps even against opponents she never met: against Blanka, 3 wins become 11.
Qwen is the weak linkIts advice is no better than none (+1.1), and 27.7 hit points per round worse than plain counting over the same logs. Against Ryu it won 0 of 30.

Two honest footnotes. Dhalsim is the exception: the playbook hurt there and Qwen helped. And the path from memory to action is fine: text laya follows advice 98% of the time. When the advice fails, the advice is the problem.

Why Qwen fails

It judged moves by how often they miss

Here is what Qwen told her against Ryu, next to what her own logged rounds say. "Net" is hit points per try.

Qwen's lessonThe countsVerdict
avoid c.mk at mid: it missesc.mk@mid: 405 tries, hits 37%, +2.2 netwrong: a winning move, banned for missing
avoid c.hp at mid: it missesc.hp@mid: 109 tries, hits 39%, +2.6 netwrong, same reason
use lp when he jumpslp@close: 574 tries, −3.1 neta losing move, advised
stop spinning_bird_kick−6.5 at mid, −18.4 up closeright
(nothing)throw@close: 100 tries, +10.0 net, her best movemissed

Against Ken the pattern repeats: "avoid c.hp at mid: it whiffs", while c.hp at mid is net positive. Mostly, the causes are in what I gave Qwen:

  1. 1 · Wrong yardstickThe words it could use were lands / whiffs / blocked / punished: rates, not results. A move that misses 60% of the time but wins more than it loses is "whiffs", so it gets banned. Nothing in its input said "net hit points".
  2. 2 · NegativityThe digest led with failures. Its lessons ran 4 avoids to 1 use. Avoiding moves made her passive: with its advice she attacked 59% of decisions, without it 84%.
  3. 3 · Recency and churnRevising after every lost round means learning from the last 60 seconds. 45–71% of the short memory changed per rewrite. "c.mk at mid" went use → avoid → use 4 times in 4 minutes.
  4. 4 · Counting in wordsAn LLM estimating rates from a long digest is worse at arithmetic than code, and it doesn't know which differences are noise.

Code counts, Qwen reasons. That's the next step: every number comes from code, Qwen starts from code's advice, and any change it makes must cite a row of the table.

What went wrong

Problems I hit, and what solved them

Most of these are harness problems, not model problems. As in part 1, that's where the time went.

Process

The 10-minute run that ran 67 minutes

I asked for a 10-minute loop and planned to stop it with kill -INT on a background job. Background jobs in a non-interactive shell ignore SIGINT, so it kept going: 37 games, all against Dhalsim, all lost, 112 memory versions.

Fix A time limit inside the loop (--minutes), and for measuring, a headless A/B with a fixed number of rounds that ends by itself.

Process

Crashes I couldn't see

Twice a job died silently: the first loop (it ran with the system Python, not the project's) and a training launch. Both times I had sent the output to /dev/null. Same mistake, twice in one day.

Fix Always keep the console log. A crash must be loud.

Data

Label bugs, caught before training

A separate, blind reviewer read the text laya label rule and found three bugs before the real run:

  • "when ryu attacks" lost its condition;
  • a condition after the colon was dropped;
  • the same lessons leaked into train and validation.

Fix A failing test for each, then the fix. Cheap compared to a wasted training run.

Coach

The notebook that stayed empty

Qwen's notebook has a change budget per round, so it can't rewrite everything at once. Its updates were bigger than the budget, the check rejected each one, and the notebook stayed empty round after round.

Fix The budget is no longer a reason to reject. Code trims the update to fit, and the rest waits for a later round.

Coach

Qwen wrote things my reader didn't expect

"drop X" wasn't read as negative. A plan line like "at mid and far" names two ranges at once. Gaps like these could crash the game mid-round.

Fix "drop" and "skip" read as negative, two-range lines are dropped with a reason, and a line the reader can't use is skipped instead of crashing the game.

Honesty

A chart that flattered me

My first progress chart had a rolling win line that started at 100%, and it compared different opponents with each other. It looked like learning. It wasn't evidence.

Fix Same opponent only, sample sizes shown, and a luck check: at a ~14% round-win rate, one match win in 16 tries is about 59% likely by chance.

Timing

Stopping waited for Qwen

Qwen reviews in the background so the game never waits for it. But stopping the loop still waited for Qwen's review in progress to finish.

Fix System 2 runs in a daemon thread: the game never waits for Qwen, and neither does a stop. A new memory swaps in at the next round.

If you try something similar

The exact setup, and what I'd tell myself

PieceWhatSettings that matter
GameStreet Fighter II (USA) SNES ROM (not included), sha1 7ddcb96e0d9fea94d9370635262ac7c28da85214, in Mesen 2 (MesenCE 1.0)headless --testrunner or a window; controller 2 plugged in with --snes.port2.type=SnesController; window at 150% speed
laya-visionthaitea/laya-vision-smolvlm-256m + my LoRA; PyTorch on Apple MPS256x256 native pixels, no upscale: the 256x224 frame padded, HUD blanked, nearest-neighbour; two frames 4 apart + a one-line note, no opponent name
text layaaac6fef/laya-mlx (ModernBERT-large, 421M) + my LoRAruns on laya-mlx (Apple MLX), not omlx, in its own venv as a helper process; LoRA r=16; max 512 tokens in; ~12 ms per decision
QwenJundot--Qwen3.8-27B-oQ4e-mtp on the omlx server (OpenAI-compatible)thinking off: a review took ~5 min with it, ~9–11 s without; temperature 0; background thread
Still open

Where it stops

Give Qwen a job it can do

Three stages, each with its own paired test:

  • A. Qwen gets the net-hit-points table and code's advice as a draft. It must at least match code_short.
  • B. Add what counting can't: conditions ("when he jumps"), opponent habits, which strengths to lead with against someone new. It must beat code_playbook.
  • C. Learning over rounds, starting from the code baseline. Over 40+ rounds it must beat a fixed code_short arm in the last third.

She almost never blocks

laya-vision rated block "likely fails" 981 times out of 993 while the opponent attacked, and 93% of the hits she takes land while she is attacking. Advice about attacks can't fix defence.

E. Honda

Nothing helps yet: 1–2 rounds won of 30 in every arm. Maybe the same defence problem.

Selection bias in the counts

"throw@close is +10" means +10 when laya thought a throw fit. Forcing it everywhere may not keep +10. The paired A/B measures the real effect, so I trust it over the table.