Qwen3.8-27B · Othello · a local model, learning to think out loud
Part Three
Part One taught the model to see the board; Part Two watched it write rules for the practice opponent. This part is the turn underneath both — the day I realised I had spent half the run measuring a model that wasn't allowed to reason at all, and what it took to fix that on one machine.
Pinocchio, the practice foe, was finished 47–17 — by a depth-six search, not the model, which never met him. So I put the model against a real opponent that looks ahead: a depth-three alpha-beta player. Nine games.
Nine games in play order, disc margin (amber won, rose lost). It steals one in three; the losses run to sixty. A player that reads the board once and commits does not hold up against something that searches.
This is the correction I keep coming back to, because it invalidated so much at once. Qwen can reason before it answers — but that mode was off, and I hadn't clocked how much it mattered. With reasoning off, asked for a legal, rule-following move, it managed it ten times out of twenty-two. With reasoning on: twenty-two out of twenty-two.
Every "it can't do this" I had written down until then — every capability I thought I was measuring — had been measured against a model that wasn't allowed to think. I threw those findings out and started again. That is a harness failure of the worst kind: not a wrong number, but a confidently right number about the wrong thing.
Thinking has a dial, and it is not "more is better". Too little and it falls back to a guess; too much and it rambled past the answer on seven turns in twelve, spending the whole budget without ever naming a square. Medium was the only setting that reliably finished — no overruns, and games that took twenty-six minutes instead of two hours.
Turning reasoning on is a flag. Making it practical — a 27-billion-parameter model thinking fast enough to play hundreds of games, on local hardware — was its own project. Speculative decoding to claw the throughput back. A context window long enough to hold a full reasoning trace without truncating mid-thought. And the server going down, over and over, under calls that ran for minutes at a time.
None of it is Othello. All of it decided whether any of this could be measured at all.
Read it next: running Qwen3.8-27B on oMLX →Once it could think and I could trust the board, the question the whole run had been built around settled fast — and not the way it had assumed. Twenty fixed positions, the model's move under different prompts, each graded by the search. Mean regret, discs lost against it, lower is better.
Every annotation made it worse; its own rules were the worst of all, more than doubling how often it blundered outright. It runs a small search in its head, and a handed number just pulls it off the board and onto the number. One discipline ran underneath all of it and never got easier: the search that grades a move may never appear in the prompt, or the model would be copying the answer key instead of earning the move — keeping that leak sealed was a fight of its own.
The instrument's finest hour was proving the instrument's own premise wrong.
The honest answer to the question Part Two left open: for eight days the harness maintained a rule library the model was better off without — and for the first few of those days it was grading a model that wasn't even thinking. The model was not the student. I was: learning how it fails, how to make it reason, and how to measure it without fooling myself.
A harness earns its verdict by being unable to cheat toward it.
So the rules come off. The thing worth improving is the one a prompt cannot reach — the model's own reasoning — and you do not deepen that with notes on the side. You change the model itself. That is the next page again, and a different repository: stop teaching the prompt, and start teaching the weights.
Part Three · the nine searcher games and the 2–62 loss are measured disc counts from the run archives (r345, r347, r349, r350). The thinking-off vs thinking-on figures and the six-prompt comparison were scored with learning off; each move graded by a depth-6 alpha-beta search (regret = discs lost against it, lower is better). Every figure is a disc count, never the model's account of itself. Read from the start →