01
Identical inputs
Turn inputs are derived once, frozen, and reused across every arm. The setting is the variable; the game is not.
writing/llm evaluation ·
Benchmarking a generative character when there is nothing to diff against
Yapword is a daily five-letter word game for web and iPhone. A persistent AI character reacts to every guess, carries jokes across the board, gives contextual hints, and closes the game with a personalized roast. This is how I tested whether lowering its reasoning budget changed the part players actually experience.
82%inference bill attributed to hidden reasoning
0hidden tokens at the minimal setting
5–10×latency at the higher settings
36hint samples with zero detected leaks
01 / the product boundary
Code owns truth. The model owns expression.
The model never decides whether a guess is valid, which letters match, what the answer is, or whether the player won. A deterministic rules engine owns the board, the score, and the outcome. The model receives structured game context and turns that truth into character.
That boundary makes the product reliable, but it does not make the character easy to evaluate. A funny line on turn two can still lead to a flat game if turn five forgets it. The property at risk is not a single response. It is the through-line.
02 / the production question
Most of the bill was invisible. Removing it was the easy part.
Roughly 82% of Yapword's inference bill was hidden reasoning tokens: billed output that never reached a player. The configuration could be lowered in one line. The real question was whether those tokens were buying continuity, timing, or better jokes.
A single-shot A/B test could not answer that. It would compare two isolated lines while missing whether the character noticed a repeated mistake, referred back to an earlier guess, or escalated the same joke through the closing roast.
the measured property
Does the character preserve a recognizable, game-specific through-line across a complete round when the thinking setting changes?
03 / the evaluation harness
Walk the whole game, not a bag of prompts.
Each arm replays a complete game turn by turn against the production prompt builder. Every successful response joins the context for the next call, followed by the postgame roast. The only swept variable is the model's thinking setting.
Two scenarios stress continuity in opposite directions: a win where one letter remains misplaced until the final guess, and a loss where the player opens with the same wrong letter six times. One offers a story to escalate; the other tests whether the character recognizes a pattern instead of merely reacting to the latest turn.
02
Ordered continuity
Each successful line joins the next turn's context in sequence, so the bench measures the through-line instead of isolated one-liners.
03
Failures stay failures
Transport errors are counted separately from quality samples and never become fake dialogue or a zero score.
04
Usage is reported
Visible and hidden tokens come from provider metadata, with run count and completeness carried beside every summary.
04 / measured sweep
The expensive settings took longer. They were not funnier.
A six-turn game produced roughly 170–216 visible tokens. On the paid thinking arms, hidden reasoning made up 95–98% of billed output and added five to ten times the latency. Higher settings changed the imagery, but the small controlled sweep did not show a clear quality gain.
| Setting | Hidden tokens · win | Hidden tokens · loss | Latency · win | Latency · loss |
|---|---|---|---|---|
| minimal | 0 | 0 | 6.0s | 5.6s |
| low | 4,065 | 5,178 | 24.5s | 29.6s |
| medium | 7,945 | 7,349 | 43.0s | 40.7s |
| high | 8,991 | 11,663 | 48.8s | 60.9s |
the through-line survived
“Your grid is beginning to look less like a strategy and more like a poorly managed petting zoo.”
At the minimal setting, the line connected RANCH to FAWNS and then carried the scene into the next turn. The continuity came from the accumulated context and voice system, not a hidden reasoning budget.
05 / the accuracy-sensitive path
Hints needed a different test.
Most Yapword output is evaluated on taste. A hint is different: the answer is present in model context, and the instruction is what stops the model from revealing it. That creates a concrete failure with an objective check.
The separate benchmark ran three answer profiles across four thinking settings and three repetitions: 36 hints. It detected zero whole-word leaks and zero unrevealed-letter flags at every setting. Minimal thinking produced a hint in 0.9 seconds; high thinking took 8.2 seconds.
36
hint samples
0
whole-answer leaks
0.9s
minimal-setting latency
The screen is deliberately modest. A regex can catch an answer that is stated; it cannot determine whether a hint gives the answer away without naming it. That remains a human review problem.
06 / what the result supports
The result is useful because its boundaries are visible.
Three runs per cell
Hosted inference is not bit-reproducible. These are small samples with a spread, not universal point estimates.
One product, one property
The sweep measures voice continuity under cost pressure. It does not establish factual accuracy or broad model safety.
A human still judges the joke
The harness automates controlled inputs, usage, latency, comparability, and leak screening. Taste remains a human decision on purpose.
The defensible conclusion is narrow: for these two full-game scenarios, at three samples per cell, additional hidden reasoning did not improve the character's multi-turn voice enough to justify its token and latency cost. That was enough evidence for a production decision. It is not a claim about every model, character, or task.
07 / public artifact
The method is executable, not just described.
I extracted and sanitized the evaluation design into a public, provider-agnostic reference model. It has zero runtime dependencies, runs on Node 20+, and tests every invariant without a network or credentials. The production prompt, character definition, provider wiring, and private analytics remain excluded.
sources and runnable code