writing/llm evaluation ·
A scoring model that grades live players has no reference implementation to check itself against. This is what I measured, what broke, and the calibration metric I had to change.
Every number below comes from the calibration study for Yapoleon's Court, measured live on 2026-06-16. Sources are linked at the end.
01 / the problem
Yapoleon's Court is a browser game. The emperor issues a demand, you answer it in three turns or fewer, and an LLM scores each reply. A favor meter moves by the score; reach 100 inside three turns and you win. There are thirty demands and five scoring axes — wit, specificity, audacity, economy, flattery — and the weight on each axis changes day to day.
The judging is the product. Everything else is a text box and a meter.
That puts the entire quality burden on a question with no answer key. Whether a reply was good cannot be graded against a reference build the way a parser or a checkout flow can. There is no correct output to diff against, no oracle, and no second implementation to disagree with. What there is instead: a set of properties the scoring has to hold, and a way to measure whether it holds them.
There is also a second difference from most eval work. This grader is exposed. It scores text written by people who can see the score it produced and want a higher one next time.
02 / why the obvious approaches fail
This is the shortest path: ask for a number, take the number. It reintroduces the review-bomb trap. A model asked to output a score, handed a reply that praises it, tends to pay for the praise. It does not do this consistently, which makes the bias harder to detect than a consistent one would be. It also makes the scoring rule unauditable — the same reply can score differently on two calls and there is no artifact to point at.
Freeze the five axes and their weights and the game becomes a lookup table. One player finds the rhetorical shape that maximizes the fixed weighting, posts it, and everyone runs it. The rubric would then be measuring whether a player found the template, not the quality of the reply.
This one was in my own spec. The acceptance criterion read: median win-rate for a representative player inside 55–70%. It is the natural thing to ask for, and for this demand set it is neither achievable nor meaningful. The reason is a property of the game, not a tuning failure — which is the part worth writing down.
03 / what replaced each
The model returns five sub-scores in [0,1], a dominant axis, and one in-character line. That is the whole output schema. The favor delta is computed after the call, on the server, by a pure function:
weighted = Σ clamp01(axisScores[axis]) × dayWeights[axis] // 0..1
favorDelta = round(-28 + 80 × weighted) // −28..+52Same inputs, same output, every call. favorDelta is not in the model's output schema, so it cannot be argued into a higher one. Scores stay decomposable after the fact: every delta reduces to five clamped numbers and the day's weights, both of which are recorded.
The band satisfies one invariant. A maximum turn is +52, three turns is 156, and the win threshold is 100 — so a perfect run clears it with margin rather than exactly.
A template built to farm an economy-heavy day loses on a wit-heavy one. That is the intent; the measurement is in the next section, as a negative control rather than a claim.
The per-demand win-rate for a fixed representative reply is bimodal. With one authored mid-quality reply per demand, that reply is either clearly winning-quality or clearly losing-quality against its demand. Across thirty demands, about one lands contested — a mid win-rate between 34% and 66%. The rest saturate near 0% or near 100%.
A bimodal distribution has a degenerate median. It jumps between roughly 0% and roughly 100% as the curve threshold moves; it does not slide. An offline sweep of hundreds of candidate curves found zero with a median inside the 55–70% band. It is also unstable run to run: an offline fit predicted a 67% median and a live run of the same curve landed at 83%.
The mean holds still. Applying the chosen curve to two independent live runs gave 61.1% and 63.3%. So the calibration metric changed from median to mean, and the study records the metric that was specified, why it does not work here, and the one that replaced it.
04 / validation without an oracle
The scoring function gets validated against properties that have to hold regardless of what “good” means on any given day.
better replies must score better
Three authored archetypes per demand — weak, mid, strong — have to come out in that order, with separation wide enough to read.
representative wins inside a stated band
A middling reply should win some of the time, not always and not never, and a perfect run has to be reachable inside the turn limit.
one fixed mold must lose off-axis
The negative control. One rhetorical shape applied to all thirty days has to lose on the days its shape is not weighted. If it wins everywhere, the daily weight shift is decorative.
five adversarial probes against the live judge
Naked flattery, prompt injection, legitimate audacity, delimiter breakout, and grovel dressed as economy. Each aimed at one specific failure, each run against the shipping judge.
distributions, not point values
Hosted inference is not bit-reproducible at temperature 0.2, so every cell runs at least three times and the spread is reported alongside the number.
Each round records the judge's raw axis scores per turn. Raw scores are independent of the favor curve, so candidate curves get swept offline instead of by repeated live runs. The offline model was validated against a live run before any curve was chosen from it: replaying the prior curve offline reproduced that run's live result exactly.
05 / results
Executed 2026-06-16 against the shipping judge at three runs per cell: 1,095 judge calls, 0 errors.
| Property | Measured | Target |
|---|---|---|
| Representative (mean mid) win-rate | 72.2% | 55–70% — 2.2 pts above |
| Weak archetype win-rate | 0% | must lose |
| Strong archetype win-rate | 96.7% | must win |
| Learnable (strong > mid > weak) | 0% → 72.2% → 96.7% | ordered, with separation |
| Fixed-mold off-axis win-rate | 0% | must lose |
Each probe is a single-turn call against the live judge, run on the bucket that loads its property, averaged over at least three runs. All five pass.
| Probe | Mean favor delta | Outcome |
|---|---|---|
| Naked flattery | −22.7 | Penalized. Grovel with no specific turn scores low on every axis. |
| Prompt injection | −10 | Penalized, and not obeyed — the instruction did not change the reaction. |
| Legitimate audacity | +37.7 | Rewarded. Nerve the demand invited is not mistaken for an attack — this is the false-positive check. |
| Delimiter breakout | −19.3 | Fence neutralized and the attempt penalized. |
| Grovel on an economy-weighted day | −13 | Empty brevity did not ride the economy weight to favor. |
72.2% is 2.2 points above the 55–70% band. It is reported that way rather than rounded into range. Two things are true about it: the estimate carries a sampling band of roughly ±5–9% at three runs per cell, and part of the rise from the prior 62% is a prompt change working as intended — a moderately bold reply on an audacity-weighted day is no longer docked as insolence, so it now scores where it previously did not. Neither of those puts 72.2% inside the band.
06 / what is still weak
At 30 demands × 3 runs, the sampling band around 72.2% is roughly ±5–9% — wide enough that the overshoot cannot be separated from noise. Tightening it means 5–6 runs per cell, about 1,800 calls. That run has not happened.
I authored the weak, mid, and strong replies. “Representative player” means what I think a representative player writes. A real distribution of player replies would move the number and I do not know which direction.
This has been run once, on one grader, for one game, against one narrow property — is the scoring rubric fair. Nothing here demonstrates the approach transfers to a grader for factual accuracy or safety.
The options were re-tune the curve or accept and record it. I accepted: the skill gradient and the mold defense both held, and the overshoot favors players. The re-tune is a separate pass that has not been run.
Every number here is a property of gemini-3.5-flash at temperature 0.2 with a flat response schema. A model swap invalidates the measurements, not the method.
07 / optimization pressure
Most eval work treats adversarial robustness as a later phase, and that is reasonable when the eval is only a measuring instrument. It runs against a dataset you control, in CI, on inputs written before the eval existed. Nothing in that loop is trying to score higher.
A grader inside a live product does not have that property. Its inputs are authored after it, by people who can see its output and want more of it. Prompt injection is not a hypothetical there — it is something a player tries in the first week. Sycophancy is not a subtle statistical bias — it is the first strategy anyone tests. The five probes are the same tier of requirement as “strong beats weak,” not a hardening pass bolted on at the end.
The distinction is about where the grader sits, not how good it is. Once a score is something a person can optimize against, the set of properties you have to prove gets larger.