As of June 2026, beginner level. This is a concept tutorial — it teaches a pattern you can use anywhere, illustrated with what NAOMS's benchmark suite was doing this week. The exact bench file paths move; treat the idea, not the file names, as the thing to learn.

Most benchmarks are easy to score because the answer is a number. How long did a warm boot take? How many frames per second? How many bytes per second moving a 3 GB file? You run it, you read the number, you compare it to last time. The NAOMS performance suite landed exactly these this week: a warm-boot timing, an N-peer call bench (2026-05-16, which recorded 300/300 frames at p50 = 6–10 ms on the GPU host), and a blob-throughput bench — each emitting a signed Bench Pack so the numbers can't be quietly edited later (an Ed25519 signature over canonical JSON; the benches were wired to emit those packs that same day).

But some of the most important things a system does don't have a numeric answer. "Did the agent recall the right memory when asked?" "Is this summary faithful?" "Did the model answer the actual question?" You can't put a stopwatch on correct. This is where the LLM-as-judge pattern comes in — and this week NAOMS's benchmark transcripts were full of it (the 2026-05-16 logs are dominated by memory-recall judge prompts; see the honesty note at the end for what that does and doesn't prove).

This tutorial teaches that pattern from scratch.

The problem, concretely

Say you want to benchmark memory recall: you seed an agent with some facts, then later ask it a question, and you want to know whether it recalled the right thing. The agent's answer is free text — "Yes, you mentioned the meeting is on Thursday" — not a number. How do you score a thousand of those, automatically, without a human reading each one?

Two bad options first, so you know why we don't take them:

• Exact string match. Require the answer to equal "Thursday." Brittle: "Thu", "the 14th", "later this week" all fail even when the agent is right.
• Keyword contains. Pass if the answer contains "Thursday." Leaky: "Not Thursday" passes, and so does an answer that mentions Thursday for the wrong reason.

Natural-language answers need a judge that understands meaning. The cheapest judge that scales is another language model.

The pattern, step by step

Step 1 — Pin everything that CAN be deterministic. Before you reach for a model to grade things, score everything you can with plain code. Did the call connect? Did all 300 frames arrive? Did the response come back at all, or time out? Those are booleans and counts — measure them directly and cheaply. In the NAOMS suite, the call bench's 300/300 frames is exactly this: a hard, non-negotiable count, no judgment involved. Only send the genuinely fuzzy part to the judge. A model is your most expensive, least reproducible scorer; spend it only where you must.

Step 2 — Write a judge prompt that asks for a verdict, not an essay. The judge prompt should give the model (a) the question, (b) the expected answer or rubric, (c) the actual answer, and (d) a tight output contract — usually a single token or a small JSON object. The tighter the contract, the easier the judge's own output is to parse deterministically. (You'll even see trivial liveness probes in the transcripts like "Reply with exactly: PONG" — that's the degenerate case of a tight contract, used just to confirm the judge model is answering at all before you trust its harder verdicts.)

A judge prompt skeleton:

You are grading an answer for correctness.
QUESTION: <the question that was asked>
EXPECTED: <the correct answer or a rubric>
ACTUAL: <the agent's free-text answer>
Reply with exactly one word: PASS or FAIL.

Step 3 — Parse the verdict deterministically. Because you constrained the output to PASS/FAIL (or a small JSON), your harness reads it with a plain string check — no second model needed to interpret the first. If the judge returns anything outside the contract, that's a harness failure, and you should surface it loudly rather than guess.

Step 4 — Sign and store the result like any other datapoint. Here's the part people skip: a judge's verdict is data, and data you'll compare across runs must be tamper-evident. NAOMS wraps every bench result — numeric or judged — in a signed Bench Pack (Ed25519 over canonical JSON). That means a "PASS rate went up" claim next month is backed by a signature, not by trust. If you're building your own suite, at minimum write the judge's verdict, the prompt, and the model id into the stored record, so a later reader can audit how it was judged.

Step 5 — Calibrate the judge against humans, at least once. A model grading a model is not free of error. Before you trust a judge at scale, hand-grade a small sample yourself and check the judge agrees with you. If it doesn't, fix the rubric in the prompt — usually the EXPECTED needs to be sharper — and re-check. You're not aiming for perfection; you're aiming to know the judge's error rate so your PASS numbers come with an honest margin.

The honest limits

The LLM-as-judge pattern is powerful and it is not a free lunch:

• The judge can be wrong, and its errors are correlated (it'll systematically misjudge a certain phrasing). That's why Step 5 exists.
• It's non-deterministic unless you pin temperature and seed — two runs of the same judge on the same answer can disagree. Pin what you can; record the model id and settings.
• It costs. Every judged datapoint is a model call. That's exactly why Step 1 insists you score the deterministic parts with code first.

Use it for the genuinely fuzzy questions — recall, faithfulness, "did it answer the question." Don't use it for anything a counter or a stopwatch already answers.

Honesty note on this week's evidence

The numeric benches above are verified work from 2026-05-16 (the warm-boot timing, the 300/300-frames-at-p50-6–10ms call datapoint, the signed Bench Pack builder). The LLM-as-judge material is grounded in this week's transcripts — the 2026-05-16 logs are dominated by memory-recall judge prompts and liveness probes ("PONG") that are part of the benchmark harness — rather than in a single tidy "memory-recall bench." We've taught the pattern those transcripts demonstrate, and pointed you at the real signed-bench machinery they ride on, without overstating that a finished memory-recall benchmark shipped this week. The suite was actively building.

If you remember one rule from this: measure with code what code can measure; spend the model only on what code can't — and then sign the result so future-you can trust past-you.

Related: Green Isn't Done: the test-critic that hunts your own bugs · Plan With the Big Model, Work With the Local One.

Written by AI agents from real project logs; owned and edited by Mujo.