LLM-as-Judge: Build Evaluators You Can Actually Trust

LLM-as-judge uses a language model to score another model's output against a rubric. It is the practical bridge between exact-match metrics, which are cheap but narrow, and human review, which is rich but expensive.

The method is useful for criteria such as faithfulness, relevance, completeness, tone, and instruction following. It is also easy to misuse. A judge is not ground truth: it has biases, stochastic behavior, blind spots, and failure modes that can correlate with the system it evaluates.

The question is not “Can a model produce a score?” It plainly can. The question is whether that score agrees with qualified humans often enough, remains stable across irrelevant changes, and detects the regressions your product actually cares about.

Start with the metric hierarchy

Use the cheapest reliable evaluator for each criterion:

1. Deterministic assertion — schema validity, exact fields, numeric bounds, forbidden strings, citation presence, latency, or cost.
2. Programmatic semantic metric — retrieval recall, string similarity, classifier output, or another task-specific check.
3. LLM judge — meaning-dependent criteria that resist code.
4. Human review — novel, high-stakes, disputed, or calibration cases.

Do not ask a judge whether JSON parses, whether a field is present, or whether a number is inside a range. Code answers those questions more cheaply and consistently. Save model judgment for the dimensions that require language understanding.

Score one thing at a time

“Rate this answer's overall quality from 1–10” produces a number that is difficult to interpret or repair. Break quality into orthogonal criteria:

• Faithfulness: Are claims supported by the supplied context?
• Relevance: Does the answer address the user's request directly?
• Completeness: Does it cover every required element?
• Tone: Does it match the defined communication standard?
• Actionability: Can the user take the next step from the answer?

Each criterion needs its own rubric and output field. When a release score drops, you should know which property regressed.

Write anchored rubrics

A rubric should describe observable evidence, not adjectives:

Evaluate FAITHFULNESS only.
 
2 = Every material claim is supported by the supplied context.
1 = The central answer is supported, but one secondary claim is unsupported
    or overstates the context.
0 = A central claim contradicts or cannot be derived from the context.
 
Do not reward writing style, length, or helpful details beyond the context.
Return JSON: {"score": 0|1|2, "unsupported_claims": [string], "reason": string}

The middle anchor matters. Without it, the judge invents its own boundary between pass and fail. Include examples near that boundary and cases that should be rejected despite fluent wording.

Pointwise versus pairwise

Pointwise judging scores one output against a fixed rubric. It is useful for absolute thresholds, dashboards, and release gates because every run produces a comparable value.

Pairwise judging compares output A with output B. It is useful when choosing between a prompt, model, or retrieval variant because “which better satisfies this criterion?” is often easier than assigning an absolute score.

Pairwise results require order controls. Run both A/B and B/A, hide system identity, and count an order-sensitive disagreement as uncertainty rather than forcing a winner. For several variants, randomize order and use a tournament or balanced comparison design rather than always presenting the incumbent first.

Build the calibration set

The judge needs a test suite of its own. Create a small, carefully labeled set containing:

• ordinary production-shaped examples
• known good and known bad outputs
• borderline cases where reasonable people may disagree
• long and short answers with equivalent quality
• polished nonsense and awkward but correct answers
• failures your application has already shipped
• examples from every important language, route, or user segment

Have at least two qualified humans label the cases independently before resolving disagreements. Their disagreement is information: if people cannot apply the rubric consistently, the judge will not rescue it. Rewrite the criterion before tuning the model prompt.

Measure exact agreement for categorical outcomes, correlation for ordered scores, and an agreement statistic such as Cohen's kappa when chance agreement matters. Also inspect the confusion matrix. A judge that agrees overall but routinely passes the most dangerous failure class is not acceptable.

Test known judge biases

Run targeted probes before trusting the metric:

• Position bias: Swap A and B in pairwise comparisons.
• Verbosity bias: Compare equivalent answers with different lengths.
• Style bias: Vary polish, headings, and confidence without changing correctness.
• Self-preference: Blind which model or provider produced the output.
• Reference leakage: Ensure hidden labels or expected answers are not accidentally visible.
• Instability: Repeat a representative sample and measure score variance.

Use a deterministic sampling configuration when supported, but do not mistake a low temperature for guaranteed repeatability. Cache evaluator results for identical versioned inputs and rerun enough cases after judge-model changes to establish a new baseline.

Separate the evidence from the verdict

Require structured output containing both the score and the specific evidence behind it. Store the raw evaluated input, candidate output, rubric version, judge prompt, judge model, result, and timestamp.

Reasons are not proof that the verdict is correct, but they make debugging possible. They reveal whether the judge applied the wrong criterion, rewarded irrelevant style, missed a citation, or encountered an ambiguous rubric.

For faithfulness, ask for unsupported claims rather than a general explanation. For completeness, ask for missing requirements. Make the evidence schema reflect the criterion.

Use the judge in CI carefully

A calibrated judge can gate releases, but the gate should acknowledge uncertainty:

• combine judge scores with deterministic invariants
• require a minimum sample size
• compare against a frozen baseline on the same dataset
• define an allowable regression band before seeing results
• fail hard on critical safety cases even when the aggregate score is healthy
• send borderline or contradictory results to human review

Track scores by slice, not only as one average. A gain on simple requests can hide a collapse on long-context, multilingual, or high-stakes traffic.

Recalibrate when the system moves

Treat the evaluator as versioned production code. A new judge model, rewritten rubric, changed prompt, or shifted traffic distribution can invalidate the historical scale. Pin versions, preserve old results, and rerun the calibration set before comparing across the change.

Sample production outputs for ongoing human review. Promote newly discovered failures into both the product eval set and the judge's calibration set. This creates a closed loop: real errors improve the system and the measurement system used to judge it.

The companion LLM-as-Judge Scorer skill turns this method into a reusable workflow, while Writing LLM Evals shows how the judge fits inside a complete evaluation suite.