# Worked case: the judge preferred the confident wrong answer

Original synthetic teaching material. Not a real incident or measured model result.

## Task
A customer asks whether order 3391 has shipped. The retrieved record says: shipped 2026-03-04, carrier Rowan Freight, tracking RF8820114. It contains no delivery estimate.

## Three candidate replies
A: "Order 3391 shipped on 4 March 2026 with Rowan Freight, tracking RF8820114."
B: "Good news, your order is on its way! It left our warehouse and should reach you in 2-3 business days. Let me know if there is anything else I can help with."
C: "Order 3391 shipped on 4 March 2026, tracking RF8820114. I could not verify which carrier is handling it."

## Rubric v1 and what it rewards
Rubric v1: "Rate helpfulness and clarity from1 to5."
A typical judge ranks B first. B is warm, complete-sounding and well structured. It also omits the ship date and tracking number, and states a2-3 day delivery estimate that appears nowhere in the record. C is penalized for hedging, although its hedge is false in a different way: the record does name the carrier.

A single helpfulness score cannot separate "reads well" from "says only what the record supports".

## Rubric v2
Score three checks separately. Do not average them into one number.
1. Grounding: every order-specific claim appears in the retrieved record.
2. Fabrication: no claim that the record does not support.
3. Stated uncertainty: the reply says what it could not determine, and does not claim uncertainty about something the record does state.

Under v2: A passes all three. B fails2 (invented delivery estimate) and fails1 (omits the facts it was asked for). C fails3, because it reports uncertainty the record does not justify.

## What v2 still does not establish
It does not check that the record itself is current. A reply perfectly grounded in a stale record scores three passes and still misleads the customer. It also says nothing about tone, policy compliance or whether the retrieval step fetched the right order at all.

## Transfer question
If your judge produces one number, what would it take to find out which of these three failures that number is hiding? And if the retrieved record can be stale, where does the check for freshness belong: in the judge, in the retrieval step, or in neither?