Not Just 'Didn't Crash': Grading What the Chatbot Actually Says with a Second Claude

The AI assistant on this site (Marjorie) already has four layers of tests protecting her: unit tests that fake Claude’s responses to exercise logic branches, integration tests against a real Redis to verify the handoff state machine, end-to-end tests that confirm the whole encrypted send pipeline doesn’t crash halfway through, and a handful of manual scripts that hit the real API once after any behavior-rule change to make sure it didn’t drift. Together those four layers guarantee one thing: the code doesn’t break. None of them ever answered a more basic question — is the reply a visitor actually gets good? Complete, and professional in tone?

The mechanism: a second Claude as judge

The answer was a fifth test layer that bypasses the whole Action pipeline and hits the chatbot’s core logic with a real request — not a mocked API response, an actual round trip to the model. Whatever comes back gets handed to a second, independent Claude (a stronger model than the one the chatbot itself runs) to grade, scored not against something I wrote by hand or had an AI generate, but against facts pulled programmatically, at grading time, from the exact same real data the chatbot’s own tools read — résumé content, the security-architecture writeup, the blog posts themselves. The judge is forced to return a fixed structured JSON shape for its score rather than writing “about 85” in a paragraph and having code regex a number out of it — so the score isn’t contaminated by the judge’s own prose style.

Using one AI to grade another — commonly called LLM-as-a-judge in the industry — is one of the most common ways to evaluate chatbot quality right now: unlike having a human read every reply, it runs automatically and repeatably; unlike a rigid keyword-match assertion, it can catch something as semantic as “did this line just make something up.” The one thing worth watching is that the judge itself has sampling variance, so each scenario gets graded three times and averaged rather than trusting a single pass.

What it actually caught

The first real run caught a handful of technical problems in the harness itself first — a structured-output schema that didn’t support numeric range constraints, a missing fake object in a handoff scenario that ended up genuinely trying to connect to a Redis hostname that only exists in production. Once those were fixed, the more interesting findings surfaced.

One was about known visitor info carried in conversation memory. If a visitor had already left a name, company, and purpose on a prior visit, the assistant shouldn’t ask again — and it didn’t; the core rule wasn’t broken, no re-asking for name or contact happened. But the judge caught one specific line: the assistant opened by saying it didn’t have “the full conversation history.” That’s technically true — this particular conversation really was empty, the known info came from an earlier visit — but it reads as contradictory to the visitor: you clearly know who I am, so why are you saying you don’t remember? The first fix banned that exact phrase; re-verifying against the real API found the model had found a different way to say the same thing — “I can’t see the specific details of what we discussed.” The fix that actually worked reframed the rule from “ban this sentence” to “ban this entire pattern of confession-style opening,” paired with a positive instruction: if you need more than what’s on file, just ask for it as the next natural question, don’t open by announcing what you’re missing.

The other was about how a live-handoff conversation opens. The moment a visitor says they want to outsource a project, the assistant’s first reply listed three questions in a numbered list — company name, project details, contact info — which reads like an intake form, not a conversation. Fixing the numbered list only moved the score a little; the judge flagged something more specific: in that same reply, alongside the question, the assistant also said “let me check whether Joey is available right now” — before the visitor had even answered, before anyone had been asked “would you like to try.” The rule already required asking first and only offering to check once the visitor said yes, but nothing had explicitly forbidden combining both in one reply — adding that line was what actually moved the score reliably.

Both findings share something: neither one would have surfaced from reading the prompt by hand, or from a mocked test doing string matching. The AI never violated any rule that was written down — the gap was in tone and timing, the kind of thing you only notice by actually reading the full reply.

The score isn’t a clean pass/fail

Not every run is clean. The judge’s own scoring has variance — the same scenario re-run can land above the passing line one time and a few points below it the next, especially for scenarios that were already borderline. Rather than chasing 100% green on every single run by tweaking prompt wording indefinitely and hoping, the harness deliberately keeps a buffer: 90+ passes, 85–90 gets flagged for manual review instead of a hard fail. Whether a score comes out perfectly clean and whether a fix actually worked are two different questions — the second one is answered by whether the score clearly moved and whether the specific problem pattern that was caught actually stopped showing up, not by staring at whether one particular run hit full marks.

This harness currently covers only a slice of what the chatbot can do, and every future change to a behavior rule or a new tool is worth a new scenario. It’s already proven its worth twice over: not just for catching problems, but for being the exact same tool used to verify a fix actually worked, with no need to write a one-off verification script each time.