One Reply Isn't the Whole Conversation: Chaining Real API Calls to Test Multi-Turn AI Behavior

An earlier post covered the harness that grades one real reply against real reference facts using a second Claude as judge. That catches a lot — tone, honesty, whether the AI stuck to a rule — but every one of those scenarios is a single, isolated turn: one simulated message in, one real reply out, scored. Some of the chatbot’s actual behavior only exists across a sequence of turns. Does it correctly recognize that a live handoff with Joey genuinely ended, three turns after the fact? Does it stop answering résumé questions and pivot into the handoff intake flow the moment a visitor’s intent turns from curious to serious? A single-turn scenario can’t test that — there’s nothing earlier in the conversation for the AI to have gotten right or wrong.

The mechanism: chain real replies, not fixtures

The first idea was to record a real conversation once, freeze it as a fixture, and replay it forever. That turned out to cost the same as the alternative — either way it’s one real API call per turn — while adding a real downside: every time a behavior rule changes, the frozen fixture is stale and has to be re-recorded by hand. The design that shipped instead just chains real calls live, inside one test file: an early turn’s real reply becomes a later turn’s conversation history, threaded through a module-level variable. Vitest already runs tests inside one describe() block sequentially in definition order, so no special scheduling was needed — just a guard (requireDefined()) that throws a clear error if a later test somehow runs before the turn it depends on, instead of failing on a mysterious undefined.

Scenarios can branch, too — the same opening question, then two different real visitor replies exploring two different outcomes. Both branches read the same ancestor’s real output; nothing about reading conflicts. Writing turned out to be the part that needed care.

What it caught — including a bug in the test itself

Extending this tree to cover more scenarios surfaced a genuine bug, but not in the chatbot: two sibling branches — one where a visitor turns out to be a real lead, one where they’re just curious — shared the exact same fake in-memory store standing in for the chatbot’s visitor-info database. In reality those two conversations would never coexist; a visitor is either one or the other. But in the test, the curious-visitor branch ran second, its real submit_lead call wrote a stranger’s contact details into the shared fake store, and a later node on the other branch read that leftover data back out as if it were its own visitor’s info. The AI accurately reported what it had just been handed — and the judge flagged it as a fabrication, because the reference facts didn’t match. The bug was entirely in the test’s plumbing: sibling branches that never coexist in real life still need genuinely separate fake state, not just separate variables that happen to point at the same store.

The chained format also made an infrastructure issue visible that single-turn scenarios rarely triggered: after a tool call, the model occasionally comes back with an empty final reply instead of the text it’s supposed to send. A one-time resample mostly covered it in earlier testing, but longer chained conversations call tools more often, and a handful of real runs showed the resample itself coming back empty too. Rather than add a second layer of prompt wording hoping it wouldn’t happen, the retry got a second attempt — three real calls total before actually failing — which measurably closed the gap without pretending sampling noise doesn’t exist.

The score is a compliance rate, not a checkbox

One scenario tests whether the assistant, already knowing a returning visitor’s name and what they want, skips the redundant question and jumps straight to offering a live handoff — using known information to actually move faster, not just avoid repeating itself. The fix that made this work landed cleanly on paper. Run against the real API six times in a row, though, it passed four. Two runs still re-asked the very question the rule says is already answered. That’s not a failure to fix the bug — the specific broken pattern from before the fix (asking the question and inventing details) is gone, replaced by an occasional, milder lapse. It’s a smaller, less capable model complying with a fairly dense system prompt at something like a two-in-three rate rather than one hundred percent, and the honest thing to do with a number like that is write it down rather than keep tweaking prompt wording indefinitely chasing a clean sweep that this model may not reliably give.