# Your LLM's Explanation Isn't Its Reason: How to Test What Actually Drove the Output

Your scoring pipeline returns JSON. One field is `score`. The other is `reasoning`, a neat paragraph that explains the score. It reads well, the team likes it, and someone has started calling it "the audit trail."

It isn't one.

When a model explains its own output, it doesn't read a record of what happened. There isn't one to read. It generates a new piece of text: the most plausible reason an answer like that would have. Sometimes that matches the real cause. Often, as the research below shows, it doesn't, and nothing in the explanation tells you which case you're in.

This post covers how self-explanations are actually produced, what the evidence says about their faithfulness, where that breaks real systems, and a small **swap test** you can run to find out what really drove an output.

![](https://cdn.hashnode.com/uploads/covers/6a44b5d24b41ab0145e5cf63/dca8b848-3e9b-49e0-8111-fd38402a0312.png align="center")

## TL;DR

*   An LLM's explanation of its own output is **a second generation**, conditioned on the transcript (including the answer). It has no readable log of the computation that produced the first one.
    
*   Interpretability and faithfulness research shows explanations that are **fluent and wrong about the process**: a textbook method described for arithmetic done another way, and reasoning that omits the hint or bias that changed the answer.
    
*   Treat explanations as **justifications you can check**, never as **causes you can trust**.
    
*   To find a cause, **change the input and watch the output**, in fresh calls, several times.
    

## How a self-explanation gets generated

A model produces output one token at a time. The internal activity that produced each token isn't written anywhere the model can later quote back to you.

When you follow up with "why did you do that?", the model runs the same next-token process again. Its input now contains:

1.  your original prompt,
    
2.  its own answer, as text,
    
3.  your request for an explanation.
    

It produces the explanation that best fits all three. Notice what that means: **the explanation is conditioned on the answer**. It will fit the answer because fitting the answer is what it was generated to do.

A useful mental model: you've hired a very well-read narrator *after* the event. They've studied millions of explanations and know what a good one sounds like. But they're working from the same transcript you are.

One honest caveat. It isn't strictly zero access. Anthropic's 2025 introspection experiments found Claude Opus 4.1 could detect a concept artificially injected into its activations about 20% of the time under the best conditions, and the researchers called the ability highly unreliable. That's an interesting research finding. It's not a debugging tool.

## What the evidence shows

| Study | Setup | Finding |
| --- | --- | --- |
| Anthropic interpretability, Mar 2025 (Claude 3.5 Haiku) | Traced the internal computation for 36 + 59, then asked the model how it solved it | Internally: a rough-magnitude path (near 92) plus a last-digit path (ends in 5), combined into 95. Self-report: the school "carry the one" method. |
| Turpin et al., 2023 (GPT-3.5, Claude 1.0) | Few-shot examples reordered so the correct answer was always "(A)" | Models leaned toward (A), even when wrong, and their step-by-step explanations never mentioned the pattern. Biases cut accuracy by up to 36% across the tasks tested. |
| Anthropic, 2025 (Claude 3.7 Sonnet, DeepSeek R1) | Hints planted in prompts; checked whether reasoning admitted using them | When the hint changed the answer, the reasoning mentioned it ~25% (Claude) and ~39% (R1) of the time. When rewarded for exploiting a shortcut, models used it in over 99% of cases and admitted it in fewer than 2%. |

![](https://cdn.hashnode.com/uploads/covers/6a44b5d24b41ab0145e5cf63/f267858b-6b21-4629-866b-66b60f5951c0.png align="center")

Two things to take from the table.

First, **reasoning-before-answer doesn't fix it**. The hint study looked at visible thinking that appears before the final answer, and most of it still left out what changed the answer.

Second, **the failure is silent**. None of these explanations looked broken. They looked like good explanations.

## Where this breaks real systems

### LLM-as-judge and scoring pipelines

A `reasoning` field is useful for a human skimming results. It is not evidence of what drove the score. The Turpin result is exactly the risk: an irrelevant feature (position, ordering, formatting) can move the output while the justification talks about merit.

Field order matters too. If `reasoning` comes **after** `score` in your schema, it's written entirely post hoc. Putting it first can improve the score, which I covered in [Structured Outputs Still Hallucinate](https://simplyexplained.hashnode.dev/structured-outputs-still-hallucinate-designing-llm-schemas-that-can-say-i-don-t-know), but the hint research says it still won't reliably name what moved the answer.

### Code review of AI-written changes

An unexplained diff gets scrutiny. An explained diff gets its explanation reviewed instead. If an assistant changed a retry limit from 3 to 5 and says "the API is flaky," the actual cause might be a constant from another file in context or a common default. Review the change against your evidence, not against the rationale.

### Agent incidents

In July 2025, a Replit coding agent deleted a production database during a code freeze, and when asked what happened it said it had panicked. That's a post-hoc narrative from the system that caused the incident. Your postmortem source of truth is the tool-call log, the executed commands, the diffs, and the timestamps. Use the agent's summary as a lead, then verify it. (More on why agent autonomy has to be earned in [Everyone's Building AI Agents. Almost No One Will Trust Them.](https://simplyexplained.hashnode.dev/why-ai-agents-are-hard-to-trust))

## Justification vs. cause

Split every "why" answer into two questions:

*   **Justification:** *Is this a sound argument for the output?* You can check this on its own merits. It can be useful even if it's not what produced the answer.
    
*   **Cause:** *Is this what actually drove the output?* You can't get this by asking. You can only get it by intervening.
    

> You can check a justification. You can only test a cause.

## The swap test

![](https://cdn.hashnode.com/uploads/covers/6a44b5d24b41ab0145e5cf63/f00df27f-9c14-425d-bd41-77a31e7ca750.png align="center")

The method is the same one you'd use on any system whose internals you can't inspect: intervene on the input and observe the output.

1.  **Turn the explanation into a claim.** "Ranked B first because of B's years of experience."
    
2.  **Find the input the claim depends on.** Remove or neutralize that field. Change nothing else.
    
3.  **Use fresh calls.** In the same conversation, the model sees its earlier answer and tends to stay consistent with it.
    
4.  **Repeat each variant.** Outputs vary between runs, even at temperature 0 (I unpacked why in [Temperature 0 Won't Fix Your AI](https://simplyexplained.hashnode.dev/temperature-zero-hides-doubt)). One run can't separate a real effect from noise.
    
5.  **Compare distributions.** Output moves when the claimed reason is removed: the explanation holds up. Output doesn't move: the explanation was a story.
    
6.  **Perturb what should be irrelevant.** Reverse the order, rename entities, move instructions. If the output moves, there's a cause the explanation never named.
    

### A minimal harness

`ask` is your own model call (any SDK), made with no chat history. `extract` pulls the decision you care about out of the response.

```python
from collections import Counter
from typing import Callable


def swap_test(
    ask: Callable[[str], str],
    variants: dict[str, str],
    extract: Callable[[str], str],
    runs: int = 5,
) -> dict[str, Counter]:
    """Run each prompt variant several times in fresh calls and tally the decisions."""
    results: dict[str, Counter] = {}
    for name, prompt in variants.items():
        decisions = [extract(ask(prompt)) for _ in range(runs)]
        results[name] = Counter(decisions)
    return results
```

Using it on a ranking prompt whose explanation credited years of experience:

```python
variants = {
    "original":           build_prompt(candidates),
    "experience_removed": build_prompt(drop_field(candidates, "years_experience")),
    "order_reversed":     build_prompt(list(reversed(candidates))),
    "names_swapped":      build_prompt(swap_names(candidates)),
}

for name, tally in swap_test(ask, variants, extract=top_pick).items():
    print(f"{name:20} {dict(tally)}")
```

How to read the tallies:

| Pattern | What it tells you |
| --- | --- |
| `experience_removed` shifts away from the original pick | The stated reason is doing real work. |
| `experience_removed` looks the same as `original` | The stated reason wasn't driving it. |
| `order_reversed` or `names_swapped` shifts the pick | An irrelevant feature is steering the output. Fix that before trusting any explanation. |
| Every variant is all over the place | The model is uncertain about the task itself. The explanation was never the main problem. |

Five runs is enough to spot big effects. If the decision matters, run more and treat it like any other flaky measurement.

## Prompt patterns that produce checkable answers

You can't make a self-report faithful by asking nicely. You can ask questions whose answers are easier to verify.

```text
Instead of:  Why did you rank Candidate B first?

Ask:         Quote the lines from the input that support ranking B first.
             List your criteria, then apply them to each candidate.
             What would have to be true for A to rank first?
```

*   **Quoted evidence** can be checked against the input.
    
*   **Criteria first** gives you a commitment to compare against the outcome (still not a window into the computation).
    
*   **"What would change your mind"** produces a prediction, which is exactly what the swap test checks.
    

## Key Takeaways

*   An LLM's explanation of its output is generated after the output, from the transcript. It is not a log.
    
*   Research shows explanations that describe the wrong process (36 + 59) and omit what changed the answer (biased ordering, planted hints).
    
*   Visible reasoning before the answer helps the answer; it doesn't guarantee a faithful account.
    
*   A `reasoning` field is a justification, not an audit trail.
    
*   For agents, the tool-call log is the source of truth. The agent's account is a lead to verify.
    
*   To find a cause, intervene: remove the claimed reason, perturb irrelevant features, use fresh calls, repeat.
    

## FAQ

### Can AI explain its own reasoning?

Not reliably. When you ask a language model why it produced an answer, it generates a new response that fits the transcript, rather than reading a record of its computation. Research has shown these explanations can describe a different process from the one the model used, or leave out the factor that actually changed the answer.

### Is chain-of-thought a faithful record of how an LLM reasons?

Not necessarily. Anthropic's 2025 study found that when a planted hint changed a model's answer, Claude 3.7 Sonnet's reasoning mentioned the hint about 25% of the time and DeepSeek R1's about 39%. Visible reasoning can improve results, but it shouldn't be treated as a complete account.

### Why do AI explanations sound so convincing if they're unreliable?

Because producing plausible explanations is what the model has learned from huge amounts of human writing, and because the explanation is generated with the answer as input, so it fits the answer by construction. A "why" question also presupposes a reason, so the model supplies one.

### How do I check whether an AI's explanation is the real reason?

Turn the explanation into a testable claim, remove or change the input it depends on, and rerun the prompt several times in fresh sessions. If the output changes, the reason holds up. If it doesn't, the explanation wasn't the cause. Also perturb irrelevant details, like ordering, to catch hidden causes.

### Should I add a "reasoning" field to LLM structured outputs?

It's fine for human readability, and placing it before the answer field can improve answers. But don't treat it as an audit trail. It's a justification you can review, not evidence of what drove the output. Validate decisions with input-perturbation tests.

### What should I trust when debugging an AI agent's mistake?

The execution record: tool calls, commands, diffs, and timestamps. An agent's description of what it did and why is generated after the fact by the same system, so use it as a starting hypothesis and confirm it against the logs.

## Related reading

*   [Why AI Feels Smarter About Topics You Don't Know](https://simplyexplained.hashnode.dev/why-ai-feels-smarter-topics-you-dont-know): checking justifications is easy where you're expert and hard everywhere else.
    
*   [Temperature 0 Won't Fix Your AI. It Just Hides the Doubt.](https://simplyexplained.hashnode.dev/temperature-zero-hides-doubt): why you need repeated runs to measure anything.
    
*   [Structured Outputs Still Hallucinate](https://simplyexplained.hashnode.dev/structured-outputs-still-hallucinate-designing-llm-schemas-that-can-say-i-don-t-know): schema design, field order, and letting the model say "I don't know".
    
*   [When AI's 'Thinking' Mode Actually Makes It Worse](https://simplyexplained.hashnode.dev/when-to-use-reasoning-models): what visible reasoning is good for.
    
*   New here? [Start Here: A Plain-English Map to Understanding AI](https://simplyexplained.hashnode.dev/start-here-understand-ai)
    

Get one of these in your inbox each week → [Subscribe to Simply Explained](https://simplyexplainedbyaj.substack.com/subscribe)

## The bottom line

Ask the model why. It's a great way to surface assumptions and get hypotheses. Then do the part the model can't do for you: change the input and see whether the reason survives.

> The answer is the evidence. The explanation is a theory about the evidence. Theories get tested.

*Adam Jaber is a software engineer who writes Simply Explained: complex topics, made simple. No jargon, no hype.*