The Exception Pass: Catching What AI Summaries Delete
AI summaries keep what's typical and cut what's surprising. Here's the mechanism, the research, a prompt, and a test you can drop into your pipeline.
TL;DR
An AI summary isn't a smaller version of your document. It's a more typical one.
The sentences most likely to vanish are the ones that limit, reverse, or doubt something else: conditions, negations, hedges, dissent, temporary states.
Fix it with the Exception Pass: use the summary for shape, then extract the exceptions verbatim as a separate step. If you ship summaries, give exceptions their own field and measure exception recall.
The bug wasn't in the diff. It was in the recap.
Here's a scenario worth sitting with. A pull request is titled "Refactor validation layer, no behavior change." The description says the same thing, plus one line near the bottom: empty strings now return 400 instead of 200.
The AI-generated PR summary reads: Refactors the validation layer for readability. No behavior change.
Every word of that is defensible. The PR is a refactor. The author did say no behavior change. The one line that contradicts the framing is the line that didn't make it.
That's not a one-off. It's the most predictable failure of AI summarization, and once you see why it happens, you can guess what's missing from any summary before you read it.
Summaries regress to the template
When a model summarizes, it isn't ranking your sentences by how much they matter to you. It's generating the kind of text that a summary of this kind of document usually contains. PR summaries talk about what was refactored. Incident summaries name a root cause and a resolution. Release-note summaries mention performance, features, fixes, and compatibility.
Whatever in your document fits that template survives easily. Whatever doesn't has to fight.
Information theory explains why that's a problem. Going back to Claude Shannon, the core idea is that information is surprise. "Improves performance" in release notes is nearly free to predict, so it tells you almost nothing. "The default for retries changes from 3 to 0" is unpredictable, so it's the most informative line on the page.
And a language model is a machine for producing likely text. Likely sentences pull hard toward the output. The surprising sentence pulls weakest of all.
A conceptual sketch, not measured data.
Think JPEG. Compression keeps smooth gradients and discards sharp, high-frequency detail your eye won't miss. Summaries keep the smooth storyline and discard the sharp turns: unless, except, not yet, only in staging. Your eye doesn't miss the pixels. Production misses the sentence.
Training likely adds to this. Models tuned to be helpful may learn that confident, broadly applicable answers get rewarded over hedged, precise ones. Same direction, extra push.
What a 4,900-summary study found
Uwe Peters and Benjamin Chin-Yee measured this on scientific writing (Royal Society Open Science, April 2025). Ten models, 4,900 summaries of abstracts and full papers. The results that matter to developers:
Model summaries were about 4.85× as likely to contain broad generalizations as expert-written summaries of the same papers (NEJM Journal Watch).
The generalizations took three forms: dropping the scope (a result about one group became a claim about everyone), changing the tense (was observed became is true), and adding advice the authors never gave.
Prompting the model to avoid inaccuracies roughly doubled the odds of overgeneralizing.
Newer models were generally not safer than older ones in their tests.
Temperature 0 summaries were about 76% less likely to overgeneralize than temperature 0.7.
Caveat: these were science papers, not PRs. But the three shifts translate directly. Dropped scope is "compatible, except when pooling is on" becoming "compatible." Changed tense is "disabled for now" becoming "disabled." Added advice is "upgrade with a normal rolling deploy" becoming "safe to roll out normally."
Five sentence types to watch
To predict what a summary dropped, don't ask "what was important?" Ask "what limited, reversed, or doubted something else?"
| Type | In the source | In the summary |
|---|---|---|
| Condition | "compatible, except when pooling is on" | "compatible" |
| Negation / reversal | "we decided not to migrate this quarter" | "discussed the migration" |
| Hedge | "probably a connection leak; not confirmed in prod" | "root cause: connection leak" |
| Dissent | "one reviewer flagged a data-residency issue" | "the team approved the design" |
| Temporary state | "cron job disabled until the backfill finishes" | "cron job disabled" |
None of these is a big sentence. Each is a small word attached to a big claim, which is exactly why it loses.
Routine documents are the riskiest ones
This is the part that feels backwards. The more routine a document claims to be, the less you should trust its summary. "Small refactor." "Resolved." "No action needed." "Minor release." The framing matches the template, so the summary agrees with the framing and drops the one line that contradicts it.
Release notes show the same shape as the PR above:
Two habits compound it:
Summaries become the record. They get pasted into tickets and planning docs. Nobody reopens the 60-message thread.
Summaries of summaries. Daily recaps feed weekly digests feed monthly updates. Every pass can cut; no pass adds an exception back.
And you can't skim your way out. Even in your own codebase, you can't notice an absence. A deleted sentence looks exactly like one that was never written.
The Exception Pass
Treat summarization as two jobs.
1. Summary for shape. What is this, and do I need to act? Summaries are good at this because shape is the typical part.
2. Exceptions, verbatim. A separate request, not a "better summary":
List every sentence in this document that limits, reverses,
or doubts something else. Look specifically for:
- conditions (only, unless, except, if, when)
- negations and reversals (not, no longer, decided against, rolled back)
- hedges (probably, may, not confirmed, only tested in staging)
- disagreement (anyone who objected or raised a concern)
- temporary states (for now, until, as a workaround)
Quote each one exactly and say where it appears.
Do not paraphrase. If you find none, say "none found".
Why this shape works: quotes point at real text and take two seconds to confirm with a find; naming the five families tells the model what to protect (which "be accurate" never does); and "none found" removes the pressure to invent filler.
3. The surprise question. Aim straight at the mechanism:
What in this document would surprise someone who has
read a hundred documents like it?
Then decide on the quotes, not the summary. Run both at low temperature if you control it. And remember that "none found" means none were found, not that none exist.
Shipping summaries? Two guardrails
If you build PR bots, incident digests, or meeting-note features, this is your users' problem now.
Give exceptions a required field. A schema is a questionnaire; the model can only answer what you ask. No field means nowhere for the exception to go. Allow an empty list so the model isn't forced to invent one:
{
"overview": "one or two sentences",
"key_changes": ["..."],
"exceptions": [
{
"quote": "exact text from the source",
"kind": "condition | negation | hedge | dissent | temporary"
}
]
}
Measure exception recall. Not "is this summary good?" but "of the exceptions in the source, how many survived?" Mark the exceptions in twenty real documents from your own systems, then:
def exception_recall(summarize, cases):
"""cases: [{"doc": str, "must_survive": [str, ...]}]
Each phrase marks one exception, e.g. "pooling", "3 to 0"."""
kept = total = 0
for case in cases:
summary = summarize(case["doc"]).lower()
for phrase in case["must_survive"]:
total += 1
kept += phrase.lower() in summary
return kept / total if total else 1.0
def test_exception_recall_does_not_regress():
# Pick a floor that matches your risk; tighten it as you improve.
assert exception_recall(summarize, CASES) >= RECALL_FLOOR
String matching is crude. A stronger version uses a second model to judge whether each exception's meaning survived. But even the crude check turns "our summaries feel fine" into a number that fails CI when a prompt or model change quietly makes things worse.
Key Takeaways
A summary is a more typical version of your document, not a smaller one.
The highest-information sentences (the surprising ones) have the weakest pull to survive.
Watch for five types: conditions, negations, hedges, dissent, temporary states.
Distrust summaries most when the document claims to be routine.
"Be accurate" backfired in research. Name what to protect and ask for verbatim quotes instead.
If you ship summaries: add an
exceptionsfield and track exception recall in CI.
FAQ
Are AI summaries accurate?
Often every sentence in an AI summary is accurate, and the summary is still misleading. The common failure isn't invention; it's omission. Summaries keep what is typical for that kind of document and tend to drop conditions, negations, hedges, dissent, and temporary states, which are often the sentences that change what you should do.
What do AI summaries usually leave out?
Sentences that limit, reverse, or doubt something else: conditions ("except when pooling is on"), negations ("we decided not to"), hedges ("not confirmed in prod"), dissent ("one reviewer objected"), and temporary states ("disabled until the backfill finishes").
Why do AI summaries drop important details?
A language model produces likely text, and a summary pulls toward what summaries of that document type usually say. The most informative sentences are the least predictable ones, so they have the weakest pull to survive. Helpfulness tuning may also favor confident, general statements over hedged, precise ones.
Does telling the AI to "be accurate" improve summaries?
Not reliably. In a 2025 study of 4,900 LLM summaries of scientific papers, prompting models to avoid inaccuracies roughly doubled the odds of overgeneralized conclusions. Naming exactly what to preserve, and asking for verbatim quotes, is more effective.
How do I get an AI to keep caveats and exceptions?
Use a separate extraction step: ask the model to quote every condition, negation, hedge, disagreement, and temporary state, with locations, and to say "none found" if there are none. Then ask what would surprise someone who has read many similar documents. Use low temperature when you can.
How do you evaluate a summarization feature for missing information?
Measure exception recall: mark the exception sentences in a set of real documents, run your summarizer, and count how many survive. Start with simple phrase matching, move to a model-based judge for meaning, and add a recall floor to CI so prompt or model changes can't silently regress it.
Related reading
Your JSON Schema Is Forcing the AI to Guess: why a schema is a questionnaire, and why missing fields force guesses.
Temperature 0 Won't Fix Your AI. It Just Hides the Doubt.: what temperature does and doesn't change.
Why AI Forgets What You Said: lost-in-the-middle, a different (positional) failure.
AI Is Brilliant at Everything You Can't Check: why your error-detection rate depends on what you already know.
New here? Start Here for the full Simply Explained map.
The bottom line
Summaries are great at telling you what kind of document you're holding. They're unreliable at telling you what makes this document different, and in engineering, that difference is usually the reason it was written.
Read the summary to learn what kind of document it is. Read the exceptions to learn what this document is.
Adam Jaber is a software engineer who writes Simply Explained: complex topics, made simple. No jargon, no hype. Get one in your inbox each week: subscribe to the newsletter.
