Detecting Truncated LLM Output: finish_reason, stop_reason and the max_tokens Tail

This is the shape of the line most tutorials use to read a model's answer:

text = response.content[0].text

It reads the answer. It never asks whether the answer is finished. And the responses where that question matters most are not a random sample of your traffic. They're your biggest and hardest inputs.

This post covers what the stop signal is on each major API, why max_tokens truncation concentrates in the tail of your input distribution, the three layers that hide it, and a small gate that makes it impossible to miss.

The field next to the text

Every major model API returns a stop signal alongside the generated text. They're named differently, but they map to the same outcomes:

Two details matter more than the table. First, only the "finished naturally" row means the model completed its answer. Second, none of these rows is an HTTP error. Anthropic's docs state that refusals come back as a normal 200. A cut-off response is a 200 too.

What a cut looks like, by output shape

max_tokens is a meter outside the model, not an instruction inside the prompt. When the count hits the limit, generation stops wherever it is. The damage depends on what the model was writing:

• Prose: stops mid-sentence. Loud.
• Strict JSON: fails to parse. Loud, unless something repairs it.
• Lists and line-delimited records: lose their tail. Every item before the cut is complete, so dropping the partial last line produces something that looks finished. Silent.
• Summaries: cover the first part of the document and never mention the rest. Silent.
• Tool calls: Anthropic documents the case where a max_tokens response ends with an incomplete tool_use block. The action never runs. Silent, and the most damaging. (Background on how these calls work: tool calling and MCP explained.)
• Streams: Anthropic delivers stop_reason in the final message_delta event. A handler that saves on stream close never sees it.

Why truncation concentrates in the tail

If length stops were spread evenly, a 0.5% rate would be a rounding error. They aren't spread evenly. Two forces push them toward the requests that matter most.

Output size tracks input size. A 200-line invoice needs roughly twenty times the extraction output of a 10-line one. A translation is about as long as its source. Your max_tokens is one number; the output your inputs need is a long-tailed distribution.

Reasoning spends the same budget first. OpenAI's reasoning guide says reasoning tokens are billed as output tokens, occupy context, and can range from a few hundred to tens of thousands depending on problem complexity. Anthropic's extended thinking docs say thinking tokens count toward max_tokens, and in manual mode budget_tokens must be less than max_tokens. Harder problems think longer and leave less room for the answer. (When that thinking helps at all is its own question: when to use reasoning models.)

The extreme case is documented: OpenAI notes a response can hit the limit before any visible output is produced. Empty answer, billed reasoning.

The consequence for testing is the important part. Eval sets are usually sampled from typical inputs, and typical inputs fit. Your evals pass for the same reason production fails.

Three layers that hide it

1. The status code. 200 means delivered.
2. The repair layer. Libraries like json_repair say they close missing brackets, fix truncated values and fill missing values with defaults so the output is valid. Useful on a finished answer with a stray comma. On a cut answer, it converts a failure into a shorter valid result. A schema guarantees shape, not truth, and not completeness either (structured outputs still hallucinate).
3. The success metric. Error-rate dashboards count exceptions. A length stop doesn't raise one.

The gate: read the reason before the text

Make the stop reason a required branch, and make unknown values fail closed. Stop-reason enums grow over time.

Anthropic Messages API

class Truncated(Exception): ...
class Refused(Exception): ...
class NeedsToolRun(Exception): ...
class UnexpectedStop(Exception): ...

def finished_text(resp):
    reason = resp.stop_reason
    if reason in ("end_turn", "stop_sequence"):
        return "".join(b.text for b in resp.content if b.type == "text")
    if reason == "tool_use":
        raise NeedsToolRun(resp)
    if reason in ("max_tokens", "model_context_window_exceeded"):
        raise Truncated(reason, resp.usage.output_tokens)
    if reason == "refusal":
        raise Refused(resp.stop_details)
    raise UnexpectedStop(reason)  # pause_turn, and anything added later

OpenAI Responses API

def finished_text(resp):
    if resp.status == "incomplete":
        raise Truncated(resp.incomplete_details.reason)  # e.g. "max_output_tokens"
    if resp.status != "completed":
        raise UnexpectedStop(resp.status)
    return resp.output_text

OpenAI Chat Completions

choice = resp.choices[0]
if choice.finish_reason == "length":
    raise Truncated("length", resp.usage.completion_tokens)
if choice.finish_reason not in ("stop", "tool_calls"):
    raise UnexpectedStop(choice.finish_reason)

Gemini (google-genai)

from google.genai import types

cand = resp.candidates[0]
if cand.finish_reason == types.FinishReason.MAX_TOKENS:
    raise Truncated("MAX_TOKENS")
if cand.finish_reason != types.FinishReason.STOP:
    raise UnexpectedStop(cand.finish_reason)

Put any repair step after this gate. Repair is for syntax noise on finished output, never for a length stop.

Sizing: raise the cap, then stop relying on it

Raising max_tokens is usually the right first move after a switch to reasoning models; OpenAI recommends reserving at least 25,000 tokens for reasoning and output when you start. But it moves the tail rather than removing it: every model has an output ceiling, input and output share the context window, and the cap doubles as your runaway-cost circuit breaker.

What actually works:

• Measure the tail. Log output tokens (reasoning broken out where reported) per request, bucketed by input size. Size the cap from the biggest bucket plus headroom, and alert on length-stop rate per bucket.
• Bound the output, not the input. Split by how much the model must write: extract lines 1–50, then 51–100; one section per summary call. More calls re-read the same context, so keep the shared prefix cache-friendly (prompt caching for AI agents).
• Prefer retry or split over continuation for structured output. Continuing a cut JSON array risks duplicated or skipped items at the seam, and a response cut during reasoning has no visible partial to continue. For a cut tool call, Anthropic's guidance is to retry with a higher max_tokens.

Leave a finish line, then test the tail

The stop reason is the primary signal. A finish line catches the layers that drop it (proxies, streaming handlers, text-only wrappers):

• a required last field in the JSON schema, which a cut response can't contain;
• a fixed end marker for free text;
• a count check against an independent source, such as your own parser's line count for the invoice.

Then add the inputs your random sample will never include:

import pytest

BIGGEST = load_cases("largest_inputs.jsonl")        # top 10 by size
HARDEST = load_cases("most_reasoning_tokens.jsonl")  # top 10 by reasoning usage

@pytest.mark.parametrize("case", BIGGEST + HARDEST, ids=lambda c: c["id"])
def test_tail_inputs_finish(case):
    resp = call_model(case["input"])
    assert stop_reason(resp) in NATURAL_STOPS
    items = parse_items(resp)
    assert len(items) == case["expected_count"]

load_cases, call_model, stop_reason and parse_items are your own helpers; the point is that the suite is built from the tail, not sampled from the middle.

Key Takeaways

• A truncated LLM response returns HTTP 200. The stop reason (finish_reason, stop_reason, status, finishReason) is the only built-in signal.
• Truncation isn't random. Output grows with input, and reasoning tokens spend the same max_tokens budget first, so the cut concentrates on your largest and hardest requests.
• Lists, repaired JSON, summaries, empty reasoning replies and cut tool calls all look finished.
• Gate on the stop reason before reading text; fail closed on unknown values; repair only after the gate.
• Size caps from the tail, split work by output size, add a finish line, and test with your biggest and hardest real inputs.

FAQ

What does finish_reason "length" mean in the OpenAI API?

It means generation stopped because the response reached the output token limit you set, not because the model finished. The text you received is partial. On the Responses API the equivalent is status: "incomplete" with incomplete_details.reason set to max_output_tokens.

Does a truncated LLM response return an error?

No. A response cut off by the token limit returns HTTP 200 like any other. Anthropic's docs note that even refusals arrive as normal 200 responses. You have to check the stop reason field to know whether the answer is complete.

Do reasoning tokens count against max_tokens?

Yes, on the major APIs. OpenAI bills reasoning tokens as output tokens and counts them against the output limit; a response can run out before any visible text appears. Anthropic's docs say thinking tokens count toward max_tokens, so the limit must leave room for the final answer.

Why does an LLM return an empty response?

With reasoning models, a common cause is that reasoning consumed the entire output budget, leaving nothing for the visible answer; the stop reason will show the limit was hit. Raise the limit or split the task. Anthropic also documents a separate case of near-empty end_turn responses after tool results.

Should I continue a truncated response or retry it?

For prose, continuation (sending the partial back and asking the model to keep going) works reasonably well. For structured output, prefer retrying with a larger limit or splitting the task, since seams can duplicate or skip items. For a tool call cut mid-arguments, Anthropic recommends retrying with a higher max_tokens.

How do I choose a max_tokens value?

Measure real output token usage, including reasoning tokens, grouped by input size, and size the limit from your largest group plus headroom rather than the average. Keep a cap as a cost circuit breaker, and split work so no single call needs an unbounded output.

Related reading

• Structured Outputs Still Hallucinate: Designing LLM Schemas That Can Say "I Don't Know"
• When AI's 'Thinking' Mode Actually Makes It Worse
• How AI Learned to Actually Do Things: Tool Calling and MCP Explained
• Prompt Caching for AI Agents: Stop Breaking Your Own Cache
• Offset vs Cursor Pagination: Why Your Pages Skip and Repeat Rows: the same "success logged, rows missing" bug, in a database.

The bottom line

An HTTP 200 from a model API is a delivery receipt. The stop reason is the line that says whether the answer was paid in full, and the requests most likely to come up short are the ones you most needed.

Don't ask whether the AI answered. Ask why it stopped.

New to the series? Start Here: a plain-English map to understanding AI. Get one of these in your inbox each week: subscribe on Substack.

Adam Jaber is a software engineer who writes Simply Explained: complex tech topics, made simple. No jargon, no hype.