Some AI products now show a panel labeled thinking, reasoning, or thoughts. Some models spend extra computation before answering, and some products expose a version of that process to you. Depending on the product, that panel is one of three different things. It might be raw scratchpad text the model produced as it worked. It might be a generated summary of that work. It might be a vendor-shaped trace, assembled and filtered for display. These are not three views of one object. They come from different mechanisms and track the real computation to different degrees. It lands before the answer, so it feels like you're watching the model think.

That feeling is the trap.

You ask for a recommendation and the model gives you one. You're careful. You don't just take it. You expand the panel: weighed this, ruled out that, landed here. Clean steps, the kind you'd have written yourself. You sign off, satisfied you checked its reasoning.

You checked coherence, not causality. The one line in your prompt that flipped the recommendation? The panel might name it, might not. Either way, nothing there can prove it drove the call.

We've all done this, and I build these systems. I still catch myself reading it like a receipt. So let me be blunt: it's the most convincing thing in the interface, and the wrong thing to treat as proof.

The panel is model-generated or model-derived text, not an instrumentation log of the computation underneath. The real computation is an enormous tangle of numerical operations across the network. The panel may be part of the run, or a summary of it, but it is not a complete English transcript of what caused the answer.

In some systems, that reasoning text may be useful working text and may even shape the answer. But the panel displaying it is not an audit log.

Recent interpretability work from Anthropic makes the same point from inside the model. This was not a test of any product's thinking panel; it was a study of the model's internal representations. The researchers found a representation that could drive an answer even when the relevant word never appeared in the visible text. An internal "spider" representation helped produce the answer "eight"; swapping that representation toward "ant" changed the answer to "six." Read it as an illustration, not a verdict: a causal internal representation can be absent from the prompt and answer.

A 2026 preprint from Richard Young studied a related, observable gap in open-weight models. Researchers inserted misleading hints, then analyzed cases where the model chose the hint's target rather than the ground truth.

In the cases where the model followed the misleading hint, the thinking tokens (the reasoning text the model generates before its answer) often revealed it while the final answer did not. In 55.4% of those hint-influenced cases, the trace contained hint-related keywords the final answer left out. On its own, that gap does not prove the trace was unfaithful, but it is common. In a further 11.8% of the same cases, neither the thinking tokens nor the final answer acknowledged the hint.

The paper frames this as models that "know but don't say." The practical lesson is less dramatic: the reasoning trace shown in the panel may reflect parts of the process and may even have contributed to the answer, but it cannot independently verify which factors caused that answer.

Even measuring trace faithfulness is unsettled. In a companion preprint, three classifiers scored the same reasoning traces differently enough to reverse which models looked more faithful.

None of this means reasoning traces are useless. The mistake is treating useful working text as accountable evidence. So when the reason behind an answer actually matters (one you're about to act on or put your name to), stop reading the panel harder. Test it.

Name the check for what it is: a sensitivity check, not a reconstruction of the model's full causal process. Before you touch anything, decide what outcome you're testing for, so you're measuring against a fixed target. Then run the original prompt as it was, several times. These models don't answer identically every time, so this gives you the baseline output variation: how much the answer moves on its own. Now change or remove the one input you suspect drove the answer, and hold everything else constant: same model, same settings, same system instructions, same tools, same context. Start every run in a fresh chat, because prior messages feed the answer too. A low temperature or a fixed seed (controls that reduce or pin down the model's run-to-run variation) may improve repeatability, but neither guarantees identical runs. Run each version at least five times for a quick check, more when the decision carries real weight.

Then compare. If the edited version reliably lands somewhere different, beyond that baseline, the input was probably shaping the answer. If it keeps landing in the same place, you have no evidence it mattered here, though that isn't proof it had none. And for a decision that really matters, remember that inputs can interact: one clean edit shows whether a single input is sensitive, not the whole causal structure behind the answer.

Make it concrete. You paste two closely matched vendor proposals, an agreed scorecard with weights, and meeting notes. Buried in the notes is an unsupported claim: "Vendor B is the likely winner." Decide up front what you're testing: which vendor ranks first. Ask the model to rank both using only the scorecard, several times in fresh chats. It puts B first on a tidy rationale that never mentions the claim. Even with a fixed scorecard the runs aren't identical, so that variation is your baseline. Delete only that claim and run it the same number of times. If B still leads with the claim but A becomes common without it, beyond that baseline, you have evidence the claim shaped the result. If both behave the same, it probably wasn't carrying the result.

What reliably moves once you account for that baseline is the part you don't have to infer from prose alone. Not every answer earns the effort, and most won't. But when the "why" is the part you're accountable for, evidence you generated beats a story you read.

Here's the line to hand the colleague who treats the "thinking" panel as proof of why the model answered: