Audit the synthesis, not the summary

The economics of qualitative work inverted somewhere in the last two years. Reading three hundred conversations used to be the expensive step, and the summary at the end was cheap because a human who had read everything wrote it. Now the reading is free and the summary arrives first, fluent and ranked, produced by something that has read everything and understood none of it in the way the phrase usually implies.

The output is not usually wrong. That is what makes it hazardous. It is directionally reasonable, well-organised, and carries the tonal markers of having been checked. The specific things it hides are structural: how old the underlying material is, which population could not appear in it, which of the confident claims were added at the synthesis step rather than said by anyone.

One habit covers most of it. Take a single claim from the output — not the one you doubt, a random one — and follow it back to the sentence a person actually wrote. Read that sentence in its original context. This takes two minutes and fails often enough that it stays interesting: paraphrases presented as quotes, two messages merged into one, a percentage computed over a denominator nobody defined.

The other habit is cheaper still: run the synthesis twice, with the framing reversed. Ask what people love, then ask what they tolerate. Two confident, well-organised, mutually incompatible answers is a more honest artefact than either one alone, and the delta between them tells you how much of the output was in the prompt rather than in the data.

None of this argues against using the tools. The synthesis genuinely is worth having; it surfaces patterns a tired human reader misses at conversation two hundred. It just does not arrive as evidence. It arrives as a hypothesis with excellent typography, and the step that converts the one into the other is small, boring, and skippable — which is precisely why it needs a name and a place in the process.