Decide the reach before you read the result

Four people replied. Is that a signal? The honest answer depends entirely on a number you were supposed to write down before you sent anything, and the reason the question feels hard afterwards is that it has already been contaminated by the four replies.

This is not a failure of statistical literacy. Nobody at the frontier is running a power calculation on twelve conversations, and they should not be. It is a failure of sequence. Stating 'twelve people from this population, and I need at least five to do the thing' takes fifteen seconds before the experiment and becomes impossible to state honestly at any point after it.

Small numbers are legitimate here. Working before metrics exist means working with counts that would embarrass a researcher, and the alternative — waiting for a sample that supports inference — means waiting until the question no longer matters. What is not legitimate is leaving the count unstated so that any outcome can be read as directional.

The population matters more than the count anyway. Five replies from people who already pay you and five from people who have never heard of you are different experiments with the same arithmetic. Naming the population in advance is what makes the difference visible later, and it is also what stops the most common quiet substitution: sending to the easiest-to-reach group and interpreting the result as if it came from the intended one.

The test for whether you did this properly is simple and slightly unpleasant. Before the results come in, write the sentence you will say if the answer is no. If that sentence does not exist yet, you have not designed an experiment — you have arranged for some encouragement.