Unrepresentative Sample

_{Jan 01 . 2010}

The sample used in an inductive inference is relevantly different from the population as a whole. Sample size does not overcome sample bias.

Sampling is a technique used by pollsters. It is a device for gathering information about an entire population from a small subset — a sample. A representative sample is one in which whatever features in the overall population deemed relevant to the issue at hand are represented in roughly the same proportions as these features are found in the population.
Johnson & Blair, Logical Self-Defense, p. 71.

[To Add: 2) Tautological Sampling 3) Tendentious Sampling]

Examples

A psychiatrist reported once that practically everybody is neurotic. Aside from the fact that such use destroys any meaning in the word “neurotic,” take a look at the man’s sample. That is, whom has the psychiatrist been observing? It turns out that he has reached this edifying conclusion from studying his patients, who are a long, long way from being a sample of the population. If a man were normal, our psychiatrist would never meet him.
Darrell Huff, How to Lie with Statistics, p. 20.

When Puerto Rico was hit by a recent hurricane, there were 10,000 claims by residents for hurricane damage. The US Government decided to base its total grant aid on finding the total of claims in the first 100 applications and then multiplying by 100. A colleague was involved in the difficult task of persuading the US government that the first 100 applications need not necessarily constitute a representative sample! Small claims are likely to come in first as they need less preparation.
Chatfield, Problem Solving, p. 16.

The Alfred Kinsey reports on male sexual habits are an example. Although he did not employ a sampling design, he interviewed only a small sample of Americans. Hence his results were a sample of the whole, and from them he reached some pretty striking conclusions. Kinsey found that one in ten men were homosexual, one in two had committed adultery, and one in six had been victimized by or had victimized another family sexually. Recent research into his data has discovered that his sample included prisoners and hospital patients in considerably larger numbers than their proportion of the actual population.
Hoffer, The Historians’ Paradox, p. 83.

In 1936, the Literary Digest mailed out ten million ballots in a political poll to try to predict whether Franklin Roosevelt or Alfred Landon would win the upcoming election. According to the two million three hundred thousand ballots returned, it was predicted that Landon would win by a clear majority. The names for the poll were randomly selected from the telephone book, lists of the magazine’s own subscribers, and lists of automobile owners. In this case, when Roosevelt won with a 60 percent majority, it turned out that the poll was biased because it was selected from higher-income respondents (owners of cars and telephones, at that time), who tended to be Republicans. The poll was biased then, because of this association between income bracket and party preference.
Walton, One-Sided Arguments, p. 225.

Critique

Show how the sample is relevantly different from the population as a whole, then show that because the sample is different, the conclusion is probably different.

Commentary

In addition to being large enough, a good sample should be representative of the population from which it is drawn. Indeed, the more representative a sample is, the smaller it needs to be to be significant. When we reason from a sample that isn’t sufficiently representative, we commit the fallacy of the unrepresentative sample (sometimes called the fallacy of biased statistics, although that name also applies to cases where known statistics that are unfavorable to a theory are deliberately suppressed).
Kahane & Cavender, Logic and Contemporary Rhetoric, p. 92.

On the importance of a sample being representative over being large…

A noteworthy subtype of the hasty generalization fallacy is hasty inference from polling results. In public-opinion polling or survey research, the incidence of an attitude, opinion, or preference measured in a small subgroup or “sample” is generalized to the larger group or ‘population’ under study. Gallup and Harris polls, for example, generalize about national voting preferences on the basis of samples of fewer than 2,000 people. A national television commercial campaign will be tested first in one region of the country to see if sales increase there before expensive national network time slots are purchased. Inferences drawn about populations that are based on surprisingly small samples can be highly reliable if the sample is appropriately selected and the method of measurement is appropriately designed and implemented.
Blair in Brockriede et al., Perspectives on Argumentation, 2006, p. 121.

On sample size and the inductive strength…

In the classic literature, one of the standard marks of a sample’s unrepresentativeness is its smallness. Traditional approaches to the fallacies seize on this factor, making hasty generalization the fallacy of mis-generalizing from an over-small sample. By these lights, any would-be analysis of this fallacy must take due notice of two factors. It must say what a generalization is. It must also say what is lacking in the relationship between an over-small sample and the generalization it fallaciously “supports”. In traditional approaches, this is all rather straightforward. A generalization is a universally quantified conditional statement. And what the over-small sample fails to provide for it is inductive strength, or high (enough) conditional probability. Let us remind ourselves that fallacies are target-relative and resource-sensitive. So we must take care to observe that, even as traditionally conceived of, hasty generalization is not a fallacy as such. It is a fallacy only in relation to a cognitive target of which the production of an inductive well-supported universally quantified conditional is an attainment-standard.
Magnani et al., Model-based Reasoning in Science, Technology, and Medicine, p. 81.