Suppose a website asks its visitors how many hours they sleep, and 10,000 people answer. A national survey asks 1,000 people chosen at random. Which tells you more about how much the country sleeps?

Almost always the survey, even though it is ten times smaller. The reason is the difference between a random sample and a self-selected one.

Who answers matters more than how many

In a well-run random survey, everyone in the population has a known chance of being picked, and the survey works hard to reach the people who are picked, including those who are busy, offline or uninterested. That's what lets a few thousand answers stand in for millions of people, with a margin of error you can calculate.

On a website, the people who answer are the people who happened to visit, and then chose to take part. Each of those steps filters who is in the sample:

  • Who visits: people who use the internet, found the site, and are interested in its topic.
  • Who answers: people with the time and the inclination, and often people with something unusual to report.
  • Who answers honestly: a few will give silly numbers for fun.

A made-up example: a poll on a site for gamers asks "how many hours a day do you spend on screens?" Its visitors probably spend more time on screens than the population as a whole, so the poll's median will be high however many people answer. Doubling the number of answers makes the wrong answer more precise, not more right.

This is called selection bias, and a bigger sample does not cure it. Some of the most famous polling failures in history came from huge samples drawn from the wrong people.

So are visitor answers worthless?

No: they are interesting on their own terms. For many everyday questions (how many browser tabs, how many unread emails, how many photos on a phone) there is simply no official statistic. Answers from thousands of people still show the range of what's normal among the people who answered, and that is worth knowing as long as nobody pretends it is more.

When AboutAverage publishes figures from its own visitors, we will always:

  • say it is a voluntary sample of our visitors, not a national figure;
  • give the number of answers and the dates they were collected;
  • show the spread (the middle half of answers, not just one number);
  • wait until at least 100 people have answered before publishing anything, and never show a small group on its own;
  • remove obvious duplicates and automated answers, and say how many we removed;
  • keep our visitors' figures separate from official statistics, never mixing the two in one number.

We also never invent starting values to make a question look busy. If nobody has answered yet, we say so.

A quick test for any poll

Ask: could the people who answered be different, as a group, from the people the headline is about? If yes, and they usually are, read the result as "among those who answered", not "among everyone".