Little Questions® and Big Data

The technical term for asking people questions is “elicitation.” At Lone Star we call it “Little Questions^®”. We think elicitation is a big deal, and we trademarked the name of how we do it.

A reasonable person might ask why we’d bother to register a trademark for this. After all, there are lots of ways to collect data, labels, classification, and other information from humans.

There are the standard business answers to the question. Of course, we wanted to stress our own methods for collecting data from humans. And, because we are driven with a high degree of process discipline, we want to stress that.

But, those fall short of explaining why we thought it was important enough to spend the money and obtain Registration Number 4978406 from the U.S. Patent and Trademark office.

One of the fourteen best practices identified in our three-year international analytics benchmarking effort was “Accommodating Uncertainty.” One component of this is accommodating human cognition of uncertainty.

Recent brain science shows there are many kinds of mental processes in the face of uncertainty. Uncertainty which feels familiar is processed in a different part of the brain than uncertainty which feels unfamiliar. Sam Savage recently wrote a great piece on this, based on Robert Sapolsky’s Behave.

There are several challenges we need to overcome when we elicit information from people. Kahneman and Tversky showed how bad we are at knowing what we know. Ask an expert about something outside their domain, and they will be nearly as confident on that topic as things where they really are an expert.

Doug Hubbard (and others) have done some great work showing that we can train people to calibrate their uncertainty. We can help them distinguish topics where they have little or no uncertainty, from topics where they have big uncertainties.

A “little question” is something a person can answer easily and naturally. It has to do with comfortable cognition. Humans are more reliable when asked this kind of question.

Some problems with a great deal of big data collected from the web relate to the ideas behind Little Questions^®.

• Little Questions^® are asked in a neutral way to avoid bias
• Little Questions^® are posed to qualified audiences who can easily answer
• Little Questions^® are framed to elicit both the answer and some measure of uncertainty

Anyone who has tried to conduct big data analysis knows how hard it is to achieve this kind of standard. For example, when we look at health status data from large populations, where does it come from? It comes from doctors and hospitals, so it is biased toward people who are sick. Even the people who are not sick are stressed out by their environment.

So, a true Little Question^® is the gold standard for data collection. That’s why we work hard to use Little Questions^® and that’s why we trademarked it.