The Myth of The “Average”

We bandy about the word “average” all the time. What exactly IS an average, and how does it help design research?

Use the average to quickly summarize something that is already a number: minutes, ages, heights, visits, etc. Don’t use the average to explain something that needs more detail. And keep in mind, the average gets “dragged up” or “dragged down” by extreme values. Sometimes it doesn’t tell you much of anything.

An example design research project might be about how people use their stoves in their kitchens. How can we use “the average” to help us design a new stove?

The average, in statistical language, is actually called “the mean,” which is a measure of “central tendency.” Researchers use central tendency to describe all their results quickly. Other measures of central tendency include the mode (the most common response) or the median (50% of responses are higher than this; 50% are lower).The mean describes the “typical” or average result.

But here’s the big myth: there is no such thing as “the average” in your data. If you ask 500 people to rate your new stove design on a scale of 1 to 10, and the average is 4, there is no guarantee that any single person actually said 4! In fact, the majority of responses could be higher than 7, but some 1s or 2s could “drag down the average.”

Worse, it makes no sense to use the “average” or “typical” in qualitative research. If you do interviews or observations, there is no way to calculate “the average.” So when you say, “the typical person has a four-element stove,” you’re actually doing a calculation. This may be actually quite false. What you may mean to say is “most people in our study have a four-element stove” (which is the mode).

Qualitative research does not accept the “typical.” It actually looks at each case individually and in enough detail to allow for exceptions or outliers. There is no “typical” case in qualitative research because you do not do calculations. You do not summarize your data in that reductionist way.

That said, how could you use “the average” in your kitchen stove study?  You can do a back-of-the-envelope calculation to summarize your data. The “average age” of your respondents, for example, will tell you about how old people are. The “average number of minutes spent cooking” will give you a snapshot of how long people spend in their kitchens. The “average purchase price of a stove” will also give you a quick snapshot. Using “the average” is to quickly summarize something that is already a number.

But the “average use of the stove”? That doesn’t make sense. Nor does the “typical grocery shopping process,” or the “average complaint of stove use.” These cannot be summarized in “the average.”

Online Surveys 101

Folks,

Below is a (very!) brief overview of online surveys. This slideshow, via slideshare, is intended for people in the Web design industry. IAs, designers, media planners, strategists, usability researchers, and producers will learn if they should, in fact, do a survey.

Why customer satisfaction surveys are useless

Many readers seem to enjoy my qualitative versus quantitative research post. I take this to mean that designers are hungry insight that beyond the requisite (and useless) customer satisfaction survey.

I’m not a huge fan of customer satisfaction surveys because they are usually 100% reliable but 0% valid; they tell you nothing (but consistently tell you nothing). Witness, for example, the Foresee customer satisfaction survey. This survey is designed to give pop up as a user leaves a Web site. They are asked a variety of customized questions and then a variety of demographic information. Foresee tallies these results regularly and even ranks your Web site (or company in general) in comparison to your competitors.

What does a designer learn from this? Almost nothing.

Why? Several reasons.

1. Consumers have “satisfaction” fatigue: consumers are surveyed to death these days. We are all familiar with the Likert scale “strongly agree” to “strongly disagree.” Few survey researchers actually rework their surveys for validity. It’s called “acquiescence bias” where people tend to just answer the same way repeatedly. Survey researchers who know better use reverse-scoring techniques; but most don’t. These surveys, therefore, result in a questionable assertion that they are actually valid representations of how consumers actually feel.
2. Satisfaction surveys breed incremental “metricism”: surveys tell you nothing new: designers need to innovate their products, Web sites, and images. Satisfaction surveys tell you nothing you don’t already know. What’s more, they may actually inhibit creativity because they draw attention to minute changes that may be due to chance alone. Once organizations become regular consumers of satisfaction surveys, even small improvements become cause for celebration — even if they don’t reflect real improvements (see number 1).
3. Surveys provide numbers, not detail: designers need thick description to make their designs truly evocative of lived experiences. Satisfaction surveys are simply stripped down representations of how people feel (or more accurately, how they say they feel). Designers need richer information to spark creative solutions.
4. Customer satisfaction is a poor predictor of looming competition: Imagine a company that had consistently high customer satisfaction scores. Imagine also that this company falls victim to “incremental metricism,” and fails to see a competitor’s new, better designed product on the horizon. This competitive product would never appear in customer satisfaction surveys. It’s possible for customers to be “satisfied,” only to have them lured away by an innovative, better design.

Instead of customer satisfaction surveys, I recommend designers pore over free trend-spotting data, like the Pew Internet and American Life Project, which is a comprehensive and rigorous survey of current attitudes and beliefs. It’s harder to pull out insights from data that don’t look like they’re relevant, but the return is so much better than from a tired, staid customer satisfaction survey.

When to do qualitative and qualitative research

In a previous post, I talked about what designers need to know about economic class. How did we learn that economic class can be “seen” in designs? How did we learn that “refined” taste is “upper” class?

In general, use qualitative research at the beginning of a design process to uncover innovations. Use quantitative research at the end of a design process to measure improvement.

It started with qualitative research, and became “refined” (no pun intended) with quantitative research. French sociology Pierre Bourdieu followed a typical arc to the narrative research by first investigating economic class in an open-ended fashion. Once he established what he thought was going on, he tested these ideas with large surveys.

If you know little about the topic, start with the qualitative. This means ethnographic observation and in-depth interviewing. Open ended questions are best. At this stage, you’re trying to find the lay of the land. If you’re designing a new car stereo for example, you may wish to start by watching people use their existing car stereos. Maybe drive around with them and ask them questions about what they like.

Once you’ve learned the basics of car stereo requirements, user needs and pain points, it’s time to test your assumptions. This is where the quantitative comes in. Close-ended questions are best here, including multiple choice, yes/no, or simply number of “successes.” Let’s say you’ve learned through your observations that people don’t like how their stereos require programming their radio stations. It’s too much bother, they told you. You think pre-programmed stations might be a good design improvement, so you create a new stereo with pre-programmed stations.

Did it work? Ask your stereo users how they like the new system after they have bought their new car. But the question is, compared to what? This is where quantitative research gets tricky. You can compare the new stereos on select models (58% of users of the new model are very satisfied, while only 32% of users of the old model are). Or you can compare before and after the improvement — the so-called “pre-and post test.” That requires time, foresight, and — you guessed it — budget.

Below is a diagram that summarizes the research “funnel” from exploration to validation.

Qualitative versus quantitative research

Many designers are self-taught, intuitive consumers of research who can translate insights into great designs. But few are trained in the arcane art of research itself. For that reason, many designers don’t know the finer differences between qual and quant research and end up using their respective results inappropriately.

Quantitative research is based on the assumption that random events are predictable, and if you compare your results to pure random results, you can discern distinctive, meaningful patterns about the social world.

Random events are relatively MORE predictable if you have more of them. Imagine if you flipped a coin 20 times. How many heads would you get? Now if you flipped it 20,000 times? You’re more likely to get an even 50/50 split — which is what most people would predict. If you got a 65/35 split with 2o flips, okay, could happen. But with 20,000 flips? No way. Something else is going on.

Translate that to design research by looking at gender, for example. Let’s say you have 20 people, 10 men and 10 women. 65% of the women choose one design, while only 35% of the men do. Is this a meaningful pattern? Impossible to say — you only have 20 people. Now if you had 200 people (100 men and 100 women) and 65 of the women chose one design, chances are you have a meaningful pattern.

This is why sample size matters in quantitative research. But, little known fact, sample size is COMPLETELY IRRELEVANT in qualitative research. Why?

Qualitative research assumes that people have meaningful experiences that can be interpreted. Notice how there’s nothing in there about “prediction” or “randomness.” People have experiences. Researchers discern what these experiences signify. That’s it. Sample size is not only irrelevant, it actually gets in the way of important insight.

Consider the case study, for example. Few people would say case studies are useless. We can learn a great deal about a single design case, where it went wrong and where it went right. The problem comes when you try to predict future events based on this single event.

If you abandon the need for prediction, then sample size never matters. You can always derive insight about design problems from even a single case. Designers that attempt to predict “success” of a single design change, for example, should test that change, repeatedly, with a probability sample.

Design research, step by step

I often have people ask me how to go about a design research project. Here’s a handy step-by-step guide..

1. The Research Question: this isn’t the same as a research topic. Research questions are answerable in a finite amount of time and yield specific, actionable answers. Some good examples:
   • “What do senior citizens find frustrating about taking their prescribed drugs?”
   • “How do high school students typically study for their driver’s licenses?”
   • “What would working mothers find valuable in shopping for groceries?

   Some bad examples:

   • “What’s an interesting new way to deliver online medical records?”
   • “How can we improve the driving experience?”
   • “What’s wrong with our marketing strategy?”
2. Determine the Method: This is likely the hardest part because every method presents a potential drawback. In general, if you don’t know much about the topic, use a qualitative method (and no, that does not mean “just do a focus group.”) If you know quite a bit about your topic but want to measure change, improvement or any other knowable quantity, choose quantitative research. That does mean “do a survey” sometimes — but not always.
3. Write and Test the “Questions”: I put that in quotes because sometimes it’s not exactly a set of questions. Maybe it’s a task flow and a set of observations the researcher must make. Sometimes it’s a semi-structured interview. Sometimes it’s a quant survey. Make sure you test these “questions,” even in a quick and dirty way with co-workers.
4. Recruit Respondents: Remember your research question? That should tell you whom you need to recruit. If you can’t figure it out, then you need to revise your research question. Be specific but not too narrow in your choices. The more requirements you impose, the smaller your potential base. If you’re doing a qualitative research project, keep interviewing until you start getting the same answers. This usually starts around 8 to 10 respondents. You’re not interested in “statistical significance” but the experiences of the people you’re talking to. For quant studies, statistical significance does apply and you should strive to have a minimum of 40 respondents. Remember though that the higher the number, the lower your margin of error.Design researchers can also rely on professional recruiters to get people for you. Good professional recruiters should get you the right people for a reasonable fee.
5. Prepare the “Stimulus”: If you’re testing a new office chair, make sure your prototype is ready. If you’re interested in something that doesn’t yet exist, consider using photographs to elicit ideas and reactions from respondents. If you’re testing a “concept” make sure that what you’re testing reflects what you really want to know. For example, a picture of a new office chair may not do you any good if what you really want to know is how comfortable the chair actually is.
6. Set Up The Research Space: This is an under-emphasized by oh-so-important aspect of research. Ethnographic research requires you to select the natural environment of your subjects, for example, and you must ensure you have been granted access to that space. If you’re interviewing, decide what kind of place might be conducive to good answers. Noisy restaurants or malls are unlikely to get you personal information, for example. This matters for quant research as well, as there’s a big difference between an in-person, a telephone, and an online survey.
7. Set Up the Interviews: For qualitative research, this takes a fair bit of back-and-forth. It’s helpful to enlist the help of a junior staff member or an administrator. Keep your records straight!
8. Determine the Data Collection Method: If you’re interviewing people in their homes, a TV camera may not be a good choice. Small digital recorders are available for iPods now (I love mine). Digital photos are also useful, but less discrete. And, if you can spare the staff, have one person dedicated to note taking. This frees up the interviewer to really engage with the participate, establish rapport, and probe for opportunistic findings. For quant research, this question usually involves technical questions like, how am I going to import these data into a data analysis tool?
9. Collect the Data: Do your interviews, watch your tasks, ask your questions, whatever. Remember to take notes throughout. These “field notes” are sometimes the most valuable you can have.
10. Answer Your Question: Remember your research question? This is where it comes in handy. You now know exactly what to do when you’re looking through your photos, or your notes, transcripts or whatever.

Links:

Survey research

Survey design tips

Intro to qualitative methods

Social research methods