The Myth of The “Average”

We bandy about the word “average” all the time. What exactly IS an average, and how does it help design research?

Use the average to quickly summarize something that is already a number: minutes, ages, heights, visits, etc. Don’t use the average to explain something that needs more detail. And keep in mind, the average gets “dragged up” or “dragged down” by extreme values. Sometimes it doesn’t tell you much of anything.

An example design research project might be about how people use their stoves in their kitchens. How can we use “the average” to help us design a new stove?

The average, in statistical language, is actually called “the mean,” which is a measure of “central tendency.” Researchers use central tendency to describe all their results quickly. Other measures of central tendency include the mode (the most common response) or the median (50% of responses are higher than this; 50% are lower).The mean describes the “typical” or average result.

But here’s the big myth: there is no such thing as “the average” in your data. If you ask 500 people to rate your new stove design on a scale of 1 to 10, and the average is 4, there is no guarantee that any single person actually said 4! In fact, the majority of responses could be higher than 7, but some 1s or 2s could “drag down the average.”

Worse, it makes no sense to use the “average” or “typical” in qualitative research. If you do interviews or observations, there is no way to calculate “the average.” So when you say, “the typical person has a four-element stove,” you’re actually doing a calculation. This may be actually quite false. What you may mean to say is “most people in our study have a four-element stove” (which is the mode).

Qualitative research does not accept the “typical.” It actually looks at each case individually and in enough detail to allow for exceptions or outliers. There is no “typical” case in qualitative research because you do not do calculations. You do not summarize your data in that reductionist way.

That said, how could you use “the average” in your kitchen stove study?  You can do a back-of-the-envelope calculation to summarize your data. The “average age” of your respondents, for example, will tell you about how old people are. The “average number of minutes spent cooking” will give you a snapshot of how long people spend in their kitchens. The “average purchase price of a stove” will also give you a quick snapshot. Using “the average” is to quickly summarize something that is already a number.

But the “average use of the stove”? That doesn’t make sense. Nor does the “typical grocery shopping process,” or the “average complaint of stove use.” These cannot be summarized in “the average.”

The promise (and failure) of Brandtags.net

I loved it when I first saw it. Brandtags.net invites users to look at a logo and type in the first thing that enters their minds. I found it fascinating — until I realized it’s yet another example of poor research perpetuating negative stereotypes of women.

Type in “Oprah” and see what happens. The top three most entered words? Fat. Black. Bitch. Yes, that’s right, Oprah, the maven of women’s media landscape is nothing more than a fat black bitch. How valid a representation of Oprah is this?

Oprah’s media universe is worth a fortune. She earned $260 million in 2007 and is worth $2.5 billion. Her daily talk show alone gets 7.3 million viewers (that’s compared to 2.9 million viewers for Grey’s Anatomy).

So I got to thinking. How is Brand Tags so wrong? So nasty? So racist? (Type in NBA or Citibank and you’ll see what I mean). Researchers are Harvard have shown how stereotypes work. We know that people rely on implicit stereotypes when they make snap judgments. This is the downside of Malcolm Gladwell’s Blink.

We live is a complex social world. We try to make sense out of it by looking for patterns. Theorists Berger and Luckman call these “typifications” or roles that we take for granted. Typifications help us because they allow us to know what to do in social situations without really thinking about it, or, as Berger and Luckman explain it, they alleviate us from making “all those decisions.”

All Brand Tags really does is tell us what those typifications are for the people who visit their site. Who is visiting their site? We don’t really know. The first rule of sampling is to ask yourself, are the people who participate systematically different from the people who don’t?

People who participate in Brand Tags are obviously Web savvy. Someone forwarded them a link and they filled it out. Perhaps they read business media because Brand Tags has gotten some press. They have the time to enter text. They are also anonymous.

Is this what you would consider a “representative sample”?

Brand Tags has promise (I myself have used it to gain insight about a few things). But it mostly has the worst of our stereotypes. Is that insight? Perhaps. But it’s not insight about Oprah — it tells us a lot about the people who are talking ABOUT Oprah.

Online Surveys 101

Folks,

Below is a (very!) brief overview of online surveys. This slideshow, via slideshare, is intended for people in the Web design industry. IAs, designers, media planners, strategists, usability researchers, and producers will learn if they should, in fact, do a survey.

What is design anthropology?

Dori Tunstall has written a fantastic post that details how a simple card sort can become a deeper exercise in analysis. Dr. Tunstall is a PhD anthropologist and the University of Illinois at Chicago. She explains how anthropology takes design research to a deeper level in card sorting, a common technique for information architecture:

“In addition to the information architecture, I delivered statements about the continued meaning of gender classifications. In the course of conducting the card sort, I learned that men and women continued to classify domestic products based on stereotypical gendered spaces of male equals outside/garage, and female equals inside.”

Dr. Tunstall clearly lays out some of the deeper implications of design anthropology later in the post. This is a must read for anyone looking to deepen their design or research practice.

Getting meaningful insights from qualitative research

The output from qualitative research is often overwhelming. Unlike quant research, qual findings are often messy and hard to decipher. Here are some techniques to manage the voluminous data of qualitative studies.

1. Start with clear research questions: in an earlier post I explained how to set up a design research project, step by step. One of the most important steps is to create a clear and answerable research question. This seems like an obvious point, but often it isn’t. Qualitative research often appears to be “just talking to people,” which gives us all the mistaken impression that it is entirely unstructured. It isn’t. Take the time to define research questions.
2. Summarize frequently: Let’s say you’ve chosen to do in-depth interviews. After each interview, take 20 minutes to write out a brief summary of what you remember being the most important points of the interview (note that this is not a substitute to taking notes during the interview). These notes are the first step toward analysis. You are reducing “clutter” and irrelevant information. You are also exploring connections with previous interviews.
3. Reduce, reduce, reduce: You will always have more data (e.g., videos, photos, transcriptions) than you can use. Be ruthless by reducing what’s important. Edit down your videos to only the clips that are most important (keep the raw data for another time). Reduce your transcriptions down to select quotes that speak to your research question (and again, save the entire transcription for another time). The goal is to have a workable set of artifacts.
4. Visualize the results: Many qualitative researchers make use of summary tables and diagrams to further summarize results. My favourite visualization method is the mental model, which can convey a huge amount of information in a synthetic way, quickly. Other tools include mind maps and even the simple bulleted list.
5. Hunt for connections: There is no science to this process. It is iterative and intuitive. But there are approaches you can use to find connections. I frequently use the “open sort” technique, with nothing more than a blank wall and post-it notes. Scribble themes onto post-it notes. Sort them into categories. Name the categories. Collapse as many categories as you can until you only have 4 or 5 “buckets” that explain your findings. If you’re researching children’s commutes to school, for example, you may have a category called “independence” which would talk about kids’ desires to be grown up, to have their own transportation method, and the knowledge to get to school. They are related only through the higher-order notion of “independence” and not the lower-order ideas of “transportation” or “age.”
6. Ask “So what?” often: When I was in journalism school, I had a professor who tirelessly quizzed us with his version of so what: “What does it mean to metro?” he would demand, meaning, why should the people of this city care? Why should your design team care about these results? What does it mean for their process? Why should the users of this product care about your results? How might it make their lives easier or more pleasant? And of course, why should bean counters of all sorts care? How much money will it actually save?

These general guidelines will help you in your journey to deciphering meaning. But no qualitative project can be save from poor research design. Make sure you’re using the right approach.

Why customer satisfaction surveys are useless

Many readers seem to enjoy my qualitative versus quantitative research post. I take this to mean that designers are hungry insight that beyond the requisite (and useless) customer satisfaction survey.

I’m not a huge fan of customer satisfaction surveys because they are usually 100% reliable but 0% valid; they tell you nothing (but consistently tell you nothing). Witness, for example, the Foresee customer satisfaction survey. This survey is designed to give pop up as a user leaves a Web site. They are asked a variety of customized questions and then a variety of demographic information. Foresee tallies these results regularly and even ranks your Web site (or company in general) in comparison to your competitors.

What does a designer learn from this? Almost nothing.

Why? Several reasons.

1. Consumers have “satisfaction” fatigue: consumers are surveyed to death these days. We are all familiar with the Likert scale “strongly agree” to “strongly disagree.” Few survey researchers actually rework their surveys for validity. It’s called “acquiescence bias” where people tend to just answer the same way repeatedly. Survey researchers who know better use reverse-scoring techniques; but most don’t. These surveys, therefore, result in a questionable assertion that they are actually valid representations of how consumers actually feel.
2. Satisfaction surveys breed incremental “metricism”: surveys tell you nothing new: designers need to innovate their products, Web sites, and images. Satisfaction surveys tell you nothing you don’t already know. What’s more, they may actually inhibit creativity because they draw attention to minute changes that may be due to chance alone. Once organizations become regular consumers of satisfaction surveys, even small improvements become cause for celebration — even if they don’t reflect real improvements (see number 1).
3. Surveys provide numbers, not detail: designers need thick description to make their designs truly evocative of lived experiences. Satisfaction surveys are simply stripped down representations of how people feel (or more accurately, how they say they feel). Designers need richer information to spark creative solutions.
4. Customer satisfaction is a poor predictor of looming competition: Imagine a company that had consistently high customer satisfaction scores. Imagine also that this company falls victim to “incremental metricism,” and fails to see a competitor’s new, better designed product on the horizon. This competitive product would never appear in customer satisfaction surveys. It’s possible for customers to be “satisfied,” only to have them lured away by an innovative, better design.

Instead of customer satisfaction surveys, I recommend designers pore over free trend-spotting data, like the Pew Internet and American Life Project, which is a comprehensive and rigorous survey of current attitudes and beliefs. It’s harder to pull out insights from data that don’t look like they’re relevant, but the return is so much better than from a tired, staid customer satisfaction survey.

When to do qualitative and qualitative research

In a previous post, I talked about what designers need to know about economic class. How did we learn that economic class can be “seen” in designs? How did we learn that “refined” taste is “upper” class?

In general, use qualitative research at the beginning of a design process to uncover innovations. Use quantitative research at the end of a design process to measure improvement.

It started with qualitative research, and became “refined” (no pun intended) with quantitative research. French sociology Pierre Bourdieu followed a typical arc to the narrative research by first investigating economic class in an open-ended fashion. Once he established what he thought was going on, he tested these ideas with large surveys.

If you know little about the topic, start with the qualitative. This means ethnographic observation and in-depth interviewing. Open ended questions are best. At this stage, you’re trying to find the lay of the land. If you’re designing a new car stereo for example, you may wish to start by watching people use their existing car stereos. Maybe drive around with them and ask them questions about what they like.

Once you’ve learned the basics of car stereo requirements, user needs and pain points, it’s time to test your assumptions. This is where the quantitative comes in. Close-ended questions are best here, including multiple choice, yes/no, or simply number of “successes.” Let’s say you’ve learned through your observations that people don’t like how their stereos require programming their radio stations. It’s too much bother, they told you. You think pre-programmed stations might be a good design improvement, so you create a new stereo with pre-programmed stations.

Did it work? Ask your stereo users how they like the new system after they have bought their new car. But the question is, compared to what? This is where quantitative research gets tricky. You can compare the new stereos on select models (58% of users of the new model are very satisfied, while only 32% of users of the old model are). Or you can compare before and after the improvement — the so-called “pre-and post test.” That requires time, foresight, and — you guessed it — budget.

Below is a diagram that summarizes the research “funnel” from exploration to validation.

Discourse analysis and design: reading “texts” for design purposes

Designers are already discourse analysts, they just don’t know it. These designers can produce more innovative ideas by adopting a more systematic approach to their intuitive discourse analysis.

Discourse analysis the practice of deciphering the meaning of “texts.” Anything can be a “text.” Television commercials, Us Weekly, a trial transcript — these are all “texts.” Famous discourse analyses include Michel Foucault’s analysis mental illness, in which he traces how we collectively think about mental illness through “texts” of it, such as “patient charts,” or the Diagnostic and Statistical Manual.
Designers intuitively analyze “texts” all the time, especially designers who work in advertising. They obsessively collect imagery and copy they find interesting. They innovate on this copy or imagery by re-tooling some of the subtle messages in them.

How to systematize discourse analysis “lite” for designers:

1. Collect more than one genre of “texts”: instead of a single medium, try collecting several media of the same theme. If you’re designing a new toy, for example, gather a TV commercial, a print ad, and a fan’s tribute Web site. These differing “texts” may tell you what is missing in toys, or what toys are unintentionally doing to the parents who buy them.
2. Look for the “silences” in texts: If you’re designing an online advertising campaign, compare texts on a single theme and ask yourself, “What is not being said?” For example, if you’re targeting women with small children, maybe you’ll find that these women are never painted as actually having personal preferences only “mother preferences.” This is a silence that you can speak to.
3. The obvious meaning is the tip of the iceberg: If you want to know what an object means in culture, you must look more deeply than the obvious. Most designers understand this intuitively, but sometimes you must make a concerted effort. When you see the famous “Diamonds are forever” ads by De Beers, the obvious meaning is one of romance, but what is the subtle meaning? Romance is fleeting but diamonds? Diamonds are forever. The ad’s brilliance lies in its ability to leverage the symbolism of the world’s hardest substance (the diamond) with the most coveted but ephemeral experience (romantic love).

The other day I was tutoring an adult learner (a highly educated one) about discourse analysis. She complained to me that she well understood quantitative methods, variables, and counting. But she saw discourse analysis as “mumbo jumbo.”

On the surface, discourse analysis looks like mumbo jumbo. But in practice, it is a tool to see both culture and the “reality” we have constructed.

Designers as playwrights: scripting design outcomes

Designers don’t really see themselves as playwrights but in reality, designers are writing scripts – complete with stage directions – for every user. And like all actors, what users really want to do is direct.

The French government learned this the hard way. In a fit of charity, the government decided Africa needed electric light. Noting that African countries often lacked centralized electricity systems, French engineers designed battery-powered lights and sent them to Africa. The lights were designed to be robust systems that could withstand the rugged African countryside. It was envisioned that many owners of these lights would proudly use them for decades. Instead, the engineers delivered lights that were difficult to install, very quickly burned out, and proved almost impossible to repair. Quite a few African homes were then decorated with useless battery packs.

What as the problem? French engineers – despite their noble intent – designed lights that were only useful to docile users. The play they wrote was in three acts:

Act 1: turn on light.
Act 2: burn out light.
Act 3: do nothing with the light ever again.

When I made toast this morning in my kitchen, the script writers for the toaster did not consider the “set” of my kitchen, nor did they consider the supporting actor, my husband.

Their script went something like this:

Act 1: User takes two pieces of toast and places them in the two slots. User pushes down the plunger. Toaster toasts the bread. User waits until bread is cool enough to handle, and places toast delicately on a plate. Curtain. Applause.

But the actual script went something like this.

Act 1: Sam pulls bread out of freezer and then pulls toaster out of the cupboard where they store it. Toaster bottom opens up (again) and spills crumbs all over the floor. Sam plugs in toaster and separates two pieces of frozen bread.

She places only one slice in a slot and presses the plunger. She begins chatting with her husband, not noticing that she chose the wrong slot for a single slice. Toast pops up, decidedly still frozen. Curtain.

Intermission: Getting orange juice

Act 2: Sam moves slice to correct slot and presses plunger again. Toast toasts and pops up. Again, while chatting with her husband, she does not realize the bread is very hot. She burns her fingers on the toast, dropping it. Curtain. Curse words.

Design scripts need to be clear, concise, and above all, consider active users. When you design a product, a print ad, or a Web site, consider the script you are writing. What are your assumptions? What is the “set” of the eventual “play”? Are there supporting characters? Consider how you want your script to end before you start writing it.

What’s wrong with ethnography?

Ethnography is bandied about frequently in business and design circles these days. And sadly, like many buzz words, its true meaning has been lost in its popularity. Let me start by saying ethnography is hot today because it provides you insight you can’t get from being far away from your target users.

Ethnographic research evolved out of cultural anthropology. Some of you may remember Margaret Mead’s famous ethnography of the Samoans (some of you may also remember the Samoans’ famous joke on her, but more on that later). Mead lived with the Samoans to decipher how their culture affected the sexual maturation of girls. She wrote copious notes on her experiences, and later, when studying elsewhere in the South Pacific, took over 25,000 photographs.

Ethnographic research is first and foremost about observation. Ethnographers are not experimenters. They do not engineer or contrive situations to elicit reactions. They observe “natural” settings, that is, where people are going about their lives. Contrary to popular belief, ethnographers also do count things — quantitative data can serve to summarize a large number of observations (e.g., how many people on the subway are carrying a briefcase?).

Ethnography is NOT simply “in-person interviewing.” Now there is such a thing as “ethnographic interviews,” which melds ethnographic observation of natural settings with in-depth interviewing techniques. I myself have used ethnographic interviews on my dissertation and gleaned great insight.

But true ethnography means months of observation and in-depth analysis of all the “symbols” that your target users use. That means paying attention to their clothes, their manner of speech, their “argot” or local shared dialect, and even the accepted practices around social events like meals, meetings, and saying goodbye.

Ethnography does have a very clear limitation, which becomes clear when you learn about the Samoan joke on Margaret Mead. It was quite common to joke about sex in Samoan culture, so when Mead asked these young girls what they did at night, they jokingly told her they spent the night with boys. Mead later reported that Samoans matured sexually much more quickly than North Americans and had little of the same repressed sexuality. But this was not at all true and Mead had been duped.

Mead’s assumptions that Samoans were sexually more liberated than North Americans affected her research. Ethnographers who do not understand issues of gender and power are condemned to repeat these mistakes. An ethnographer interviewing workers must understand that when they tell her they “like having a laptop,” they have a need to portray themselves as “team players.”

An ethnographer must understand that when he interviews women in their kitchens, they are demonstrating their “proper” roles as women and may have a vested interest in portraying themselves as more “homey” than they really are.

Ethnographers are not objective. They are part of this thing we call society. As such, they have biases, just like everyone else. Good ethnographers understand that designing new laptops or new kitchens is about understanding the target user’s place in society as well.

Further reading:

http://onlineqda.hud.ac.uk/resources.php#Ethnography