The Myth of The “Average”

We bandy about the word “average” all the time. What exactly IS an average, and how does it help design research?

Use the average to quickly summarize something that is already a number: minutes, ages, heights, visits, etc. Don’t use the average to explain something that needs more detail. And keep in mind, the average gets “dragged up” or “dragged down” by extreme values. Sometimes it doesn’t tell you much of anything.

An example design research project might be about how people use their stoves in their kitchens. How can we use “the average” to help us design a new stove?

The average, in statistical language, is actually called “the mean,” which is a measure of “central tendency.” Researchers use central tendency to describe all their results quickly. Other measures of central tendency include the mode (the most common response) or the median (50% of responses are higher than this; 50% are lower).The mean describes the “typical” or average result.

But here’s the big myth: there is no such thing as “the average” in your data. If you ask 500 people to rate your new stove design on a scale of 1 to 10, and the average is 4, there is no guarantee that any single person actually said 4! In fact, the majority of responses could be higher than 7, but some 1s or 2s could “drag down the average.”

Worse, it makes no sense to use the “average” or “typical” in qualitative research. If you do interviews or observations, there is no way to calculate “the average.” So when you say, “the typical person has a four-element stove,” you’re actually doing a calculation. This may be actually quite false. What you may mean to say is “most people in our study have a four-element stove” (which is the mode).

Qualitative research does not accept the “typical.” It actually looks at each case individually and in enough detail to allow for exceptions or outliers. There is no “typical” case in qualitative research because you do not do calculations. You do not summarize your data in that reductionist way.

That said, how could you use “the average” in your kitchen stove study?  You can do a back-of-the-envelope calculation to summarize your data. The “average age” of your respondents, for example, will tell you about how old people are. The “average number of minutes spent cooking” will give you a snapshot of how long people spend in their kitchens. The “average purchase price of a stove” will also give you a quick snapshot. Using “the average” is to quickly summarize something that is already a number.

But the “average use of the stove”? That doesn’t make sense. Nor does the “typical grocery shopping process,” or the “average complaint of stove use.” These cannot be summarized in “the average.”

The promise (and failure) of Brandtags.net

I loved it when I first saw it. Brandtags.net invites users to look at a logo and type in the first thing that enters their minds. I found it fascinating — until I realized it’s yet another example of poor research perpetuating negative stereotypes of women.

Type in “Oprah” and see what happens. The top three most entered words? Fat. Black. Bitch. Yes, that’s right, Oprah, the maven of women’s media landscape is nothing more than a fat black bitch. How valid a representation of Oprah is this?

Oprah’s media universe is worth a fortune. She earned $260 million in 2007 and is worth $2.5 billion. Her daily talk show alone gets 7.3 million viewers (that’s compared to 2.9 million viewers for Grey’s Anatomy).

So I got to thinking. How is Brand Tags so wrong? So nasty? So racist? (Type in NBA or Citibank and you’ll see what I mean). Researchers are Harvard have shown how stereotypes work. We know that people rely on implicit stereotypes when they make snap judgments. This is the downside of Malcolm Gladwell’s Blink.

We live is a complex social world. We try to make sense out of it by looking for patterns. Theorists Berger and Luckman call these “typifications” or roles that we take for granted. Typifications help us because they allow us to know what to do in social situations without really thinking about it, or, as Berger and Luckman explain it, they alleviate us from making “all those decisions.”

All Brand Tags really does is tell us what those typifications are for the people who visit their site. Who is visiting their site? We don’t really know. The first rule of sampling is to ask yourself, are the people who participate systematically different from the people who don’t?

People who participate in Brand Tags are obviously Web savvy. Someone forwarded them a link and they filled it out. Perhaps they read business media because Brand Tags has gotten some press. They have the time to enter text. They are also anonymous.

Is this what you would consider a “representative sample”?

Brand Tags has promise (I myself have used it to gain insight about a few things). But it mostly has the worst of our stereotypes. Is that insight? Perhaps. But it’s not insight about Oprah — it tells us a lot about the people who are talking ABOUT Oprah.

What is design anthropology?

Dori Tunstall has written a fantastic post that details how a simple card sort can become a deeper exercise in analysis. Dr. Tunstall is a PhD anthropologist and the University of Illinois at Chicago. She explains how anthropology takes design research to a deeper level in card sorting, a common technique for information architecture:

“In addition to the information architecture, I delivered statements about the continued meaning of gender classifications. In the course of conducting the card sort, I learned that men and women continued to classify domestic products based on stereotypical gendered spaces of male equals outside/garage, and female equals inside.”

Dr. Tunstall clearly lays out some of the deeper implications of design anthropology later in the post. This is a must read for anyone looking to deepen their design or research practice.

Getting meaningful insights from qualitative research

The output from qualitative research is often overwhelming. Unlike quant research, qual findings are often messy and hard to decipher. Here are some techniques to manage the voluminous data of qualitative studies.

1. Start with clear research questions: in an earlier post I explained how to set up a design research project, step by step. One of the most important steps is to create a clear and answerable research question. This seems like an obvious point, but often it isn’t. Qualitative research often appears to be “just talking to people,” which gives us all the mistaken impression that it is entirely unstructured. It isn’t. Take the time to define research questions.
2. Summarize frequently: Let’s say you’ve chosen to do in-depth interviews. After each interview, take 20 minutes to write out a brief summary of what you remember being the most important points of the interview (note that this is not a substitute to taking notes during the interview). These notes are the first step toward analysis. You are reducing “clutter” and irrelevant information. You are also exploring connections with previous interviews.
3. Reduce, reduce, reduce: You will always have more data (e.g., videos, photos, transcriptions) than you can use. Be ruthless by reducing what’s important. Edit down your videos to only the clips that are most important (keep the raw data for another time). Reduce your transcriptions down to select quotes that speak to your research question (and again, save the entire transcription for another time). The goal is to have a workable set of artifacts.
4. Visualize the results: Many qualitative researchers make use of summary tables and diagrams to further summarize results. My favourite visualization method is the mental model, which can convey a huge amount of information in a synthetic way, quickly. Other tools include mind maps and even the simple bulleted list.
5. Hunt for connections: There is no science to this process. It is iterative and intuitive. But there are approaches you can use to find connections. I frequently use the “open sort” technique, with nothing more than a blank wall and post-it notes. Scribble themes onto post-it notes. Sort them into categories. Name the categories. Collapse as many categories as you can until you only have 4 or 5 “buckets” that explain your findings. If you’re researching children’s commutes to school, for example, you may have a category called “independence” which would talk about kids’ desires to be grown up, to have their own transportation method, and the knowledge to get to school. They are related only through the higher-order notion of “independence” and not the lower-order ideas of “transportation” or “age.”
6. Ask “So what?” often: When I was in journalism school, I had a professor who tirelessly quizzed us with his version of so what: “What does it mean to metro?” he would demand, meaning, why should the people of this city care? Why should your design team care about these results? What does it mean for their process? Why should the users of this product care about your results? How might it make their lives easier or more pleasant? And of course, why should bean counters of all sorts care? How much money will it actually save?

These general guidelines will help you in your journey to deciphering meaning. But no qualitative project can be save from poor research design. Make sure you’re using the right approach.

Why do ethnography?

Ethnographic research is mandatory for all design. Why? Because the role of design is to improve people’s lives. This you cannot do unless you know what people’s lives really are like — and not what charts and graphs and tables are like. Why do ethnography? Here are some clear reasons.

1. People don’t know what they actually want: Would anybody ask for a translucent mirror? That’s what they now get at Prada. The dressing room’s at Prada ’s flagship New York store allow you to do something you normally do — but better. Shoppers can first view outfits on themselves, then can invite their friend’s to view their outfit — but turning the mirror into a window. Instead of coming out of the dressing room, leaving your handbag behind, you can instead simply click a button with your foot, and show your new outfit to a friend.This innovation did not come from asking people what they want, but by thinking about the process of buying and trying on clothes.
   
2. Context matters: Most people who design mobile phones don’t think that electricity has anything to do with their product. But they are wrong. Researchers in Africa have learned that when the power goes out, people can’t charge their mobile phones. The solution? Various forms solar and wind-powered chargers.Designers must know where their product will be used. Deep insight into that context can only come from knowing the context.
3. People lie: A well known example of urban ethnography finds a contradiction. People say they want a quiet space to eat lunch, but when you watch lunchtime routines in urban spaces, people do anything but seek out quiet spaces. Now are they lying to be naughty? To be elusive? No, they lie because they believe the “normative” or “should do” practice of eating lunch is a quiet experience.Actual experience plays out much differently.
4. Designers design symbols — which can’t be understood through numbers: The reason why people love quantitative research so much is because it is short and easy to communicate. You know the “average” household income, instead of having to think of all the possible household incomes. You know how many people answered “yes” to a pre-defined question.Designers are designing or adapting symbols. They cannot do so without knowing what they represent. But you can’t summarize symbols. Symbols *are* summaries already — and not numerical summaries.A national flag conveys many ideas for people within that nation state (and many more for those outside it). Likewise, a kitchen stove is a symbol that conveys much about the household, gender relations, and family life. This cannot be conveyed in the “average” number of kitchen stoves.

Many designers will take numbers or focus group research or even usability test results and design their products. They may even improve people’s lives that way. But short observational research provides “thick description” that all designers need.

The Brand as A Self: Web Design as Impression Management

Brands have few opportunities to come alive, and the Web is one of those opportunities. Make sure the brand gives off the right impression. Researchers have found that a company’s Web site particularly shapes how a person views that company’s innovation and concern for its customers. In other words, the Web site is even more important in “giving off” the right impression.

Brands introduce themselves to people much in the same way that people introduce themselves to people. And just like for humans, brands often “give off” more information than they explicitly mean to provide. This is especially true for Web sites: the brand online is the same as a “self,” and must manage its impression just as people do.

We have all experienced this: you meet someone and develop an immediate sense of what they’re about. You have figured out that this person works in, say, finance, and he has money and children and likes nautical sports. You also find him curt, arrogant and a bit full of himself. Is it something he said specifically? No, not specifically. He did snap at the waitress. And he did mention something about a regatta. He also casually tossed his credit card down when the bill came, rudely brushing aside protestations from the most senior person at the table.

One of my favourite theorists, Erving Goffman, tells us there is an impression you GIVE, and then there is the impression you GIVE OFF. “Selves,” as Goffman puts it, engage in impression management using subtle symbolic signals.

Designers often implicitly think of their particular product — whether it be a kitchen product or a print ad — as something that “gives off” an impression. But this is much more important for immersive experiences like Web sites. A company’s Web site in particular is an immersive experience that gives off countless symbolic cues.

Some observers call this phenomenon “cross channel synchronicity,” or simply just “user experience.” The Web site is key to “giving off” the right impression for a company and its brand because it is the living embodiment of that company.

How should graphic and interaction designers create their products? Keep in mind the following:

• The brand is a “self” on the Web. This is a great opportunity but designers also run the risk of “giving off” the wrong impression immediately through interactions that suggest a stand-offish, arrogant, or selfish brand.
• Brand-critical interactions must be done right: I have had many clients who appear unconcerned about appear small interaction problems of their Web site. But if these interactions revolve around mission-critical symbols of your business, make sure they’re done right. If your brand identity if “fun,” ensure that interactions are full of fun, not hard work. If your brand identity is “trustworthy,” over-communicate that message in interactions.
• Provide the expected “props”: In an earlier post, I showed how individuals use symbolic cues, or “props” to manage impressions. Doctors use stethoscopes, for example, despite the fact that fewer than 40% of them know how to use them properly, mostly because patients EXPECT them to carry them. Web site designers should remember what users expect in terms of “props.” Does your brand really need AJAX? Are visitors surprised to find their is no flash element? Are visitors expecting form fields to have in-line editing?

What Designers Can Learn From Facebook’s Beacon: the collision of “fronts”

The blogosphere (and even the regular old newspaper-sphere) is alight with stories of Facebook’s online advertising flop, Beacon. What can designers learn from this flop? It’s not about privacy; it’s about the presentation of self. People have different “selves” for different places — virtual or otherwise — and designs must be consistent with these variety of selves.

Boing Boing’s Cory Doctorow posted an interesting story on InformationWeek that predicted the decline of Facebook because of its own success. He predicts that the more people that are one Facebook, the more confusing it is. Your “creepy coworkers,” your boss, and your friends you met at Burning Man are all in the same “place,” making it confusing, embarrassing and difficult for everyone.

What Doctorow is really describing is sociologist Erving Goffman’s notion of “the front.” Using the theatre as a metaphor Goffman argued that we actually “perform” multiple selves. Each place we go has a “front” that we learn to incorporate. A front has a wardrobe, a setting, a decor, make-up, a script and stage direction. We have a “front stage self” that we perform for everyone to see, a “back stage self” for only our closest intimates to see, and a “core self,” which is deeply private.

A doctor, for example, has a front that includes an office, a lab coat, a stethoscope and medical jargon. This is her “front stage” self. But when she’s talking to her best friend, she may use a “back stage self,” being less formal, not wearing a lab coat, or using less formal language. Her “core” self is secretly wishing she were a full-time marathoner, but she tells no one that.

Facebook’s Beacon didn’t work because it forces people to use multiple fronts AT THE SAME TIME. If I tag a recipe from Epicurious.com, but I broadcast that fact to friends that perceive me to be a party girl, I have a collision of fronts. If my boss demands to be my friend, I have a collision of fronts. If I rent The Notebook on Netflix, and my friends think I am a Goth, I have a collision of fronts.

Facebook’s Beacon forces its users to combine multiple selves. Goffman considers the collision of fronts to be a source of embarrassment or shame. Take, for example, the hilarious “Meeting in a Swimming Pool” gag on Just for Laughs. Swimmers have their swimming front (including a bathing suit, casual demeanour) and forced into a meeting, with its serious demeanour and fully clothed attendants. This is embarrassing.

Facebook has done the same thing by forcing its users to expose their selves to different fronts simultaneously. It is embarrassing, even shameful.

What Designers Can Learn From Facebook’s Beacon

• Discover your users’ fronts: If you are designing a product or a virtual place, ask your potential users what they consider the character of this “place” to be. Is is a formal place? Is it a casual atmosphere? What kinds of “props” are expected here? What would be an embarrassing topic of conversation or incident?
• Design using the theatre metaphor: Make the product consistent with that place, as if you were writing a play. Ensure that what you design is part of a script that users understand or expect.
• Pay attention to embarrassment: If your users mention shame or embarrassment in any way, gently press them about it. Discover the character of the “collision of fronts” that is the source of that embarrassment, and, above all, avoid forcing users to feel embarrassment.

Update: The New York Times is reporting that Facebook’s lawyers have not succeeded in having documents about its founder Zuckerman removed from an online magazine. These documents are “embarrassing.”

Update (12/19/07): Mashable is reporting that FB is now allowing people to “group” their friends, but they haven’t quite mastered the collision of fronts problem.

When to do qualitative and qualitative research

In a previous post, I talked about what designers need to know about economic class. How did we learn that economic class can be “seen” in designs? How did we learn that “refined” taste is “upper” class?

In general, use qualitative research at the beginning of a design process to uncover innovations. Use quantitative research at the end of a design process to measure improvement.

It started with qualitative research, and became “refined” (no pun intended) with quantitative research. French sociology Pierre Bourdieu followed a typical arc to the narrative research by first investigating economic class in an open-ended fashion. Once he established what he thought was going on, he tested these ideas with large surveys.

If you know little about the topic, start with the qualitative. This means ethnographic observation and in-depth interviewing. Open ended questions are best. At this stage, you’re trying to find the lay of the land. If you’re designing a new car stereo for example, you may wish to start by watching people use their existing car stereos. Maybe drive around with them and ask them questions about what they like.

Once you’ve learned the basics of car stereo requirements, user needs and pain points, it’s time to test your assumptions. This is where the quantitative comes in. Close-ended questions are best here, including multiple choice, yes/no, or simply number of “successes.” Let’s say you’ve learned through your observations that people don’t like how their stereos require programming their radio stations. It’s too much bother, they told you. You think pre-programmed stations might be a good design improvement, so you create a new stereo with pre-programmed stations.

Did it work? Ask your stereo users how they like the new system after they have bought their new car. But the question is, compared to what? This is where quantitative research gets tricky. You can compare the new stereos on select models (58% of users of the new model are very satisfied, while only 32% of users of the old model are). Or you can compare before and after the improvement — the so-called “pre-and post test.” That requires time, foresight, and — you guessed it — budget.

Below is a diagram that summarizes the research “funnel” from exploration to validation.

Discourse analysis and design: reading “texts” for design purposes

Designers are already discourse analysts, they just don’t know it. These designers can produce more innovative ideas by adopting a more systematic approach to their intuitive discourse analysis.

Discourse analysis the practice of deciphering the meaning of “texts.” Anything can be a “text.” Television commercials, Us Weekly, a trial transcript — these are all “texts.” Famous discourse analyses include Michel Foucault’s analysis mental illness, in which he traces how we collectively think about mental illness through “texts” of it, such as “patient charts,” or the Diagnostic and Statistical Manual.
Designers intuitively analyze “texts” all the time, especially designers who work in advertising. They obsessively collect imagery and copy they find interesting. They innovate on this copy or imagery by re-tooling some of the subtle messages in them.

How to systematize discourse analysis “lite” for designers:

1. Collect more than one genre of “texts”: instead of a single medium, try collecting several media of the same theme. If you’re designing a new toy, for example, gather a TV commercial, a print ad, and a fan’s tribute Web site. These differing “texts” may tell you what is missing in toys, or what toys are unintentionally doing to the parents who buy them.
2. Look for the “silences” in texts: If you’re designing an online advertising campaign, compare texts on a single theme and ask yourself, “What is not being said?” For example, if you’re targeting women with small children, maybe you’ll find that these women are never painted as actually having personal preferences only “mother preferences.” This is a silence that you can speak to.
3. The obvious meaning is the tip of the iceberg: If you want to know what an object means in culture, you must look more deeply than the obvious. Most designers understand this intuitively, but sometimes you must make a concerted effort. When you see the famous “Diamonds are forever” ads by De Beers, the obvious meaning is one of romance, but what is the subtle meaning? Romance is fleeting but diamonds? Diamonds are forever. The ad’s brilliance lies in its ability to leverage the symbolism of the world’s hardest substance (the diamond) with the most coveted but ephemeral experience (romantic love).

The other day I was tutoring an adult learner (a highly educated one) about discourse analysis. She complained to me that she well understood quantitative methods, variables, and counting. But she saw discourse analysis as “mumbo jumbo.”

On the surface, discourse analysis looks like mumbo jumbo. But in practice, it is a tool to see both culture and the “reality” we have constructed.

Qualitative versus quantitative research

Many designers are self-taught, intuitive consumers of research who can translate insights into great designs. But few are trained in the arcane art of research itself. For that reason, many designers don’t know the finer differences between qual and quant research and end up using their respective results inappropriately.

Quantitative research is based on the assumption that random events are predictable, and if you compare your results to pure random results, you can discern distinctive, meaningful patterns about the social world.

Random events are relatively MORE predictable if you have more of them. Imagine if you flipped a coin 20 times. How many heads would you get? Now if you flipped it 20,000 times? You’re more likely to get an even 50/50 split — which is what most people would predict. If you got a 65/35 split with 2o flips, okay, could happen. But with 20,000 flips? No way. Something else is going on.

Translate that to design research by looking at gender, for example. Let’s say you have 20 people, 10 men and 10 women. 65% of the women choose one design, while only 35% of the men do. Is this a meaningful pattern? Impossible to say — you only have 20 people. Now if you had 200 people (100 men and 100 women) and 65 of the women chose one design, chances are you have a meaningful pattern.

This is why sample size matters in quantitative research. But, little known fact, sample size is COMPLETELY IRRELEVANT in qualitative research. Why?

Qualitative research assumes that people have meaningful experiences that can be interpreted. Notice how there’s nothing in there about “prediction” or “randomness.” People have experiences. Researchers discern what these experiences signify. That’s it. Sample size is not only irrelevant, it actually gets in the way of important insight.

Consider the case study, for example. Few people would say case studies are useless. We can learn a great deal about a single design case, where it went wrong and where it went right. The problem comes when you try to predict future events based on this single event.

If you abandon the need for prediction, then sample size never matters. You can always derive insight about design problems from even a single case. Designers that attempt to predict “success” of a single design change, for example, should test that change, repeatedly, with a probability sample.