Category Archives: Research Methods

Qualitative versus quantitative research, Part II

Thousands of people arrive at this blog wanting to know what is the difference between qualitative and quantitative research. Qualitative versus quantitative research is by far the most popular post on this blog. In that first post, I explained why sample size doesn’t matter in qualitative research. In this post, I explain why qualitative research is generally a better approach for design research.

Notice how the qualitative process is iterative with the going back and forth from data to sense-making or developing theory. It is flexible and can change direction easily.

Qualitative design process

Double click for a larger image

Double click for a larger image

And the quantitative design process is very linear, and does not include an iterative component:

Double click for larger image

If your design process involves an iterative prototyping phase, for example, then qualitative research is likely the best approach for you. Notice also that qualitative research necessarily involves the researcher putting herself in the shoes of the user. Quantitative research does NOT require the researcher to see through the eyes of the user.

Designers often want to empathize with their users. They want to understand their experiences and pain points. They want to know what their users are thinking. This is why qualitative research is often better suited to design research.

See also this embedded slideshow from my research design class. This should give you the basic differences between the two.


Research Design Course: Follow along on slideshare

I am currently teaching a Research Design and Qualitative Methods course at Ryerson University. This is a core course for an interdisciplinary group of students, from social work, to business, to psychology, to sociology to…well you get the picture.

I will be uploading slides from my lectures regularly. See them all at:

I have toyed with adding audio, but so far my students do not appear to be too interested. Are you? If so, let me know and I will add audio to my slide space.

Knowing your end-user: an anthropological primer

What do product designers need to know about their end-user? This post provides a broad-stroke overview of the kinds of questions you should answer before you design a new product, particularly new technology products.

The “value orientation model” of anthropology is a great starting point for product design. Your product has to fit within a person’s existing value system. Think about the automobile for example. Is your end-user an SUV type or a Smart Car type? Here’s how to narrow the focus.

  1. Human nature: Describe what the typical end-user believes about human nature (e.g., humans are generally good; humans are generally bad; humans are neither good nor bad). Hint: SUV drivers may think humans are generally bad, so we need to protect ourselves with BIG CARS.
  2. Time sense: Describe the typical end-user’s relationship to time (e.g., focus on the future; focus on the now; focus on the past). Smart Car drivers may think that the future matters, so they buy smaller more environmentally friendly cars.
  3. Person-Nature relationship: Describe the typical end-user’s orientation to nature (e.g., nature is to be dominated; nature is to be revered; nature is to be ignored). SUV drivers think nature should rule them. Just kidding.
  4. Social relations: Describe the typical end-user’s relationship to others (e.g., individualistic or “dog eat dog”; collective or: “we’re all in this together”). SUV drivers are definitely dog-eat-dog. Hence the BIG CAR.
  5. Space: Describe the typical end-user’s relationship to space (e.g., people control space; people live in harmony with space; space controls people). Smart Car drivers may believe that people should live in harmony with space, so they buy a smaller car, to park in urban settings, but also a car so they can conquer space and drive to the country for the weekend.

An additional set of questions around technology devices is also critical for technology designers:

  1. What is the typical end-user’s primary interactive device? Surprise! It may be a TV remote control!
  2. What other interactive devices does the typical end-user have?
  3. What is the primary frustration the typical end-user has with his or her current primary device?

Do you know the answers to these questions? If not, how will you know whether you’re designin for an SUV driver or a Smart Car driver? You can find out the basics to these questions through a few simple steps:

  1. Review any secondary value-based research, including omnibus surveys.
  2. Complete quick and dirty observations of your primary end-users.
  3. Survey a larger group of your primary end-users.
  4. Summarize and segment these findings to create value-based design personas
  5. Design a fabulous product!

The Myth of The “Average”

We bandy about the word “average” all the time. What exactly IS an average, and how does it help design research?

Use the average to quickly summarize something that is already a number: minutes, ages, heights, visits, etc. Don’t use the average to explain something that needs more detail. And keep in mind, the average gets “dragged up” or “dragged down” by extreme values. Sometimes it doesn’t tell you much of anything.

An example design research project might be about how people use their stoves in their kitchens. How can we use “the average” to help us design a new stove?

The average, in statistical language, is actually called “the mean,” which is a measure of “central tendency.” Researchers use central tendency to describe all their results quickly. Other measures of central tendency include the mode (the most common response) or the median (50% of responses are higher than this; 50% are lower).The mean describes the “typical” or average result.

But here’s the big myth: there is no such thing as “the average” in your data. If you ask 500 people to rate your new stove design on a scale of 1 to 10, and the average is 4, there is no guarantee that any single person actually said 4! In fact, the majority of responses could be higher than 7, but some 1s or 2s could “drag down the average.”

Worse, it makes no sense to use the “average” or “typical” in qualitative research. If you do interviews or observations, there is no way to calculate “the average.” So when you say, “the typical person has a four-element stove,” you’re actually doing a calculation. This may be actually quite false. What you may mean to say is “most people in our study have a four-element stove” (which is the mode).

Qualitative research does not accept the “typical.” It actually looks at each case individually and in enough detail to allow for exceptions or outliers. There is no “typical” case in qualitative research because you do not do calculations. You do not summarize your data in that reductionist way.

That said, how could you use “the average” in your kitchen stove study?  You can do a back-of-the-envelope calculation to summarize your data. The “average age” of your respondents, for example, will tell you about how old people are. The “average number of minutes spent cooking” will give you a snapshot of how long people spend in their kitchens. The “average purchase price of a stove” will also give you a quick snapshot. Using “the average” is to quickly summarize something that is already a number.

But the “average use of the stove”? That doesn’t make sense. Nor does the “typical grocery shopping process,” or the “average complaint of stove use.” These cannot be summarized in “the average.”

The promise (and failure) of

I loved it when I first saw it. invites users to look at a logo and type in the first thing that enters their minds. I found it fascinating — until I realized it’s yet another example of poor research perpetuating negative stereotypes of women.

Type in “Oprah” and see what happens. The top three most entered words? Fat. Black. Bitch. Yes, that’s right, Oprah, the maven of women’s media landscape is nothing more than a fat black bitch. How valid a representation of Oprah is this?

Oprah’s media universe is worth a fortune. She earned $260 million in 2007 and is worth $2.5 billion. Her daily talk show alone gets 7.3 million viewers (that’s compared to 2.9 million viewers for Grey’s Anatomy).

So I got to thinking. How is Brand Tags so wrong? So nasty? So racist? (Type in NBA or Citibank and you’ll see what I mean). Researchers are Harvard have shown how stereotypes work. We know that people rely on implicit stereotypes when they make snap judgments. This is the downside of Malcolm Gladwell’s Blink.

We live is a complex social world. We try to make sense out of it by looking for patterns. Theorists Berger and Luckman call these “typifications” or roles that we take for granted. Typifications help us because they allow us to know what to do in social situations without really thinking about it, or, as Berger and Luckman explain it, they alleviate us from making “all those decisions.”

All Brand Tags really does is tell us what those typifications are for the people who visit their site. Who is visiting their site? We don’t really know. The first rule of sampling is to ask yourself, are the people who participate systematically different from the people who don’t?

People who participate in Brand Tags are obviously Web savvy. Someone forwarded them a link and they filled it out. Perhaps they read business media because Brand Tags has gotten some press. They have the time to enter text. They are also anonymous.

Is this what you would consider a “representative sample”?

Brand Tags has promise (I myself have used it to gain insight about a few things). But it mostly has the worst of our stereotypes. Is that insight? Perhaps. But it’s not insight about Oprah — it tells us a lot about the people who are talking ABOUT Oprah.

Online Surveys 101


Below is a (very!) brief overview of online surveys. This slideshow, via slideshare, is intended for people in the Web design industry. IAs, designers, media planners, strategists, usability researchers, and producers will learn if they should, in fact, do a survey.

What is design anthropology?

Dori Tunstall has written a fantastic post that details how a simple card sort can become a deeper exercise in analysis. Dr. Tunstall is a PhD anthropologist and the University of Illinois at Chicago. She explains how anthropology takes design research to a deeper level in card sorting, a common technique for information architecture:

“In addition to the information architecture, I delivered statements about the continued meaning of gender classifications. In the course of conducting the card sort, I learned that men and women continued to classify domestic products based on stereotypical gendered spaces of male equals outside/garage, and female equals inside.”

Dr. Tunstall clearly lays out some of the deeper implications of design anthropology later in the post. This is a must read for anyone looking to deepen their design or research practice.