Tag Archives: quantitative research

The essence of qualitative research: “verstehen”

“But how many people did you talk to?” If you’ve ever done qualitative research, you’ve heard that question at least once. And the first time? You were flummoxed. In 3 short minutes, you can be assured that will never happen again.

Folks, qualitative research does not worry about numbers of people; it worries about deep understanding. Weber called this “verstehen.” (Come to think of it, most German people call it that too. Coincidence?). Geertz called it “thick description.” It’s about knowing — really knowing — the phenomenon you’re researching. You’ve lived, breathed, and slept this thing, this social occurrence, this…this…part of everyday life. You know it inside and out.

Courtesy of daniel_blue on Flickr

Courtesy of daniel_blue on Flickr

You know when it’s typical, when it’s unusual, what kinds of people  do this thing, and how. You know why someone would never do this thing, and when they would but just lie about it. In short, you’ve transcended merely noticing this phenomenon. Now, you’re ready to give a 1-hour lecture on it, complete with illustrative examples.

Now if that thing is, say, kitchen use, then stand back! You’re not an Iron Chef, you are a Platinum Chef! You have spent hours inside kitchens of all shapes and sizes. You know how people love them, how they hate them, when they’re ashamed of them and when (very rarely) they destroy them. You can tell casual observers it is “simplistic” to think of how many people have gas stoves. No, you tell them, it’s not about how many people, it’s about WHY they have gas stoves! It’s about what happens when you finally buy a gas stove! It’s about….so much more than how many.

Welcome to the world of verstehen. When you have verstehen, you can perhaps count how many people have gas stoves. Sure, you could determine that more men than women have them. Maybe you could find out that more of them were built between 1970 and 80 than 1990 and 2000. But what good is that number? What does it even mean?

When you’re designing, you must know what the gas stove means. You must know what it means to transform your kitchen into one that can and should host a gas stove. You must know why a person would be “ashamed” to have a gas stove (are they ashamed of their new wealth? do they come from a long line of safety-conscious firefighters?). You must know more than “how many.”

So the next time someone asks you, “how many people did you talk to?”, you can answer them with an hour-long treatise about why that doesn’t matter. You can tell them you are going to blow them away with the thick description of what this thing means to people. You are going to tell them you know more about this thing than anyone who ever lived, and then, dammit, you’re gonna design something so fantastic, so amazing that they too will be screaming in German. You have verstehen!

See my discussion about sampling methods in qual and quant research for more insight into the reasons why “how many” is irrelevant in qualitative research.


Designers are from Venus, Six Sigmas are from Mars

DT has a great post over at Design Sojourn that discusses Six Sigma methodology and how it relates to design. He cites Tim Brown at IDEO who argues that Six Sigma is essentially Newtonian, while design thinking is quantum. In his own design work, DT expressed doubts about using Six Sigma:

After studying the Six Sigma process, I point blank said: “There was no way any of my designers are going to be judged on the quality and success of a design based on how many sketches or iterations we did before we deliver it.”

Both Brown and DT cite Sara Beckman, who recently discussed the topic in the New York Times. Beckman reviews how Six Sigma focuses on incremental improvements, while design and design thinking focuses on big changes. For those of you who aren’t familiar with Six Sigma, it’s a method pioneered by Motorola, which aims to reduce the number of errors to 3 in one million. The “six sigma” refers to six standard deviations. The number of errors should be at the extreme end of the normal curve, or between + or – 3 standard deviations, represented by the Greek symbol sigma.

I argue that design is more complementary to the “interpretivist” paradigm of qualitative research while Six Sigma is positivist. Interpretivists don’t believe the world is a static place. They see reality as being continuously created by you, me and other social actors. There is no such thing as “The Truth” in interpretivist approaches, just different versions of the truth. Typical methods of interpretivists are ethnography, in-depth interviewing and discourse analysis. Positivist research, on the other hand, assumes that reality is static. Positivists believe that “The Truth,” is out there to be discovered. Typical methods would include quantitative surveys.

Designers should focus on interpretivist methods, therefore. They should uncover different versions of the truth using observation and interviewing, as well as deep reflection on symbols and their meanings. Surveys and other quantitative methods are more Six Sigma in that they can measure improvement over time. Designers ought to consider measuring improvement, but starting with qualitative approaches is best.

Customers more satisfied when served by white males

In an interesting study, researchers at UBC have found that customers express higher satisfaction when they’re served by white men than by women or people of colour — even when their behaviour is exactly the same. Marketing professor Karl Aquino expressed surprise at the findings, as he told The Globe and Mail

“We had thought there would be some bias going on in the sense of people who were males or whites would be rated more positively,” Mr. Aquino said

“But we didn’t anticipate that for performing the same behaviours, the women and minorities would actually be rated lower,” he said of the study to be published in the Academy of Management Journal.

This study should not be surprising at all.

What this study demonstrates is what Raymond Breton calls the “symbolic order”; we unconsciously place white men at the top of our social hierarchy. We do this in multiple ways, including placing art, culture and ideas at the top of an invisible ladder. Public Enemy sums it up nicely in “Fight the Power”:

None of my heroes don’t appear on no stamp

We know that people have largely unconscious reactions of sexism and racism, oftentimes without even realizing it. It is likely that these unconscious ideas bleed into marketing research easily, especially when such studies are quantitative in nature, and therefore lack the thick description or deep probing offered by qualitative approaches.

This finding has wide-reaching implications. First, when companies use customer satisfaction surveys, they must be aware of the inherent inaccuracy of these surveys. You may believe you’re accurately measuring actual satisfaction, but this study shows that frequently, we don’t measure any such thing. Secondly, such surveys are often used to award bonuses or even job security. As we know in academia, student evaluations are frequently what stands between a scholar and a full-time position. If we know that customer satisfaction is driven by factors other than actual performance, then we are likely to be unwittingly simply rewarding membership in a dominant group.

Read the entire story on The Globe. It’s worth a think.

The evolution of qualitative sociology

The blog Economic Sociology has a great post on the “evolution” of qualitative sociology. They note, quite rightly, that the notion of “evolution” is implicit in much of social science, even if it has no bearing on the subject matter at hand. Many sociologists place quantitative research “on top” of the research “evolutionary ladder,” even when there is no such thing as a ladder when it comes to good research design. Interestingly, the fathers of sociology themselves would be on the “lower rung” of that methodological ladder:

The works of Marx and Weber, like virtually all the classic literature in the field, were based on qualitative, historical methodology (Durkheim’s quantitative study Suicide being a notable exception).

This post just reinforces to me why the design process is so important for social scientists. One must design a research project to solve contextual problems, just as one designs, say, a chair. You cannot “solve” questions of why or how by using quantitative methods. It is simply impossible.

Are why and how somehow “less than” questions than “how many” or “how fast”? I don’t think so. Indeed Economic Sociology points out that even Darwin was not concerned with “how many” but more with “how,” and few accuse him of being “unscientific.”

Mind the gap: qualitative insights and strategy

It’s very common to turn to numbers first when strategizing about new products, policies, or social movements. But nuanced, sideways or “integrative” thinking often requires more than just numbers. This is where qualitative research can help you.

Most people are trained to think of “research” as numbers and “hard facts.” That approach will lead to very specific, numerical questions when crafting new strategies. What are the most popular products consumers want? What are the top five frustrations with our current policy? What are the top Web sites that progressive people visit?

But imagine there was no such thing as numerical “evidence.” Imagine instead that you were trying to figure out how to innovate without the benefit of any kind of counting. What kinds of things would you consider to be insight?

Why do consumers get frustrated with their telecommunications service providers? How and in what ways do citizens react to our policy on childcare? What kinds of digital tools do progressive people use in everyday life?

These second sets of questions are far more likely to yield what qualitative researchers call “thick description.” Thick description fills the gaps between numbers. If I told you Superbowl 36 ended with a score of 20-17, you’d miss all the detail and the drama of the late-in-the-game push by the Rams, and the final Patriot field goal that ultimately won the game. Thick description tells you the entire story, not just the numerical summary.

If policymakers know that 49% of parents are frustrated with no childcare policy, that doesn’t begin to explain a day in the life of a working parent. Spend a day with a working parent and a sick child, and you will begin to understand all the detail and the drama of childcare.

If you spend time with person who is interested in progressive causes, you may learn that they spend more time using their mobile phone than their computer. Or perhaps you learn that for them, computers = work. That may lead you to think that mobile campaigns are better than Web-based campaigns.

Qualitative research intended to fill the gaps that numerical data inherently possess. If you rely too heavily on numerical data, you miss a great deal of nuance that could ultimately result in true innovation.

Qualitative versus quantitative research, Part II

Thousands of people arrive at this blog wanting to know what is the difference between qualitative and quantitative research. Qualitative versus quantitative research is by far the most popular post on this blog. In that first post, I explained why sample size doesn’t matter in qualitative research. In this post, I explain why qualitative research is generally a better approach for design research.

Notice how the qualitative process is iterative with the going back and forth from data to sense-making or developing theory. It is flexible and can change direction easily.

Qualitative design process

Double click for a larger image

Double click for a larger image

And the quantitative design process is very linear, and does not include an iterative component:

Double click for larger image

If your design process involves an iterative prototyping phase, for example, then qualitative research is likely the best approach for you. Notice also that qualitative research necessarily involves the researcher putting herself in the shoes of the user. Quantitative research does NOT require the researcher to see through the eyes of the user.

Designers often want to empathize with their users. They want to understand their experiences and pain points. They want to know what their users are thinking. This is why qualitative research is often better suited to design research.

See also this embedded slideshow from my research design class. This should give you the basic differences between the two.

The Myth of The “Average”

We bandy about the word “average” all the time. What exactly IS an average, and how does it help design research?

Use the average to quickly summarize something that is already a number: minutes, ages, heights, visits, etc. Don’t use the average to explain something that needs more detail. And keep in mind, the average gets “dragged up” or “dragged down” by extreme values. Sometimes it doesn’t tell you much of anything.

An example design research project might be about how people use their stoves in their kitchens. How can we use “the average” to help us design a new stove?

The average, in statistical language, is actually called “the mean,” which is a measure of “central tendency.” Researchers use central tendency to describe all their results quickly. Other measures of central tendency include the mode (the most common response) or the median (50% of responses are higher than this; 50% are lower).The mean describes the “typical” or average result.

But here’s the big myth: there is no such thing as “the average” in your data. If you ask 500 people to rate your new stove design on a scale of 1 to 10, and the average is 4, there is no guarantee that any single person actually said 4! In fact, the majority of responses could be higher than 7, but some 1s or 2s could “drag down the average.”

Worse, it makes no sense to use the “average” or “typical” in qualitative research. If you do interviews or observations, there is no way to calculate “the average.” So when you say, “the typical person has a four-element stove,” you’re actually doing a calculation. This may be actually quite false. What you may mean to say is “most people in our study have a four-element stove” (which is the mode).

Qualitative research does not accept the “typical.” It actually looks at each case individually and in enough detail to allow for exceptions or outliers. There is no “typical” case in qualitative research because you do not do calculations. You do not summarize your data in that reductionist way.

That said, how could you use “the average” in your kitchen stove study?  You can do a back-of-the-envelope calculation to summarize your data. The “average age” of your respondents, for example, will tell you about how old people are. The “average number of minutes spent cooking” will give you a snapshot of how long people spend in their kitchens. The “average purchase price of a stove” will also give you a quick snapshot. Using “the average” is to quickly summarize something that is already a number.

But the “average use of the stove”? That doesn’t make sense. Nor does the “typical grocery shopping process,” or the “average complaint of stove use.” These cannot be summarized in “the average.”