Those of you out there who’ve tried it know: recruiting research participants is HARD. Here are a few insights from the research to help you with better recuitment.
Personalized contact with respondents, followed by pre-contact and aggressive follow-up phone calls *: Don’t count on a form letter, email or random tweet to do the job. Capitalize on your personal relationship with that person. If you don’t have a personal relationship, ensure that you use the person’s name, and for God’s sake, spell it correctly!
Once you’ve made initial contact, you are not done. Not by a long shot. Make sure you speak to the person (you can do this through IM or email if you’d like) to give them more information. They’re now interested. Don’t stop! One more step!
Follow up 1 week after initial contact. Assuage any fears they may have. Answer any questions honestly. And above all, be available for more information.
External researchers with social capital are best**: University-based researchers have been shown to have the best participation rates, but you don’t have to be a professor. Researcher Sister Marie Augusta Neal of Emmanuel College achieved a near perfect response rate because of her close ties to the respondents and their communities. The lesson here is, if you hire a consultant, make sure they’re trusted. Even better if they personally know the people to be recruited.
Monetary incentives have no effect, unless money is offered “no strings attached”***: Little known fact: the best way to use a monetary incentive is to offer it, up front, with absolutely no strings attached. The “free” money makes people feel more indebted socially. Evidence of this effect can be found in the book Freakonomics. Researchers found that daycare centres that levied late penalties on tardy parents actually had moreof a late-pickup problem than those that levied no fine. Why? Because the parents reduced their relationship to the daycare as a mere transaction. Use the “gift economy” approach and ensure a feeling of indebtedness. My personal favourite is a coupon for a single iTunes song at $.99. It is cheap but appears to have great value. Offer it, up front, and then ask for participation
* Cook, C., F. Heath, and R. Thompson. 2000. “A Meta-analysis of Response Rates in Web or Internet-based Surveys.” Educational and Psychological Measurement 60:821-836.
** Rogelberg, S., A. Luong, M. Sederburg, and D. Cristol. 2000. “Employee Attitude Surveys: Examining the Attitudes of Noncompliant Employees.” Journal of Applied Psychology 85:284-293.
***Hager, M., S. Wilson, T. Pollak, and P. Rooney. 2003. “Response Rates for Mail Surveys of Nonprofit Organizations: A Review and Empirical Test.” Nonprofit and Voluntary Sector Quarterly 32:252-267. Singer, E. (2006) Introduction: Nonresponse Bias in Household Surveys. Public Opinion Quarterly, 70, 637-645
Why does sample size not matter in qualitative research? Because of the assumptions that qualitative researchers make, namely, that the social world is not predictable. Qualitative researchers believe that people are not like molecules or other objects; people’s actions are not predictable.
But quantitative researchers DO believe that social activity IS predictable. So when they compare their observations of social activity to what would happen in purely random results, the difference says something. Let’s say you were to research people’s preferences for a particular interactive feature. Say you’re wondering if young people will like a radio button more than older people. First, you model what results you’d expect if you’d just flipped a coin. Then you use a probability (random) sample, and compare those results to purely random results. Is there a difference?
If there is a difference between them, you can infer that indeed, something other than chance (in this case, age) affect people’s preferences.
Qualitative researchers don’t agree that such things can be reliably predicted. That’s why they don’t bother with expensive and involved random sampling. See all these details below from my research design course.
Thousands of people arrive at this blog wanting to know what is the difference between qualitative and quantitative research. Qualitative versus quantitative research is by far the most popular post on this blog. In that first post, I explained why sample size doesn’t matter in qualitative research. In this post, I explain why qualitative research is generally a better approach for design research.
Notice how the qualitative process is iterative with the going back and forth from data to sense-making or developing theory. It is flexible and can change direction easily.
Qualitative design process
Double click for a larger image
And the quantitative design process is very linear, and does not include an iterative component:
Double click for larger image
If your design process involves an iterative prototyping phase, for example, then qualitative research is likely the best approach for you. Notice also that qualitative research necessarily involves the researcher putting herself in the shoes of the user. Quantitative research does NOT require the researcher to see through the eyes of the user.
Designers often want to empathize with their users. They want to understand their experiences and pain points. They want to know what their users are thinking. This is why qualitative research is often better suited to design research.
See also this embedded slideshow from my research design class. This should give you the basic differences between the two.
I am currently teaching a Research Design and Qualitative Methods course at Ryerson University. This is a core course for an interdisciplinary group of students, from social work, to business, to psychology, to sociology to…well you get the picture.
I will be uploading slides from my lectures regularly. See them all at:
I have toyed with adding audio, but so far my students do not appear to be too interested. Are you? If so, let me know and I will add audio to my slide space.
What do product designers need to know about their end-user? This post provides a broad-stroke overview of the kinds of questions you should answer before you design a new product, particularly new technology products.
The “value orientation model” of anthropology is a great starting point for product design. Your product has to fit within a person’s existing value system. Think about the automobile for example. Is your end-user an SUV type or a Smart Car type? Here’s how to narrow the focus.
Human nature: Describe what the typical end-user believes about human nature (e.g., humans are generally good; humans are generally bad; humans are neither good nor bad). Hint: SUV drivers may think humans are generally bad, so we need to protect ourselves with BIG CARS.
Time sense: Describe the typical end-user’s relationship to time (e.g., focus on the future; focus on the now; focus on the past). Smart Car drivers may think that the future matters, so they buy smaller more environmentally friendly cars.
Person-Nature relationship: Describe the typical end-user’s orientation to nature (e.g., nature is to be dominated; nature is to be revered; nature is to be ignored). SUV drivers think nature should rule them. Just kidding.
Social relations: Describe the typical end-user’s relationship to others (e.g., individualistic or “dog eat dog”; collective or: “we’re all in this together”). SUV drivers are definitely dog-eat-dog. Hence the BIG CAR.
Space: Describe the typical end-user’s relationship to space (e.g., people control space; people live in harmony with space; space controls people). Smart Car drivers may believe that people should live in harmony with space, so they buy a smaller car, to park in urban settings, but also a car so they can conquer space and drive to the country for the weekend.
An additional set of questions around technology devices is also critical for technology designers:
What is the typical end-user’s primary interactive device? Surprise! It may be a TV remote control!
What other interactive devices does the typical end-user have?
What is the primary frustration the typical end-user has with his or her current primary device?
Do you know the answers to these questions? If not, how will you know whether you’re designin for an SUV driver or a Smart Car driver? You can find out the basics to these questions through a few simple steps:
Review any secondary value-based research, including omnibus surveys.
Complete quick and dirty observations of your primary end-users.
Survey a larger group of your primary end-users.
Summarize and segment these findings to create value-based design personas
We bandy about the word “average” all the time. What exactly IS an average, and how does it help design research?
Use the average to quickly summarize something that is already a number: minutes, ages, heights, visits, etc. Don’t use the average to explain something that needs more detail. And keep in mind, the average gets “dragged up” or “dragged down” by extreme values. Sometimes it doesn’t tell you much of anything.
An example design research project might be about how people use their stoves in their kitchens. How can we use “the average” to help us design a new stove?
The average, in statistical language, is actually called “the mean,” which is a measure of “central tendency.” Researchers use central tendency to describe all their results quickly. Other measures of central tendency include the mode (the most common response) or the median (50% of responses are higher than this; 50% are lower).The mean describes the “typical” or average result.
But here’s the big myth: there is no such thing as “the average” in your data. If you ask 500 people to rate your new stove design on a scale of 1 to 10, and the average is 4, there is no guarantee that any single person actually said 4! In fact, the majority of responses could be higher than 7, but some 1s or 2s could “drag down the average.”
Worse, it makes no sense to use the “average” or “typical” in qualitative research. If you do interviews or observations, there is no way to calculate “the average.” So when you say, “the typical person has a four-element stove,” you’re actually doing a calculation. This may be actually quite false. What you may mean to say is “most people in our study have a four-element stove” (which is the mode).
Qualitative research does not accept the “typical.” It actually looks at each case individually and in enough detail to allow for exceptions or outliers. There is no “typical” case in qualitative research because you do not do calculations. You do not summarize your data in that reductionist way.
That said, how could you use “the average” in your kitchen stove study? You can do a back-of-the-envelope calculation to summarize your data. The “average age” of your respondents, for example, will tell you about how old people are. The “average number of minutes spent cooking” will give you a snapshot of how long people spend in their kitchens. The “average purchase price of a stove” will also give you a quick snapshot. Using “the average” is to quickly summarize something that is already a number.
But the “average use of the stove”? That doesn’t make sense. Nor does the “typical grocery shopping process,” or the “average complaint of stove use.” These cannot be summarized in “the average.”
I loved it when I first saw it. Brandtags.net invites users to look at a logo and type in the first thing that enters their minds. I found it fascinating — until I realized it’s yet another example of poor research perpetuating negative stereotypes of women.
Type in “Oprah” and see what happens. The top three most entered words? Fat. Black. Bitch. Yes, that’s right, Oprah, the maven of women’s media landscape is nothing more than a fat black bitch. How valid a representation of Oprah is this?
Oprah’s media universe is worth a fortune. She earned $260 million in 2007 and is worth $2.5 billion. Her daily talk show alone gets 7.3 million viewers (that’s compared to 2.9 million viewers for Grey’s Anatomy).
So I got to thinking. How is Brand Tags so wrong? So nasty? So racist? (Type in NBA or Citibank and you’ll see what I mean). Researchers are Harvard have shown how stereotypes work. We know that people rely on implicit stereotypes when they make snap judgments. This is the downside of Malcolm Gladwell’s Blink.
We live is a complex social world. We try to make sense out of it by looking for patterns. Theorists Berger and Luckman call these “typifications” or roles that we take for granted. Typifications help us because they allow us to know what to do in social situations without really thinking about it, or, as Berger and Luckman explain it, they alleviate us from making “all those decisions.”
All Brand Tags really does is tell us what those typifications are for the people who visit their site. Who is visiting their site? We don’t really know. The first rule of sampling is to ask yourself, are the people who participate systematically different from the people who don’t?
People who participate in Brand Tags are obviously Web savvy. Someone forwarded them a link and they filled it out. Perhaps they read business media because Brand Tags has gotten some press. They have the time to enter text. They are also anonymous.
Is this what you would consider a “representative sample”?
Brand Tags has promise (I myself have used it to gain insight about a few things). But it mostly has the worst of our stereotypes. Is that insight? Perhaps. But it’s not insight about Oprah — it tells us a lot about the people who are talking ABOUT Oprah.
Below is a (very!) brief overview of online surveys. This slideshow, via slideshare, is intended for people in the Web design industry. IAs, designers, media planners, strategists, usability researchers, and producers will learn if they should, in fact, do a survey.
Dori Tunstall has written a fantastic post that details how a simple card sort can become a deeper exercise in analysis. Dr. Tunstall is a PhD anthropologist and the University of Illinois at Chicago. She explains how anthropology takes design research to a deeper level in card sorting, a common technique for information architecture:
“In addition to the information architecture, I delivered statements about the continued meaning of gender classifications. In the course of conducting the card sort, I learned that men and women continued to classify domestic products based on stereotypical gendered spaces of male equals outside/garage, and female equals inside.”
Dr. Tunstall clearly lays out some of the deeper implications of design anthropology later in the post. This is a must read for anyone looking to deepen their design or research practice.
The output from qualitative research is often overwhelming. Unlike quant research, qual findings are often messy and hard to decipher. Here are some techniques to manage the voluminous data of qualitative studies.
Start with clear research questions: in an earlier post I explained how to set up a design research project, step by step. One of the most important steps is to create a clear and answerable research question. This seems like an obvious point, but often it isn’t. Qualitative research often appears to be “just talking to people,” which gives us all the mistaken impression that it is entirely unstructured. It isn’t. Take the time to define research questions.
Summarize frequently: Let’s say you’ve chosen to do in-depth interviews. After each interview, take 20 minutes to write out a brief summary of what you remember being the most important points of the interview (note that this is not a substitute to taking notes during the interview). These notes are the first step toward analysis. You are reducing “clutter” and irrelevant information. You are also exploring connections with previous interviews.
Reduce, reduce, reduce: You will always have more data (e.g., videos, photos, transcriptions) than you can use. Be ruthless by reducing what’s important. Edit down your videos to only the clips that are most important (keep the raw data for another time). Reduce your transcriptions down to select quotes that speak to your research question (and again, save the entire transcription for another time). The goal is to have a workable set of artifacts.
Visualize the results: Many qualitative researchers make use of summary tables and diagrams to further summarize results. My favourite visualization method is the mental model, which can convey a huge amount of information in a synthetic way, quickly. Other tools include mind maps and even the simple bulleted list.
Hunt for connections: There is no science to this process. It is iterative and intuitive. But there are approaches you can use to find connections. I frequently use the “open sort” technique, with nothing more than a blank wall and post-it notes. Scribble themes onto post-it notes. Sort them into categories. Name the categories. Collapse as many categories as you can until you only have 4 or 5 “buckets” that explain your findings. If you’re researching children’s commutes to school, for example, you may have a category called “independence” which would talk about kids’ desires to be grown up, to have their own transportation method, and the knowledge to get to school. They are related only through the higher-order notion of “independence” and not the lower-order ideas of “transportation” or “age.”
Ask “So what?” often: When I was in journalism school, I had a professor who tirelessly quizzed us with his version of so what: “What does it mean to metro?” he would demand, meaning, why should the people of this city care? Why should your design team care about these results? What does it mean for their process? Why should the users of this product care about your results? How might it make their lives easier or more pleasant? And of course, why should bean counters of all sorts care? How much money will it actually save?
These general guidelines will help you in your journey to deciphering meaning. But no qualitative project can be save from poor research design. Make sure you’re using the right approach.
Sam Ladner, PhD, is interested in technology and work. She writes this blog to help designers create technology that works for people (and not the other way around)
