Quantitative UX Research. Big Data. Data Science. Analytics. They’re all about figuring out exactly what’s going on. But sometimes they miss the mark. Until they get help from good old qualitative research.
A study by Microsoft and the University of Florida about academic collaboration used several iterations of qualitative (quant) research to inform and improve their quantitative (quant) models.
“Combining qualitative and quantitative approaches enables scientists to cultivate a deeper understanding of both what network structures exist and why these cultural processes developed. This dual analysis adds depth to both sides of the analysis.”
In this post, I’ll summarize that study. I’ll focus on the high-level ebb and flow instead of the specific findings or methods.
Round One: Ethnography (Qual)
“We began our project by conducting six months of participant observation at a large university.”
Round Two: Model Creation (Quant)
“We then used [scientific collaboration] data to map the university’s scientific collaboration network.”
Round Three: User Interviews (Qual)
“Detecting a variety of different communities also allowed us to test multiple networks with participants during our interviews. This user feedback helped us understand the key differences between these different community visualizations and modify our criteria to create models that better fit reality and their cultural context.”
Round Four: Profile analysis (Quant)
“We performed a profile analysis of each researcher or clinician identified in an emerging community to determine their research topics”
Round Five: Semi-structured interviews (Qual)
“Semi-structured interviews with three researchers to learn about their collaboration experiences and ask for feedback on the visualizations.”
Round Six: Update Models (Quant)
“The user feedback from this stage let us know that we needed to revisit our criteria.”
Round Seven: Additional Feedback (Qual)
“We solicited additional feedback from six researchers on our revised visualizations via email… we [collected] additional ethnographic data on these emerging research fields through interviewing sixteen scientists and clinicians.”
Round Eight: More Profile Analysis (Quant)
“We then conducted another round of web profile analysis to evaluate whether or not the pairs were viable… we identified fifteen pairs of thirty investigators and sent them an email outlined the study’s goals and incentives… three pairs were asked to submit a full grant proposal. These proposals were peer-reviewed and the institute we partnered with selected which pair received a pilot award of up to $25,000 to complete their project.”
Did you catch that? EIGHT iterations
The goal of the study was to accurately model the entire scientific collaboration network of the university so that it could find the most compatible pairs of researchers. If they had stopped after the first quantitative model, they would have missed the mark.
“It is important to create visualizations that fit people’s mental models of the phenomenon you are trying to map or they will simply dismiss your results as irrelevant.”
Instead, they relentlessly tweaked their model then tested it by talking to the people that make up that network.
Maybe we don’t all have time to do eight rounds of research, but two or three is clearly better than one!
The bird’s-eye view and the worm’s-eye view
To paraphrase an analogy from Nobel Prize Winner Muhammad Yunus explained, birds see a lot, but not in much detail. Worms don’t see everything, but they see it up close. Alternating between the bird’s-eye view (quant) and the worm’s-eye view (qual) allows you to get the best of both worlds.
This study was a great example of that iteration.
Shane Gryzko's Blog 