How Quant UXR works in the real world

Edit: I’m transitioning from WordPress to Medium. To see the latest version of this article, see How Quant UXR works in the real world on Bootcamp

---

When I wrote How to get started with Quantitative UX Research, one of the steps was “Learn about Quant UXR in the real world”. I included a couple articles from Google and Facebook. These were good high-level overviews of what Quant UXR means at those companies, but they don’t really get into the details.

Since then, I’ve found a few wonderfully detailed case studies of what Quant UXR involves.

This is a collection of those case studies.

IDEO

Just Add Water: Lessons Learned from Mixing Data Science and Design Research Methods to Improve Customer Service by Ovetta Sampson (case study, 42 minute read, free EPIC account required)

Yes, it’s long. But it’s a wonderful deep-dive into the journey of a mixed-method team as they tackle a big, juicy problem: improving the fractured experience of customers trying to book a cruise.

There’s no way to know of course if the results would have been as good without the mix of a multidisciplinary team. But it’s a certainty that it took both ethnographic research and data science to pull off the right solution.

Ovetta Sampson, Just Add Water

Speaking of Ovetta Sampson, I also recommend her talk Toward a More Perfect Union: Marrying User Research and Data Science for Human-Centered AI Design (video, 38 minutes). This isn’t a detailed case study, but I have to mention it anyway. It’s is a FANTASTIC overview of how User Research and Data Science teams can work together, with a good reminder of where that data comes from:

Say it with me: all data is created by people and all people create data

Ovetta Sampson, Toward a More Perfect Union

Amazon

Humanizing Quant and Scaling Qual to Drive Decision-Making by Lauren Morris and Rebecca Gati (case study, 34 minute read, free EPIC account required)

Another EPIC case study, in more ways than one. I love the story that Morris and Gati tell. They started with a quant-focused culture, then brought the UX research team together with the analytics team (much like Ovetta Sampson describes for IDEO in the links above). They then reviewed past qualitative studies to build a framework to describe what their users need and iterated on that framework to use Customer Experience Outcomes (CXOs).

A CXO captures a discrete customer need, derived from research insights, as 1) a “job” a product must do to address the need and 2) the criteria customers use to judge how well a product does the job… Each one was measurable… They intentionally had no mention of implementation or a solution, as they were meant to describe durable customer problems.

Morris and Gati, Humanizing Quant and Scaling Qual

The team then ran a field study to test their CXOs and included data scientists as well as executives in the field work. Later, they built a survey framework to quantify the performance of each CXO and quantified the correlation between these CXOs and their key business metric: customer engagement. Lastly, they created a dashboard to show this data.

Since this integrated qualitative-quantitative program has come online, it has become a key input into the Prime Video decision-making process at all levels of the organization

Morris and Gati, Humanizing Quant and Scaling Qual

Fidelity

“Right problem, right solution, done right”—The Vanguard of User Research (ft. Jen Cardello) (podcast episode, 49 minutes)

This isn’t quite a case study, but I had to include it because Jen Cardello gives a lot of great insights on the structure of her team and a few tactics that she uses. For example, “the Atlas”, which is a map of insights across customer segments and their jobs-to-be-done. This is brilliant, because it makes it very clear which areas are “unresearched”.

Hopefully the Atlas itself will provide us with that mechanism to point to white space and say, “Hey leadership, wouldn’t it be great if we actually could turn that box green, because we knew things about that.”

Jen Cardello

She also talks about a “qual-quant-qual sandwich” to help ensure the team is focusing on the Right Problem. First, qual identifies pain points and customer needs. Then the Outcome-Driven Innovation process is used to quantify how well customer needs are being met. Lastly, another round of qual identifies root causes.

User Testing

Why Data Science and UX Research Teams are Better Together by Chris Abad (video, 44 minutes)

This isn’t quite as detailed, but like most of the other case studies I’ve seen, it describes qual people working closely with quant people. I love the back-and-forth conversation about 12 minutes in:

Researcher: “People make copies of their studies but not very often. It’s probably just a strange workaround. What do you see?”
Data scientist: “Actually, 80% of studies are copied. 80% of those studies are copied again. But why?”
Researcher: “Hannah mentioned in an interview, she typically copies studies from a ‘chain’ of studies. Does that help?”
[Data scientist creates a data visualization of the “chains”]
Researcher: “I thought people were creating templates. I can see that’s not it now. Maybe these chains are actually projects?”

Chris Abad

Got any more case studies?

Do you know of a Quant UXR case study that I didn’t cover here? Let me know and I’ll add it to this page!