Evaluating the impact of user research
This is a blog about methodology, I won’t be sharing actual results.
Depending on the maturity of User Centred Design and User Research in your organisation, measuring the impact and Return on Investment of user research may be relatively easy. Especially if you are consistently tracking UX metrics and the changes in the measurements can be directly traced to changes made to products and services based on research insight.
However, not every organisation has this matrix of metrics established. In some organisations, the standard metrics of conversion rate, abandonment rate, and loyalty don’t easily apply.
You may have to find a more qualitative way to understand if the research being done is having an impact on the user experience you are delivering. This is an example of an internal user research approach and interviewed all UX people who’d done user research in a 12-month period.
What we asked about:
- High-level project summary
- Purpose of the research and intended outcome
- What the research was based on [assumption, business need, past research]
- Who suggested the research be done?
- What was measured in this work? [Metrics]
- How were the results shared? Were the results of the research shared outside the immediate team?
- How were the results used?
Start a thing | Stop a thing | Fix a thing | Make a decision | Nothing | Other
- What was the size of the things done? Easy or fundamental changes?
- Were they invited to important meetings because of this research?
- How long did it take you to complete this research?
- Does old research get referred back to when new work is started?
Logistics
Because of the highly sensitive nature of asking these kinds of questions, especially if the answer is my research wasn’t used, no recordings were made of the research sessions. Only the moderator, notetaker and participant were present during the session. Only the two doing the research saw the raw data.
How we analysed the data
The table below demonstrates how we coded and summarised the data:
Domain | Project Name | Level of maturity | Quant / Qual | Tactical / Strategic | Type of research | Purpose | Based on | Type of usage | Measures | Level of Impact | Cost
The level of maturity was assessed using Jared Spool’s framework: https://articles.uie.com/how-a-team-matures-its-user-research-integration/
The purpose was broken down into 6 sub-categories: subjective, objective, validation, understanding the problem, prioritisation, and unclear.
Type of usage was broken down into 5 sub-categories: Design the right thing, design the thing right, stop work, contextual knowledge, and validation of ideas
We combined these results with a health check survey we ran at the same time as doing this research, to better understand people’s perception of working in UX and how the Re Ops team were doing.
How we communicated the results
We tailored the findings depending on the audience:
- Head of UX: the full picture
- Programme UX Leads: a programme-specific picture
- All UXers: what makes impactful research? Coaching on how we can make our research more impact / how we can work together towards the vision of what we want the vision of user research to be
Why we choose to have 3 slices of the narrative
These are sensitive messages to communicate because we are not in an ideal world where all research is used, and all value is maximised
- We didn’t think it was useful to compare domains to each other — they are working on different things, with different resources, pressures, cultures, and ways of working
- The Head of the department should have the full picture to enable evidence-based decision-making for the teams
- We didn’t want UXers to feel criticised because not every piece of research had an impact — we wanted to focus on the future, and make positive changes
The elements of impactful research
“Happy families are all alike; every unhappy family is unhappy in its own way.”
Leo Tolstoy — Anna Karenina
Impactful research consistently has the same elements. Research that doesn’t have an impact, is not impactful for many unique reasons.
Impactful research has:
- Clear defined purpose and well-defined research questions
- [if appropriate] useful measurements of experience
- Agreement from the whole team that it is a priority
- Based on something — assumptions, business drivers, past research
- Can be tactical or strategic
- Using the appropriate research method for the question being asked
What affects the impact of research is not just the skill and experience of the person doing the research, but also the environment and culture the research is being done in.
Image credit: Etienne Fang — Amazon — March 2021 (Advancing Research Conference)
To realistically increase the impact of research and UX maturity, work needs to be done at multiple levels:
- Individual learning and development
- Community training and upskilling
- Cross-disciplinary awareness raising and training
- Incentivising teams to work in an evidence-based, user-centric way
It’s not about blame and labelling people, teams, or programmes as maturity or immaturity but working together to raise awareness, skills and maturity to solve customer problems and deliver values to users.
