Communicating the ROI of UX within a large enterprise and out on the streets
JD Buckley, Principal, UX Strategy & Research, ADP Innovation Center
UX is being recognized as a significant business differentiator for new as well as established organizations. This recognition has come with inevitable questions regarding how best to express the value of design teams and quantify the influence of design thinking methods and practices on a company’s bottom line. Design teams are seeking new ways to develop and refine UX return on investment (UX ROI) models for use as communication tools across a wide range of products, services and complex business environments. In this talk, JD Buckley will discuss an emerging model to communicate and measure the impact UX teams provide to businesses. Further, she’ll examine how this evolving process framework and model might be applied across other intricate environments inside and outside enterprises, including as a tool to reflect the impact of an anti-gun violence educational toolkit for middle-schoolers and educators.
The second theme of the conference, Communicate, was introduced by Margot Dear.
Communicating our value to business is important, but we need the right skills and techniques to engage in dialogue about our value. More and more companies are investing in UX; ADP has grown from 40 to 300, Verizon bought Moment Design, and Capital One bought Adaptive path not that long ago. These presenters will be sharing tips and tools from their experiences, which we can apply to our own workplaces.
JD Buckley is a Principal UX Strategist & Researcher at ADP, and she has worked with a number of companies including Kaiser, Disney and others. JD is going to share an ROI model, and how that is a tool that can be used to communicate the value and measurement for UX in an organization. She will share how it can be leverage in both for profit and non-profit organizations.
—
The latest buzz in our field is is ROI of UX. IBM and Frog have both published on this topic, and Design Management Insttrute and other organizations have as well. Companies are embracing UX research and design methods, and UX is finally finding itself at the grown-up table. But now, executives are asking us to quantify the value of our efforts. While that can seem challenging in consumer products, it can be really daunting in the enterprise. Buzz words aside, the question is whether ROI for UX can be be calculated, and if so, what would it take? Could it help communicate your team’s value to the organization?
JD is going to share two stories today. The first story takes place in a large organization, and the second story shows how they took that same framework into a completely different context – an antigun violence initiative.
She works for ADP, which we know as the payroll company. Five years ago, executive leadership at ADP decided to follow the lead of other companies and invest in innovation centers, which included the hiring 300 designers. They believed design was a critical part of their innovation strategy. The team she is working with is focused on improving the compliance payroll process (e.g. paying federal taxes, wage garnishment), and to do so they follow a user-centered design process. The team interviews stakeholders, conducts end-user validation, does iterative prototyping, and tests usability with stakeholders. Some time ago, they received a green light for a redesign of their flagship product, which had been delayed for the past few years . The organizational support for the redesign gave them a moment to celebrate, but also to reflect.
The UX team realized that if they wanted to demonstrate their impact, they needed to benchmark their users’ current experience as a point of comparison. Along the way, they found a lot of theory but very limited specifics, especially when you consider the complexity of the enterprise context.
They opted for a mixed methods approach:
A key learning for them was that they couldn’t just jump into a benchmarking study; they first had to identify most important users and their top tasks. Then they could conduct a benchmark study, and later (after the redesign) collect the same data again. Their aspiration was to tie the outcomes of their UX work to company performance metrics.
The redesign release date was set, so they had to capture the benchmark before that – so they had to move quickly. Hours before the redesign release they completed the first benchmark study, while simultaneously building a sort of framework for building ROI. This numbered list looks easy enough, but it was not:
Just getting access to users was a challenge. Their subject matter expert’s knowledge was more subjective than expert. They realized that it was better to start with a broad array, rather than getting too specific, too soon. For them the moment of comparison was more of a sigh of relief. They learned over time that tying the efforts back to company profits requires thinking like an economist.
Metrics:
Connecting metrics to UX efforts is tough, and tying to company performance was really tough. Fortunately, a new CFO had just joined, and she seemed to have a good understanding of UX and what they were trying to do. She became a crucial partner in helping them focus on the right KPIs.
In the end, however, all of the metrics were pointless if they couldn’t be made actionable. So one absolutely critical step for them was connecting their Top Tasks to the prioritization of items on the product roadmaps:
To that end, the UX team helped refine goals and business success metrics for those features. The Service Profit Chain Model from Aaron Powers at AthenaHealth has informed their current thinking:
You have organizational metrics that you collect across the organization, like customer satisfaction, NPS. If you look at that as a model, how do they all contribute to revenue growth and profitability?
The team has completed two studies, and they are working on a third. They have seen a marginally (but statistically significant) improvements. They can now focus on the areas that need work, to deeply identify why, and what needs to change. They are in a cadence of collecting data twice a year.
What is exciting now is that their meetings with executive are about having a measurable impact on the bottom line. Now they talk about design and metrics in the same sentence.
JD has been reflecting on how building this framework helped them communicate important things like user-centered design (including cross-functional discovery), the value of prototypes, and testing against top tasks. Although they are encouraged by what they have accomplished, they started to wonder how flexible the framework might be.
So here is the second story …
JD teaches at Art Center College of Design, in a program called designmatters. They work with public, private, and social sector organizations to create a better, more humane future. Could their framework evolve to focus on improving people’s lives? She was co-teaching around an anti-gun initiative called Where’s Daryl?, and she was curious to see if it could be re-used in this very different, troubling context.
Kids that are touched by violence in some way are more likely to end up in prison.
Unfortunately, the CDC limits the use of funds around gun violence education. the Where’s Daryl? program shows how this boy’s life has been derailed in part due to his fascination with guns. In a fourteen week term, her students needed to create a digital experience. They also had the goal of collecting usage data, and ultimately measuring the social impact of the program. She shared the ADP framework with the team, and they collectively agreed that with some modifications, it could provide a strong foundation.
Through their work, the class created basic segments of health educators, and then they did fieldwork with educators. The research process was very humbling – they saw overcrowded classrooms and the limited supplies firsthand. They also learned that half the middle schoolers had firsthand experience with guns.
In a second step, the team identified Customer Tasks, and looked at the California credentialing system to ensure they were focused on things that would drive stakeholder buy-in for what they were doing.
Their next steps were to define metrics and connect to KPIs for the entire project:
Their main stakeholder had benchmarked the current experience, and even compared it to the new one using surveys of students and teachers. From that he had produced a recommendations report. The design students contributed by proposing a website with high quality, downloadable, printable assets. There were also feedback channels for the teachers, and engagement metrics like kit downloads. Data was looked at by jurisdiction. There was also a chatbot for students to engage / reflect on the curriculum, and fun social media elements like Snapchat filters (selfies with Daryl).
The stakeholder has scheduled to present to the LA school board to expand the program, and he feels confident about his upcoming meeting. The user-centered approach and metric-based framework allows him to articulate benefits for an audience, and effectively tie it to curriculum goals. For him, it was also valuable to continuously collect data; the measurement will support the ongoing evolution of the program.
What became most salient to her during the program is that gun violence data has been hard to collect, because it has been hard to understand the scope. Without benchmarks, there is no evidence of the current state, all that results in no communication. At the end of the day her experience in the classroom reinforced the need to measure the impact and scale of a problem. If we can describe the problem in this way, we can articulate the value of investing in UX to fix it.
There are 0 comments