Standardizing Product Metrics for Leaders, Designers, and Everyone
Jennifer Cardello, Head of Design Ops, aetnahealth
There is nothing more frustrating than dedicating your blood, sweat, and tears to developing a product, only to see it shut down for no particular reason. We still lack standards for measuring potential opportunities and solutions—so leadership continues to base decisions on intuition, personal experience, and other factors that often barely correlate with success. Organizations large and small need agreed-upon measures of potential product-market fit for their concepts and solutions, ones that help establish unmet needs and lead to designs that users understand and want to use. The DesignOps team at Athenahealth has solved this problem by creating a standardized measurement framework—including qualitative and quantitative instruments—that helps product teams measure their concepts and solutions early and often. These are measures that leaders can use to make informed investment decisions across the larger portfolio, and that free product teams to be awesome at what they do: designing, managing, and developing products that lead to better experiences for users.
Jennifer is gong to talk about the delusions that happen inside our organizations. Jen has 20 years of experience in the UX Design field. She has worked in many different industries, and she was a Norman Nielsen Group (NNG) instructor. Until recently she was at Athena Health, but she is starting a new role at Fidelity soon.
Introduction
Jen started at Athena in 2014. They help run hospitals, health systems. For those clients scheduling is important functionality. Shortly after she arrived, a major release to their scheduling solution was cancelled.
Five step process to sadness:
- Propose a project
- Gain leadership approval
- Research and design and learn
- Build your alpha
- Leadership kills it just before launch 😞
Why did that happen?
- Company goals changed – ok
- It wasn’t going to impact bookings, so resources were repurposed – ok
- Too much change for our customers – ok
- But the one that scared her the most was “because they said so”
Those teams have lost their sense of agency, because they are not in charge of how decisions get made. That in turn leads to bad behaviors – skipping research, cutting corners on design work, and doing skunkworks projects.
Athena builds cloud-base software to help run medical practices. This ultimately enables those organizations to deliver better care, enable teamwork, and operate more efficiently. They are a purpose-driven company, and very earnest about it. They have been hugely successful. They have so much amazing knowledge that CDC comes to them to understand the progression of flu season.
The HITECH Act drove market demand for qualified companies. But (in part due to the fact that this was a government mandate), there was very low satisfaction across the board:
Their client base – and those of their competitors – were all unhappy. Over time, clients felt that too much was being put on their plates. Are our clients design-forward? This is her Litmus Test. They are:
- Seeking human-centered innovation. They really mean it.
- Recognizing design as a strategic lever. It’s not just a support function.
- Ready, willing, and able to invest in design.
There were 65 designers are already in place, and the fact that Athena was mission driven was a big plus for her. But she found a very different situation when she showed up to start work
This is a high level view of the claims process:
She did a brief walkthrough – it was a rabbit’s warren of experience. This is the result of incrementalism. It was clear that the we do / you do model with their clients was not working out well. They received very difficult feedback:
Designers were frustrated, and other teams were not seeing the value of design. She was getting feedback that she needed to make it easier for engineers, that design was getting in the way of shipping. This company passed her litmus test! Why isn’t design having an impact at scale?
When an organization is not honest with itself about success drivers, it under-invests in experience design.
You were born on third so you think you hit a triple.
In B2B healthcare, they are on the left. The truth is that your users aren’t happy – they are trapped. However, due to a variety of reasons (including new industry data standards, and users having a greater voice in decision-making) it became easier to switch to a competitive product. So, the company had to increase design influence across the entire organization. They had to capture ALL the attention, establish a sense of urgency, and deliver the goods. They did the following:
- Proved that design impact outcomes
- Created a shared language for design, so we can all talk together
- Redesigned product decision-making
Proving that design impacts outcomes
Does design “move the needle”? The first thing they needed to figure out was what what metrics matter. For them it was Retention and Referrals. They dug for data and found a lot of it, including 60K survey response from the app about the users perception of experience. They cleaned the data and ran an analysis. Those efforts confirmed unequivocally that users’ perception of the product does impact retention and referrals.
Create a shared language for design
Clients were tired about the ‘you do’, and we were trying to cut their workload down. There was a lot of talk about taking on client tasks, but she was also clear that they could improve what was left. They evaluated the tasks of their users across products (we can read a Medium post released today about this). And they worked with colleagues across the organization to find 55 of these critical workflows:
They were looking for things with high usage, high friction, or that could be a competitive differentiator:
Using the NNG Heuristics they did a series of scoring sessions. They were three hour sessions in which they (designers, product managers, SMEs) would watch a video. They would go through the heuristics and arrive at a score, which they they aligned on in real-time. Everyone learned a lot during that process.
Then they did in-depth audits about what could be changed so it worked better. They all used the same sprint cycle as the4engineering team, because they wanted to inform backlogs as they were learning. Their process was to record for the first two days, do the collaborative scoring sessions, and conduct five days of audit. In nine sprints (18 weeks) they created the beginnings of behavior change in the organization. They use this behavior change framework for all their design operations initiatives:
There is never enough attention on this topic.
They reviewed 55 workflows and made 3250 findings. They did a meta-analysis afterwards, breaking it down by users and by heuristics to look for patterns. Finally, the wrote detailed reports for each team,
The task she showed earlier scored a .78 out of 4.0. They were able to simplify and streamline it, ultimately reducing the time to task completion by 50%.
They also took the time to teach the heuristics to the product managers so they could participate in the evaluation.
Redesign decision-making
They created an Experience Measurement Framework:
These make it very low friction to run their own testing, and it also means that teams don’t have to reinvent the wheel each time. Concept validation was one area of change which has been really important to drive improved decision-making around the products.
They had one feature “tabs” that had been sitting in Alpha for awhile, and they were going to kill it. But using their framework they did a quick concept test and realized it was a great feature that they should ship.
The scoring enables them to make decisions about where to invest, and it gives them a (much-needed) way to make product decisions.
Her new and improved Litmus Test includes making sure that the company is being honest about where they are, so they are truly ready to tackle the changes that need to be made.
There are 0 comments