public digitalThe public digital logo

A practical guide: how to test and learn

This article is included in our book, Adopting the practice of test and learn. It was written by Connie van Zanten. 

You can download a PDF version of this book, or visit this page to request a printed copy.

When an organisation is able to test and learn, it has a much better chance of designing and delivering products and services that truly meet the needs of its users.

It’s especially useful when you are delivering in a context where there is complexity and risk.

At its simplest, taking a test and learn approach involves articulating assumptions and hypotheses about what you think will work, and then testing them with real users in the smallest, fastest and cheapest way possible, in order to learn if those assumptions and hypotheses are correct.

Taking this approach creates ‘feedback loops’ between ideas and reality. The more feedback loops you have, the more confident you can be that whatever you’re changing or attempting will work in practice, and have the impact you intended.

Taking this approach will improve the quality of your products and services. It will see you deliver impact more quickly, and is the safest and surest way to scale effective change.

Read on for some practical guidance that will help you take a test and learn approach.

test-and-leaen-2.png

Be clear about the outcomes you’re aiming for

Whatever transformation, change or improvement you are attempting, it will be driven by some organisational purpose.

Rather than focusing on the thing you are trying to build, or the deadline you are trying to meet, ask: What are you trying to achieve? What difference are you expecting to make? And what indicators will tell you if you are succeeding? These are the cornerstones around which teams can convene and deliver.

  • If your intended outcomes are not clear, clarify them.

  • If they are not widely known, communicate them - over and over.

  • If they are lengthy and complicated, simplify them to make them easy to remember.

The process of setting your intended outcomes can work best as a collaborative exercise, especially between leaders, key stakeholders and those that will be responsible for doing the work day-to-day.

When outcomes are clearly articulated, widely understood and oriented as the focus of your work, you will have much more freedom to work at pace in testing ideas and hypotheses that you think will achieve impact.

Build a multidisciplinary and cross-functional team

Taking a test and learn approach works best when your assumptions and hypotheses are formed collaboratively by a multidisciplinary and cross-functional team. Multidisciplinary means having a range of specialist skills in practices that enable user-centred design and delivery. Cross-functional means having people from different ‘functions’ of your organisation actively working together.

Gathering this range of experiences, perspectives and skills gives you the best chance of identifying feasible and impactful improvements, and testing them in safe and realistic ways.

Who you need will depend on the problem you’re trying to solve, but at its core, the team needs to bring together expertise across operational service delivery, technology delivery, strategy, design and enabling functions like procurement, HR or legal. You are aiming to build a group of people that collectively:

  • Have a deep understanding of the policies, strategies or laws governing the service and its users.

  • Have hands-on and recent experience of frontline service delivery.

  • Have design and delivery management skills, to complement deep subject matter expertise.

  • Are empowered to make decisions about how to run the service based on what they learn about the user experience, without having to seek approval through multiple layers of governance.

Depending on the service and how it’s delivered, this can also mean bringing together people that work in different organisations.

Once the team has built a shared understanding of how the service operates currently - end-to-end and front-and-back stage - it will be able to use its collective expertise and experience to decide what to try doing differently, and design feasible experiments which will help them understand the impact of that intervention.

Taking the time early on to build a shared understanding of how things work now will vastly accelerate the team’s ability to make decisions and collaborate effectively.

test-and-learn-1.png

Find a starting point for testing that sets you up for success

Has clearly bounded scope and scale. Perhaps you can identify one contained cohort of users for whom you will try doing things differently. Or you might start with one specific point or problem in the product or service. Often, it’s helpful if your team is equipped to take on or support operational responsibility for delivering the service while you’re experimenting.

  • Fits with the organisation’s strategic priorities. You won’t be wasting your efforts on something that’s not valuable for your organisation.

  • Has sponsorship and ownership. You have strong leadership interest and ‘cover’, and the freedom and protection to work differently.

  • Is end-to-end. You will be able to look at and affect change across the entire user experience. This matters because if you change something in one place, its impact is likely to be felt elsewhere too.

  • Will enable the team to have rapid impact. In other words, you can reasonably expect to demonstrate tangible, measurable impact for users within 3-4 months.

  • Has short feedback loops. You’ll be able to set up and run short, small tests that enable you to learn directly and rapidly from users and those that deliver the service.

Work in the open

Taking a test and learn approach means accepting and embracing ambiguity, and the fact that we can’t know all the answers upfront. That’s uncomfortable, and often counter-cultural. To manage that discomfort, the team will benefit from adopting a position of radical transparency, or as we call it at Public Digital, they should commit to ‘working in the open’.

This means the team puts conscious effort into making their work visible to people who are not on the team, including senior stakeholders and other colleagues who are interested in or whose work impacts the user experience.

It means continuously sharing information about the team’s work and learnings, including not just the successes, but work in progress and efforts that didn’t go as planned.

Face risks head-on

While experimenting may sound risky, it is actually an incredibly effective way of managing and mitigating risk in complex, ambiguous environments.

As with any experiment, a crucial part of designing your tests must be considering what might go wrong, how you will know, how you will mitigate against that, and how you will ensure you are able to act quickly to respond.

The following steps will help you to do that:

  • Ensure the experiment is contained, for example by setting a defined testing period and a fixed number of users.

  • Work out and agree in advance what to look out for as indicators the test isn’t working. It might be that the experiment itself isn't working, it's not having the intended impact, or it's leading to outcomes you didn't anticipate.

  • Undergo different kinds of tests - like comprehension and usability testing - to ensure the experiment is safe before running it in an operational context. It’s also always a good idea to do a practice run first.

  • Once the experiment is up and running, stay close to it. This is especially important at the start, so you can act quickly as soon as you notice anything amiss or unexpected. A fundamental aspect of taking a test and learn approach is that the team is hands-on and present - they don’t set up an experiment and then walk away.

Turn governance into a service

Every moment spent preparing for, or reporting to, a governance body is time not spent learning from users, iterating on designs, or delivering value. To test and learn effectively, the team’s time must be treated as gold, and they need an approach to governance which reflects that.

Imagine that your governance function is a service provided to delivery teams, designed to make it as easy as possible for them to learn and deliver value quickly and safely. How would you design it to be lightweight, responsive and helpful? What would it look like if governance worked as an enabler, helping delivery teams to be in control, and move at pace in learning and delivery?

For leaders, that means:

  • Embedding governance principles into day-to-day delivery processes.

  • Going to the team where the work is happening, rather than making them come to you.

  • Streamlining approval processes and setting parameters in which the team can act to reduce the need for approvals.

  • Focusing on coaching and unblocking over policing and reporting.

  • Shifting from rigid stage gates to progressive, real-time guardrails that evolve as you learn more about what will and won’t work.

For delivery teams, it means:

  • Being open and transparent about what you are doing and why.

  • Committing to sharing what you’re learning.

  • Investing consistent effort in making the work visible as you go, and inviting people in to see it.

Start small, prove the impact

Adjusting to this way of working isn’t easy. Organisations are often used to managing change and risk in a very different way, and a new approach can be challenging and uncomfortable.

So, start small. Don’t pick the hardest problem, or the biggest one. Focus on getting started, allowing yourselves to learn how to test and learn, and showing that it works.

Set your outcomes and share them far and wide. Work in a multidisciplinary, cross-functional team. Consider the criteria and conditions that will set you up for success. Show the rest of the organisation what you’re doing, why, and what you’re learning as you go. Name and tackle the risks head on. Work across teams and leaders to develop an approach to governance that is lightweight and responsive, to allow you to learn and deliver at pace.

And as we tell our clients, the main thing is to just get started.

A team reviews post it notes on walls

Previous: 

Next:

About Connie van Zanten

Connie is a Director at Public Digital and an agile delivery and transformation specialist. She has worked with governments, charities and health organisations, including the NHS Race and Health Observatory, where she led the sickle cell discovery project, elevating the voices and needs of those with sickle cell disease. Prior to joining Public Digital, Connie led a team of agile coaches at Cancer Research UK.