Dave Rogers
Partner
What is 'Test and Learn'?
Earlier this week, Pat McFadden (Minister for Intergovernmental Relations of the United Kingdom) outlined ambitions for the reform of the state in the UK. He referred to a “test and learn” culture as an example of what public sector reform can – and should – look like. But what exactly does ‘test and learn’ mean?
Test and learn is a way of working that teams can adopt –from small teams, through to whole organisations. It’s an enduring, cyclical way of working: test, learn, test, learn, test, learn and so on. It embraces the maxim that anything that works at scale grew from something that already worked at a much smaller scale. This smaller scale might be a geographic region, a particular subset of users defined by a characteristic, or perhaps a narrow set of initial functionality. Using test and learn cycles, you go from the small, to the big. You scale up gradually, avoiding the pitfalls of a ‘big bang’ approach.
Testing is trying what works and what doesn't work - in a manner that is safe and considerate of cost. Learning is gaining insight into why certain things work and other things do not. Crucially, the learning is from real users, in real circumstances. The best kind of learning will be surprising, and will shape delivery in a profound way.
Test and learn is Universal Credit in 2014: starting with only 20 users in Sutton – and growing area by area to deliver a nationwide service.
Test and learn is Netflix: starting with DVDs in the post, eventually disrupting the global TV and film industry.
Test and learn is Hackney Council’s ‘Here to Help’ service: providing proactive help to residents at scale. It began with emergency food deliveries during COVID-19 and evolved by applying those lessons to develop a new approach to supporting residents affected by the cost-of-living crisis.
Test and learn is a set of practices that have been shown to work in private and public sectors, on a local, national and global scale.
Importantly, test and learn is not a delivery method, it’s a holistic approach that applies to every phase and every aspect of an organisation’s work. It is:
Applied at every phase - when starting, when scaling up, and throughout the long term provision of a large-scale service.
Applied to strategy, to delivery, to the operating model, to how you create your teams and shape the organisation.
Related to every aspect of a service – its users, its technology, its places and regions, its commercial supply chain, its people, its legal and legislative framework.
A technical note for those familiar with design, delivery and operational methodologies
Test and learn is an approachable, plain language term to encompass and expand upon many established industry or domain-specific methodologies, but avoids the jargon and connotations that these come with.
Test and learn is not, in itself, new; as the test-learn cycle exists at the heart of many established methodologies. For example: we have agile (e.g. Scrum, Kanban, Lean), design thinking, ‘Lean Startup’ (e.g. minimal viable products), A/B Testing and Multivariate Testing, Six Sigma and more broadly, the scientific method.
A magnet for modern, effective ways of working
To test and learn well, there are a set of practices that are necessary. But helpfully, test and learn creates a ‘pull’ for these practices which become easier to adopt as the work continues.
Firstly, test and learn teams must be multidisciplinary. They need a breadth of skills and experience to be able to go through the test and learn cycles quickly, and expertly. If the team needs to regularly hand off to other teams, the whole process will slow to the speed of the slowest hand-off.
Secondly, test and learn teams thrive under modern, servant leaders – people whose behaviour is open and collaborative, people who trust teams to deliver outcomes while remaining open-minded about the outputs along the way. This allows for a learning culture, a willingness to accept new information and change the plan in response.
Thirdly, test and learn teams work in the open. They build trust in those around them by showing them what they’re learning, and explaining the reasons behind the choices they make. This mitigates the anxiety that can be felt midway in the journey when scaling up from the small to the necessarily large.
It’s quietly radical
Test and learn is a quietly radical idea. It’s quiet, because the idea itself doesn’t seem obviously radical. It may strike a lot of people as simply being common sense. If you’re solving complex societal, systemic problems, of course you wouldn’t assume to know the right answer years in advance of delivery. Of course you’d try things, see what works, what doesn’t.
It’s radical, because most organisations – UK government included – don’t really have the operating model and culture to support test and learn at scale. Test and learn leads to deep organisational change, to new ways of funding, governing, buying and staffing. Most importantly, it changes the culture to be more humble, open and built on trust.
It’s not making it up as you go
For long-term buy-in to the test and learn approach, it’s crucial to recognise its caricature - an oversimplified portrayal that can invite ridicule and rejection. This is when test and learn is seen simply as ‘making it up as you go’ – an approach with no direction, plan or foresight.
Effective test and learn is guided by a strong sense of purpose. It is oriented toward outcomes (changes that really matter to users) rather than outputs (apps, websites, call-centres or drop-in centres).
Effective test and learn has a plan. This plan isn’t linear but rather a routemap showing various potential routes through the problem space. It’s a plan that embraces uncertainty,where the detail and precision decreases the further into the future you go. It’s not a wasteful exercise in false certainty.
Effective test and learn forecasts; it uses the information available to make reasonable predictions of the future. It shows the uncertainty in those predictions, but also engages with people who need to understand the consequences of different kinds of forecasts – such as people planning budgets and grappling with affordability and constrained resources.
Effective test and learn can still mean making commitments. Rather, when leaders plan events, or make time-bound promises, they do so understanding the uncertainty of plans and forecasts. Test and learn teams understand they may need to adapt their priorities to respond to events and promises.
The adoption of a test and learn culture has the potential to profoundly transform public and private sector organisations alike, enabling them to navigate complexity with agility and humility. As McFadden highlighted earlier this week, the test and learn approach offers a pragmatic yet radical alternative to traditional methods, fostering innovation through iteration and evidence-based learning. At its core, test and learn is about creating teams, organisations and services that are adaptive, human-centered and capable of driving impactful change at scale.
So let’s get to work.
Written by