public digitalThe public digital logo

Where does test and learn come from?

This article is included in our book, Adopting the practice of test and learn. It was written by Lara Sampson.  

You can download a PDF version of this book, or visit this page to request a printed copy.  

Designing for human behaviour

If you are building a bridge or a motorway, this traditional approach works. You need a stable blueprint to follow, and you are unlikely to learn something halfway through its construction that means you should radically change your design.

But if you are building services to be used by people to achieve some sort of behaviour change - perhaps how they manage their money, or make buying choices, or engage with health providers - then any ‘blueprint’ is almost guaranteed to be wrong. This is because human behaviour, unlike the construction and use of a motorway, is inherently unpredictable.

Here’s an example: Universal Credit - the UK’s working age financial and work support service - is household based. That means couples need to apply as a unit, share their information with each other, and receive the money together to manage. No other country in the world operates a benefit system like this. Consider how many different ways couples in the UK might manage their money: who organises the admin, and how they use different pots for different purposes. You can try to nail a solution down and roll it out, but you will be orders of magnitude wrong.

Which is why the traditional ‘waterfall’ approach to change doesn’t work: It’s rooted in a deeply flawed assumption that you can define a solution upfront in the absence of any empirical evidence of how the humans using your services will behave in response.

This inherent unpredictability is compounded by the new reality of the internet era. The context in which your services will operate is changing exponentially faster year on year, making the behaviour of your users even harder to predict.

Instead, designing for human behaviour demands a hypothesis-based approach with a deep willingness to change the design as you go.

The birth of ‘agile’

Decades ago, software developers realised that taking this linear ‘waterfall’ approach was not only risk laden, it was also unnecessary. While you may not easily be able to change the design of a bridge halfway through, the beauty of software development is that you can change its design as often as you like - as long as you are set up for that. They called this new approach ‘agile’ and every internet-era successful business is built on its principles.

However, most large organisations with long heritages don’t have agile software development as part of their origin stories. While some may make the commitment to ‘be agile’, and put pressure on their teams to do so, the cultural and operational obstacles in their way make this a near impossible task.

In fact, it’s the unfeasibility of the sudden adoption of agile in pre-internet organisations that sparked the pointless war of waterfall versus agile - one which many organisations have spent hours in combat over.

The benefits of agile - the reduction of risk through rapid, early delivery and continuous improvement in response to user needs - are only really available when the whole organisation is walking to the same drumbeat. The attempts to put agile technology delivery into waterfall project and programme management (‘wagile’ briefly featured in the lexicon) is truly the worst of all worlds.

Rather than trying to force ‘agile’ on pre-agile organisations, developing a feasible approach to change depends on rethinking what it means to be a modern day organisation. It means being ready to change your policy and strategy in response to what you learn, and replacing your static target operating model with one that is dynamic and curious as to how it should change and adapt. It means being ready to change the governance of your organisation to value empirical evidence of what works through delivery at pace, over plans which promise that delivery. It means starting small and iterating, avoiding big, one-off investments and ‘big bang’ delivery.

It is about much more than how you do technology, or digital. It is about who and how you are as an organisation. And it is enabled by the idea of test and learn.

The birth of test and learn

Test and learn is a way of thinking that makes the rewards of agile approaches accessible to large organisations with vast estates of legacy systems and heritage ways of working, enabling them to pursue transformation at scale and with impact.

It is a complex idea that, when applied, rewires everything about how an organisation thinks and operates, but is broadly defined by three core features:

Starting with problems to solve and outcomes to achieve, not solutions to deliver. This approach begins with a clear understanding of what you're trying to achieve for users and the organisation, rather than jumping straight to how you'll achieve it. It means developing hypotheses about what might work, then gathering empirical evidence by testing those hypotheses with real users in real operating environments as early as possible.

Being humble about hypotheses, plans and strategies, and being ready to change them when you learn new things. Test and learn assumes that your initial designs might not work, and creates space to continuously test and refine them. It understands that wherever you get to on day 1, you will make the design better on day 2, 10 and 50. In fact, that process of incremental change is endless, because test and learn is based around the reality that transformation is not a time-boxed process, and your services will need to be sustained.

Bringing professional experts together into teams with the autonomy to make choices and governance that supports them. Testing and learning means bringing together people with the relevant skills, knowledge and experience in multidisciplinary teams. It is only with this expertise, gathered together at an early stage, that teams can create hypotheses about what might work and test those out in controlled ways. Crucially, these teams need the authority to act on what they learn.

All organisations can test and learn

Agile methodology isn’t universally applicable. The conditions for its success are limited to organisations with software development in their DNA.

But while test and learn is informed by many of the principles of agile, its application is organisation-wide, and so it belongs to everyone. Rather than requiring pre-existing conditions to be successful, it recognises the need to adapt organisational conditions to begin with.

Most organisations aren’t naturally set up for test and learn. Its practices may feel unfamiliar, even uncomfortable, and it can be hard to fully grasp until you've tried it yourself.

But in our work supporting countless clients to test and learn, we’ve seen that all organisations, whatever their context and challenges, are capable of making the changes to accommodate this approach and start transforming more effectively.

Next:

About Lara Sampson

Lara is a Partner at Public Digital, where she has supported clients including the Home Office, the Mayor’s Office for Police and Crime (MOPAC), Arts Council and Unicef UK. Previously, Lara led the product development of Universal Credit, taking it from its inception in 2013 through to 6 million customers in 2021. She has also held roles as a senior leader in the Department for Work and Pensions, and as a digital strategy and delivery expert in the private sector.