Why metrics matter in Digital Public Infrastructure

15 October 2024

Digital Capability Building

Earlier this month Public Digital attended the Global DPI summit in Cairo. Read our main takeaways.

The summit highlighted the rapid advancement of the DPI sector, and the brilliant work being done by digital agencies across the world to harness its benefits.

However, as PD’s Director of Strategy and Research, I’ve been looking at how defining and measuring DPI impact remains a challenge. Measuring impact is critical to help make the case for investments, build political support and ensure that deployments deliver public value.

At the summit, I joined a panel discussion alongside Caribou Digital, the Digital Impact Alliance, Mastercard, and DFS Labs to explore this question of metrics in relation to DPI: what metrics matter most, to whom, and why? How can we best capture and use these? And what factors disrupt the use of metrics?

Given Public Digital’s experience working inside and with governments around the world to support their digital transformation and DPI journeys, I spoke to the government perspective.

A panel from the DPI Summit featuring Public Digital, Caribou Digital, the Digital Impact Alliance, Mastercard, and DFS Labs — A panel at the DPI Summit featuring Public Digital, Caribou Digital, the Digital Impact Alliance, Mastercard, and DFS Labs

I shared two key insights with the panel:

The value of DPI should be measured from the perspective of service users

Pramod Varma, co-chair of the Centre for Digital Public Infrastructure and former Chief Architect of India’s Aadhaar, opened the Summit with a call to us to:

“Think from a user perspective, put the user at the centre. If we start with thinking about systems and platforms, that’s nothing to do with the person.“

This is a really important reminder: DPI is about reimagining public service delivery for a digital era - delivering services that are responsive, adaptable and impactful for service users. And our metrics need to reflect this.

It can be easy and tempting as a government to focus on technical aspects, like implementation of systems, or on value to government in the form of efficiencies. However, it’s really important to measure value from the perspective of service users. This is how we get beyond measuring adoption, and towards outcomes.

One great example of this is the Bangladesh Government Innovation Lab, which has put citizen-centricity at the core of its definition of success: reduction of the “TCV”, i.e. the time, the costs and the number of visits that citizens need to access public services. The lab has achieved considerable impact on this metric.

The main constraints to capturing metrics are culture and mindset

The biggest constraints to capturing and using metrics, in our experience, are around these two factors.

Firstly, governments don’t always have a culture of measurement in place. As civil servants we can often make assumptions about user needs, or assume that digital investments automatically lead to services that are used by and benefit people.

To tackle this, in the UK’s Government Digital Service, all service owners were required to publish 4 key metrics on public service dashboards - digital uptake, successful completion of services, cost per transaction, and user satisfaction.

This practice of finding, collecting and publishing data embedded a measurement culture that’s actually outlasted the service dashboard itself.

The second challenge is a culture where measurement tends to be disconnected from service delivery. The traditional approach to IT programmes is to start with a massive business case upfront to model the return on investment, then implement a large scale programme over many years. Measuring and evaluating impact happens at the very end.

In a digital era, policy making, service delivery and measurement must become much closer together - start small, test and learn with users, and build in rapid feedback loops to continually iterate and improve services.

A diagram showing rapid feedback loops. — Rapid feedback loops are enabled by small-scale testing and learning

We can then use small, tangible successes against metrics that speak to different interests (political, policy, people) to build buy-in and scale.

As we think about the promise of DPI in the form of ‘whole-of-government’ and ‘whole-of-society’ value, we mustn’t get stuck simply setting out strategies. The best government digital efforts so far have demonstrated what can be achieved when you create a culture that measures the right things, and focussed on taking the first steps towards delivery and impact.

That mindset is needed to realise DPI’s potential in practice, where we think big, but start small and bold.