News

Why sense-making must be part of your city’s Monitoring and Evaluation efforts

Conventional Monitoring and Evaluation (M&E) approaches are often designed to measure linear changes over time to help create accountability for whether progress is being achieved. The problem is that, when dealing with non-linear challenges and systems transformation, linear approaches tend to fall short of capturing whether our cities are actually advancing towards sustainable development in a holistic sense. What’s more, do we really follow what our indicators are telling us, or is M&E mainly a box-ticking exercise?

In the second online peer learning session organised by the Tallinn Sustainability Governance Hub on 8 November 2023, we brought together around 30 city practitioners from across 13 cities to unpack the role of M&E in understanding whether transformational change is really happening. This required us to take a step back and ask ourselves: do our current M&E frameworks and approaches foster learning and collective problem-solving capabilities? Do our Key Performance Indicators help to improve sustainability governance? If not, what else do we need to learn to achieve long-term goals?

You can find a recording of the session and all presentations on the event page here.

From results-oriented to reflexive M&E approaches

The session was led by Ryan Bellinson, Senior Research Fellow in Cities, Climate and Innovation at University College London and Principal at Collaboraction Insights, who began by introducing the importance of understanding systems change as an open, non-linear process. He suggested the need to move from a predominantly ‘result-oriented approach’ towards a ‘reflexive approach’ to M&E, which requires a careful look at the values and institution we currently work within.

In essence, this means evaluating a city’s interventions from a dynamic value perspective, insights that draw on research from Ryan and his colleagues on “Mission Metrics” – an innovation approach for using M&E to support long-term learning to accelerate action, applied to the case of Greater Manchester, UK (read this policy brief for further background).

Our M&E approaches hence need to combine various types of indicators, as well as sense-making tools, in order to capture systems transformation. Only considering “direct measures”, like tonnes of CO2 emissions reduced, will say little about the ecosystem-wide effects of, for example, a new low traffic zone regulation. We need to complement this learning-oriented approach with other intelligence coming from e.g. interaction with the community where action takes place.

Next: understanding behaviours and engaging the whole community

After the introductory lecture, Robyn Smith, coordinator for Greater Manchester’s Mission-Based Approach, and Ville Taajamaa,  2030 Agenda manager from the City of Espoo, provided their reflections from working on M&E in the context of systems change.

In Greater Manchester, the biggest change in term of moving to a more learning-oriented approach to M&E has been to integrate M&E more tightly into the governance of its 5-Year Environmental Plan and Climate Mission. Whereas M&E used to be something created after a strategy was already developed and focussed on direct measures, they have gradually worked to make it an integrated part of how the plan is both designed and implemented. The next steps worked on now consider whole systems change more seriously, for example incorporating analysis of co-benefits of CO2 reductions, and addressing areas that are hard to monitor, like behaviour change. This will likely mean the need for developing more sense-making capabilities within the city administration and in its interactions with local communities.

In the city of Espoo, one central question for the city that has already produced two Voluntary Local Reviews of its local implementation of UN SDGs, is how to do this process better together with the community. This implies a reflexive approach to M&E like the one Ryan presented.

Group reflections: is sense-making part of the game?

During the final hour of the session, participants worked in small groups to discuss how their current M&E approaches reflect the three layers of direct and dynamic indicators and sense-making tools.

Most of the participants agreed that many of the existing indicators used in their cities include a mix of direct and dynamic indicators. But the common sense was that there is often a gap in how indicators are used to guide actions towards systems change. One city representative mentioned that the creation of narratives from direct indicators can make the process move to the sense-making layer and actually create understanding of holistic sustainability.

Another challenge raised in the groups was that data on some indicators are not available frequently enough to use for e.g. changing course of direction of strategic interventions, and some important dynamic indicators may not be under the direct influence of city administration. To what extent does it make sense to still monitor them? What is the story that can be told about them in relation to the city’s strategic work and sustainability governance?

When it comes to what cities need to learn through their M&E approaches, many participants shared that the visualisation, communication and political commitment to act on demonstrated results is perhaps more pressing than gathering new data. On the one hand, communicating results better could help to reduce public disbelief in government agendas, but sometimes even where there are clear data points showing undesirable results, the call to action by political leaders is weak. A culture of data-driven decisions is lacking.

Looking ahead, accessible dashboards, story-telling techniques and improved understanding of collective impact and unintended consequences of policies were some of the learning needs highlighted in the group discussions.

All in all, the need for cities to invest in analytical, sense-making and story-telling capabilities appear to be one main conclusions drawn from the session. The use of AI tools will be an interesting aspect to consider in the future, as we continue to develop our understanding and framing of sustainability governance in cities.

Follow our work

The Tallinn Sustainability Governance Hub will continue to be active beyond the European Green Capital year 2023, with more sessions to deep-dive into topics relevant for how we can govern sustainable development and close the action gap towards the SDGs.

If you want to stay updated, please drop an email to: Stina.Heikkila@tallinnlv.ee

More information and reading  

Bellinson, R., Berry, M. and Cadelli, E. (2023). Mission metrics: Policy evaluation tools for cities to optimise learning for the green transition. UCL Institute for Innovation and Public Purpose, Policy Brief series (IIPP PB 23). Available here.

Reach out to Ryan here: rbellinson@gmail.com

Launch of Tallinn’s Peer learning hub on Sustainability Governance during the Urban Future conference in Stuttgart, Germany. Read about it here.

Espoo Voluntary Local Review. Available here.