Evaluation tends to sit in the background of change projects, coming to the surface when evidence is needed to continue the project, pivot in another direction or call it as not achieving the project’s expectations.
Conventionally, it has also operated on a transactional basis – measuring the return on investment made by a funding organisation or corporate of government sponsor.
Recent developments, which we’ll explore here, question both the idea of measuring as well as what is being evaluated and what those evaluations are based on. These are fundamental shifts in the nature of evaluation.
Interesting they align with what we think of as designerly approaches to change, namely that change is iterative rather than foxed; it starts and continually returns to the impact of change on those involved directly and indirectly; and that the process itself is a learning experience, continually reflecting on the experience and what is working, not working or surprising us.
It can be useful to see this as extending evaluation instead of replacing one form with another. In other words, the conventional ROI is still required, though it is enhanced by bringing in new considerations and alternative knowledge systems.
I’ve extracted ideas from two panel sessions(see below) that point to the questions about what and how to evaluate and the emergent practices coming from these explorations.
As Penny Hagen notes, evaluation becomes the learning dimension of change, reviewing and reflecting as we go, making agile moves to improve and discover in a cyclical rather than linear manner and importantly, co-designing how we learn from the process.
In your development of an implementation model, you are asked to embed evaluation into the DNA of the model. That is, to see it as a positive value that allows you to gain a clear view on how the process is going and who it is impacting.
As your project is speculative it is not possible to test your assumptions, in the field. However, it is useful to challenge what you think will work using design research methods. For example, the personas you generate will include people who have alternative cultural knowledge systems. What mechanisms do you have in your process that enables these alternative ways of knowing to drive change? Who do you include in your team or advisors to ensure that there is a deep knowledge sharing, not a tokenistic nod to cultural diversity? What are the‘charters’ you refer to in the formation of your model? For example, the UN Declaration of Human Rights, the UN Sustainable Development Goals, or the Australian Indigenous Design Charter. Recent Government reports are also valuable, such as the Women’s Workforce Participation and Respect@Work.
Sketching an evaluation practice
A good place to start is to sketch out where and how evaluation might work in your overarching implementation model. Map out the skeleton of the implementation model, its stages as you see them, the people/personas, the milestone points and alerts(where you think things could go awry or where there is potential for innovation).
Overlay this with where and what, how and with whom you might evaluate. Do this before listening to the talks below, then return afterwards and consider the new moves you might make using an enhanced evaluation practice.
Once you have a rough sketch of how the implementation model and your evaluation process intersect, you can begin to define the critical moments and objectives for your model.
Begin with a Theory of Change
A Theory of Change:“explains how activities are understood to produce a series of results that contribute to achieving the final intended impacts. It can be developed for any level of intervention – an event, a project, a programme, a policy, a strategy or an organization. A theory of change can be developed for an intervention:
• where objectives and activities can be identified and tightly planned beforehand, or
In other words, a ToC is a way to note down the collective objectives for your project and the means to achieve them. As a theory rather than a plan, it is open to being tested and changed as required. There are many different ways to represent a ToC diagrammatically. A simple map or chart is all that is required for this project.
In Evaluation For Impact, TACSI(The Australian Centre for Social Innovation) uses this diagram to show the alignment between design and developmental and capability evaluative practices(https://tacsi.org.au/journal/design-and-evaluation-for-impact/):
Alternative views of evaluative practices
Romlie Mokak is a Commissioner, Australian Productivity Commission. He is a Djugun man and a member of the Yawuru people, appointed in 2018 as the first Indigenous Policy Evaluation Commissioner. In his role he has given prominence to indigenous forms of evaluation to ascertain the effectiveness of programs for indigenous development, as well as how it might play out in arenas beyond an indigenous focus. He is joined in a seminar by Craig Ritchie, a Dhunghutti man from the Biripi nations, the Chief Executive Officer at the Australian Institute of Aboriginal and Torres Strait Islander Studies(AIATSIS), Adjunct Professor Muriel Bamblett(Victorian Aboriginal Child Care Agency), Dr Ganesh Nana(New Zealand Productivity Commission) and Professor Gemma Carey(Centre for Social Impact) on evaluation from an indigenous perspective.
While there is an acceptance for the need of conventional measures to track the health of indigenous communities, they propose broader, more holistic ways to shift towards a positive evaluation approach. This is achieved by injecting indigenous values into the process.
Craig Ritchie outlines, first, the assumptions made about the underlying belief in colonising power structures that the indigenous population would be eliminated.
An alternative narrative starts to appear when the perspectives of the‘excluded’ are incorporated into the overarching account of how humans live in a society.
Sketching an evaluation practice
Begin with a Theory of Change
Alternative views of evaluative practices