Evaluation - Focus on 'Diversity'

I delivered a webinar for the Diversity Arts Australia Fair Play Program on evaluation with a focus on ‘diversity.’ You can access the slides here. I have copied in the main ones below.

There is also a lot more detail on various aspects of evaluation in the other resources on this website if you want to get into the nitty gritty of developing an impact framework or understanding theory of change.

Evaluation is an exercise in power

When evaluating a project which involves people with lived experience of marginalisation, it’s very important to understand that evaluation is not neutral, but is an exercise in power. It can be an exercise in sharing power, giving it away; or it can be an exercise in entrenching power, and maintaining the status quo.

I recommend that before you start the process of evaluation, ask yourself honestly:

Will your evaluation itself make things better or worse for the people you would like to help?

As a rule of thumb, when you go into evaluation of projects with people with lived experiences of marginalisation, remember your starting point should be: justice, co-design, working with evaluators with lived experience, and ownership.

When you decide what to evaluate, the most important thing to do is: ask the intended ‘beneficiaries’ of the project. I use quotation marks for ‘beneficiaries’ because it is not necessarily the case that your project is benefiting others more than it is benefiting your own organisation - see above re power.

What do the people, who are the intended beneficiaries of this project, want to see change?

There are a bazillion ways to collect data. The main ones are: self-assessment, observations (be super careful with these, so you don’t disempower the people you are trying to support), interviews and questionnaires.

I generally recommend questionnaires (which someone asks you) over surveys (which you send out and hope people respond to). This is because people with lived experiences of marginalisation tend to be under-represented in survey responses.

Play devil’s advocate. Try to prove the opposite of what you hope are the outcomes of your project.

All the way through the evaluation you need to check in on a range of ethical considerations around consent, privacy, risk of harm and power.

Make sure you have put in place the support people need to respond to your evaluation.

When you design your tools, make sure you work with ‘critical friends’ who also have lived experience of the forms of marginalisation of the group you are working with. And be prepared to change things.

We often use a ‘before and after’ tool, which allows people to jot down words or do drawings of how they felt ‘before’ and ‘after’ a project experience.