Do arts interventions actually change the way people behave?

Can a community arts event actually get people to do more exercise? Does a digital app which encourages you to dance at random hours of the day improve your sense of social connection? If you run around with a story and a list of clues, do you develop a greater sense of belonging in your town? This article summarises some research we did in the context of arts for health promotion to develop a framework for evaluating the impact of VicHealth's active arts strategy. I am sharing it because it might help a little with your thinking if you are trying to work out how to find out if your arts activity actually changes the way people behave, without the benefit of a randomised controlled trial (research nerd joke ;-). If anyone has further insights to share, please do!

The social model of health

Health promotion activities take place within the context of the social model of health attempts to address broader influences on health, such as social, cultural, environmental and economic factors, rather than respond directly to disease and injury. It focuses on policies, education and health promotion to effect social change to provide the prerequisites for health.

Social models of health attempt to:

• Change the broader determinants of health e.g. gender, socioeconomics, built environment, social inequities

• Empower individuals and communities with skills, knowledge and self-efficacy to make positive health decisions

• Access to health care, addressing physical and cultural barriers e.g education, location, language, culture

• Collaborate across departments and stakeholders

Health promotion models and theories of change

Health promotion models and theories of change vary in terms of their emphasis. Earlier theories tended to focus on personal decision-making behaviours, whilst later theories take into account personality factors, self-efficacy, intrinsic and extrinsic motivations to change. Broadly speaking, health promotion models and theories of change can be grouped as follows:

• Behavioural change theories

• Ecological theories and models

• Planning models

• Communication theories

• Nursing theories of health promotion and change (Raingruber, 2014).

How health promotion theories can inform arts intervention theories of change

When we were developing the VicHealth Active Arts Strategy theory of change, we identified the following as particularly helpful and apposite health promotion theories:

• Social cognitive theory

• Self-determination theory

• Salutogenic theory

• Diffusion of innovations theory

Social cognitive theory(SCT) is based on vicarious learning. Behaviour is learned via observation, imitation, positive reinforcement and noticing the benefits to others. SCT emphasises self-efficacy, which refers to a person’s sense of confidence to act (Raingruber, 2014: 58-9).

Self-determination theory focuses on intrinsic motivations for action. People who are intrinsically motivated are interested and satisfied by doing the activity, rather than focusing on an extrinsic reward such as recognition or income.

Intrinsic motivation can flourish when a person has a sense of connection with others, a sense of agency in their own lives and a feeling of competence to deal with one’s environment and produce positive outcomes (Raingruber, 2014: 59).

Salutogenic theory focuses on building people’s ‘strong sense of coherence,’ which results in seeing the world as meaningful and manageable. Coherence can be developed through positive relationships and meaningful pursuits (Raingruber, 2014: 67).

Diffusion of innovations theory describes the stages of change as: knowledge, persuasion, decision, implementation and confirmation. A health promotion program would be adopted if it could demonstrate that it is: better than other options; compatible with existing values and needs; trialable; producing observable results; easy to use, understand and communicate; adoptable with a minimal investment of time and risk; and usable with a moderate level of commitment. Roles in the adoption of innovations (including innovative health approaches) include opinion leaders, change agents and change aides.

Approaches to the Evaluation of Health Promotion Impacts

There are a number of different approaches to evaluating the impacts of arts-based (and other) health promotion activities. The key factors in choosing an approach are:

• What is the purpose of the evaluation?

• What are our resources the evaluation?

• Are the causal links between proximal indicators to distal outcomes already established?

Three of the main evaluative approaches which we tend to adapt and draw from when measuring the social and behavioural impact of an arts program include the following:

• Incremental evidence evaluation (commonly known as a program logic or impact evaluation)

• PRECEDE/PROCEED model

• RE-AIM framework

Incremental evidence evaluation (program logic / impact evaluation)

In this approach, researchers gather evidence about the effectiveness of an health promotion program or intervention in its real world setting, effecting impact on a behavioural determinant of health outcomes (Nutbeam, 1998). It can be understood as a program logic evaluation, or impact evaluation, where the desired impacts of the program are also determinants of positive health outcomes (Thorogood and Coombes, 2010: 12).

This approach allows researchers to evaluate the effectiveness of programs happening in the real world, rather than focusing on expensive, experimental situations such as random-controlled trials. Change occurs on a continuum. If links are already well-established, then you do not have to prove causal relationships again. Instead, you can focus on proximal indicators which are known to be related to distal outcomes (Green and Tones: 1999).

In deciding what level of outcome to measure, researchers consider how the evaluation will be used. The evaluation can occur at various levels of complexity, as follows:

• Process evaluation – how was the program carried out?

• Adequacy – did the expected changes occur?

• Plausibility – were the outcomes actually due to the intervention vs external factors?

• Probability – did the intervention have a (health) effect?

PRECEDE-PROCEED Model (PPM)[1]

The PPM model aims to describe proximal, intermediate and distal outcomes of health promotion programs. The approach is based on the assumption that interventions will be effective if they:

• Come from the community

• Are well planned

• Are based on data

• Are seen by the community as feasible

• Include multiple strategies woven together

• Rely on feedback and progress evaluation (Green and Kreuter, 1992)

The PRECEDE part of the model focuses of educational factors that influence change, whilst the PROCEED part looks at the importance of ecological factors.

The model involves the following steps:

1. Social assessment: community members identify their own health promotion needs

2. Epidemiological assessment: Identify the health problems of the community using national statistics

3. Behavioural and environmental assessment: Identify factors contributing to the problem. Factors are ranked according to importance and feasibility of changing them. Most changeable and most important factors are priority targets.

4. Educational and ecological assessment: Identify predisposing, reinforcing and enabling factors for the desired health behaviours, and develop measurable objectives.

5. Administrative and policy assessment: Design interventions and the resources, organisational changes and circumstances required for success.

6. Process evaluation: Evaluating the implementation of the intervention.

7. Impact evaluation: Evaluating the impact of the intervention on the reinforcing, predisposing and enabling factors, behaviours, lifestyle and environment.

8. Outcome evaluation: Evaluation of health outcomes.

The RE-AIM Framework[2]

This framework is designed as a planning tool to select between health promotion projects for investment. Projects are assessed according to five dimensions:

1. Reach: how many people will be influenced by the program

2. Efficacy/effectiveness: produces positive outcomes with few unintended consequences

3. Adoption: participation rate and representativeness

4. Implementation: process evaluation as to whether the program was implemented as intended

5. Maintenance: long-term utilisation of the given health behaviour, and whether a program is sustainable even if resources change.

[1]PRECEDE stands for ‘predisposing, reinforcing and enabling constructs in educational/environmental diagnosis and evaluation.’ PROCEED stands for ‘policy, regulatory, and organisational constructs in education and environmental development.’

[2] RE-AIM stands for reach, efficacy/effectiveness, adoption, implementation and maintenance (Glasgow, Vogt and Boles, 1999).

In conclusion

Most evaluations of arts interventions happen on limited budgets and timeframes. You may not have a large sample population and a large budget, but don't despair! Evaluation is still possible and worthwhile, because you may be able to evaluate whether your project is contributing to the determinants of behavioural change. And that is the start of the answer.

References

Bailey, Jackie and Yang, Hung-Yen. (2014). Play Me I’m Yours: Evaluation Report and Appendices. Arts Centre Melbourne: Melbourne.

Foreman-Wernet, L and Dervin. B. (1011). ‘Cultural Experience in Context: Sense-Making in the Arts.’ The Journal of Arts Management, Law and Society 41: 1-37.

Gilmour, J, MacDowall, L and Oliver, J. (2013). The Arts, Physicality and Connection: An Evaluation of VicHealth’s MOTION Program. VicHealth: Melbourne.

Glasgow, R, Vogt, T and Boles, S. (2011). ‘Evaluating the public health impact of health promotion interventions: the RE-AIM framework.’ Social Science and Medicine 73: 383-390.

Green, J and Tone, K. (1999). ‘Towards a secure evidence base for health promotion.’ Journal of Public Health Medicine 21(2): 133-9.

Green, L and Kreuter, M. (1992). ‘CDC’s planned approach to community health as an application of PRECEDE and an inspiration for PROCEED.’ Journal of Health Education 23(3): 124-147.

Kelaher, M, et al. (2014). ‘Evaluating community outcomes of participation in community arts: a case for civic dialogue.’ Journal of Sociology 50(2): 132-149.

Loy, C, et al. (2013) ‘Crowd Counting and Profiling: Methodology and Evaluation.’ In Modeling, Simulation, and Visual Analysis of Large Crowds. Ali, S et al. (Eds). Springer: New York. 347-382.

McCarthy, Kevin, et al (2004). Gifts of the Muse: Reframing the Debate about the Benefits of the Arts. RAND Corporation: New York.

Nutbeam, D. (1998). ‘Evaluating health promotion – progress, problems and solutions.’ Health Promotion International 13: 27-43.

Radbourne, J, et al. (2009). ‘The Audience Experience: Measuring Quality in the Performing Arts.’ International Journal of Arts Management 11(3): 16-29.

Raingruber, Bonnie. (2014). Contemporary Health Promotion in Nursing Practice. Jones & Bartlett Publishers: Burlington MA.

Sherwood, N and Jeffery, R. (2000). ‘The Behavioural Determinants of Exercise: Implications for Physical Activity Interventions.’ Annual Review of Nutrition 20: 21-44.

Synergistiq. (2014). Evaluation of MOTION: Final Report. VicHealth: Melbourne.

Thorogood, Margaret and Coombes, Yolande. (2010). Evaluating Health Promotion: Practice and Methods. Oxford University Press: Oxford.

Victorian Government Department of Human Services. (2003). Measuring Health Promotion Impacts: A Guide to Impact Evaluation in Integrated Health Promotion. Department of Human Services: Melbourne.

Watson, R and Yip, P. (2011). ‘How many were there when it mattered? Estimating the sizes of crowds,’ Significance, September, 104-107.

Hearts and minds - an explanatory model of impact

Here is something I have been toying with for a while - how to get into one diagram the way Sarah, Yen and I think about impact.

When I talk about evaluation (to anyone who will listen to me ramble about it - mostly social and health workers, mental health programmers and arts professionals), I describe the impact of an experience or program in terms of how it affects an individual's:

• heart (their emotional response)

• mind (the person's intellectual and attitudinal change)

• body (behaviours - what the person does)

• spirit (aesthetic change, unitary change - i.e how all the changes coalesce to change a person)

• world (social, community and cultural change which results from the program or experience)

Sometimes evaluation can scare people with its academic language. Of course it is important to have theoretical rigour. We can have this, and still use a language and framework that makes sense to people intuitively.

I am not much of an illustrator, so here it is in smart art! It's a work in progress...but hopefully the model can help if you are trying to conceptualise and articulate the impact of  what you or your organisation does, and then evaluate that impact.

Click here for a PDF of the model

Hearts and minds, people. Hearts and minds.

Validated instruments to use when assessing the impact of the arts

Hi guys, There was a lot of interest in my blogette about resources for measuring cultural value, so I thought I might share some other tools which I find quite helpful in BYP Group's work around assessing and measuring the impact of the arts.

I have compiled a table of useful and even better, VALIDATED instruments which might be used when assessing the impact of the arts on personal capacity development e.g. self-esteem, self-efficacy, self-concept, emotional regulation, theory of mind, creativity etc.

Validated-Arts-Research-Instruments-Table

Enjoy! And of course email or comment on this blog if you have any more to add to the list or have any questions.

Jackie

BYP Group does not take responsibility for the accuracy of the information contained in external links.

Useful resources about quantifying cultural value

A quick scratch pad of useful reports to do with quantifying the value of arts and culture (and also a bit about alternative financing for a bit of light reading).

Understanding the alternative finance market (NESTA 2016)

Quantifying the social impact of culture and sports (UK 2014)

Quantifying and valuing the wellbeing impacts of culture and sport (UK 2015)

The 2015 report of the Warwick Commission on the Future of Cultural Value (UK)

Validating the links between arts and liveability (US)

The Matrix for Making Decisions

A friend of mine contacted me today, saying she wanted to do 'the Bailey matrix' to help her decide whether to start her own business. The matrix is something that, back when we were graduates in Canberra almost 20 years ago, I would draw up for friends in the hope of it helping them decide what they wanted to do next with their lives. Using a spreadsheet.

Another friend, Andrew Dempster, used a similar matrix which we all called the Dempster matrix (Hi Andrew!). I can't remember now if it was Andrew or I who innovated the weighting approach - where you weight different options according to their fulfilment of various important goals. It doesn't really matter who did it - but it was a turning point for the utility of the matrix, allowing you to give a higher score to things which mattered the most to you. Effectively, it introduced the variable of subjective value into a spreadsheet.

All jokes aside, the matrix is actually a very useful tool for making decisions. Not so much that you use the scores religiously to determine what job you might take, but that the process itself helps you to clarify whats important to you, and how various options will help you to achieve those things.

I actually use a variant of the matrix these days with clients when trying to establish if their program budgets are going towards the right areas, based on their overall goals or mission. It is slightly more complex - it includes things like duration of engagement, level of subsidy, and stakeholder target groups - but the principle is the same. It is a way of measuring your actions (real or proposed) against your values. Yes, using a spreadsheet.

So here it is in all of its glory - the Matrix of Making Decisions. Use it as a guide only :-).

I am happy to share the Matrix under a Creative Commons License - Share Alike/ Non-Commercial. This just means, please don't go out and use it to make your millions (or if you do, please tell me how you managed it. I've only ever gotten the occasional beer or block of Dairy Milk from it ;-).

• The Matrix

Jackie Bailey

Principal, BYP Group

Validated instruments for use in arts impact research

I was putting together a table of various validated instruments which have been or could be used in arts impact research, and thought it might be useful for others working in this area. The table is a list of instruments which can be used to measure empathy, self-esteem and self-efficacy, wellbeing and so on. I have also included links to the instruments if they are available for free or purchase online.

You can access the table in html or pdf.

• Validated Arts Research Instruments Table - html
• Validated Arts Research Instruments Table - pdf

Please feel free to comment - the list is not exhaustive and doubtless there are some in there which have more or less efficacy than others!

Happy measuring :-)

Jackie Bailey, Principal, BYP Group

Can we measure the value of enduring artworks?

(This post is a continuation of my ruminations on Ann Markusen's work)

There is something else I saw in Ann Markusen's report on California's arts and cultural ecology which I found really interesting.

I saw these words:

'California’s arts and cultural nonprofits generate new and enduring artworks—they commission an estimated 41,000 theater, dance, musical compositions, and artworks annually (p.33)'

and had one of those moments of nerdy, impact evaluation realisation (you know the ones ;-).

I have never tried to capture the value of the enduring nature of works of art. Sure, we all talk about heritage value, bequest value etc (thanks to John Holden). But normally when I do impact or value assessments for arts clients, I am very focused on the intrinsic experience of the art and the social, educational and personal outcomes for the participants.

I have always included 'contribution to society and culture' as an element of Artistic Vibrancy (a framework for measuring the health and impact of an arts organisation). But I never thought of measuring the contribution to culture through the creation and continued public experience and enjoyment of enduring works.

How can you value the contribution of Shakespeare, for example? Well, it's invaluable - as you can can always say with the arts. But you can try to translate the value of enduring works into a language that funders can work with ('Invaluable' does not cut it in the Cabinet room).

For example, could you start to count the number of works likely to endure over say, 2, 3, 5 years and longer? You could probably come up with a statistical formula based on big data (PhD project anyone?!) And then how many people are likely to see it, engage with it, perform it, derive some meaning from it, and be transformed by it (a kind of pyramid effect)?

I think this would be completely awesome or terribly hard to believe ,depending on how carefully it is done. There is nothing worse for arguments about arts impact than seemingly ambit claims of creating $X in value. But when it's done well, it's - well, invaluable.

I'd love to hear other evaluators' thoughts on this. Do you think it could be done and done well? Or perhaps it has been done?

Jackie Bailey - Principal, BYP Group

Appreciative Inquiry - change and evaluation rolled into one

We are often asked about the different types of evaluation approaches.  We will try to summarise them from time to time here, on our site, and hope this helps people sort through the jargon of evaluation and get to the bottom of what suits you and your needs best. We have started with an explanation of Appreciative Inquiry, because I was asked about it at a presentation in Melbourne to arts organisations.

What is an Appreciative Inquiry Evaluation?[1]

An Appreciative Inquiry evaluation is based on the principles of Appreciate Inquiry (AI), an organizational development method which is focused on building on what an organization does well.  The principles of AI are summarized below.

In an AI evaluation, evaluators and participants work together to share their views of the present, and “co-create” the future. An AI evaluation does not ignore problems, but approaches them as opportunities for change.

In an AI evaluation, the evaluator and participants:

• become fully engaged in the learning journey
• work to “co-create” the future
• acknowledge that there are multiple, equally valid interpretations of reality
• share their individual interpretations of reality, with an aim to gain a shared understanding of experiences
• envision possible positive futures which build on present strengths
• use language and foster relationships which create that positive future

In a pure AI evaluation, traditional evaluative methods – eg qualitative and quantitative research – are used only as the need arises, and are driven by participants.

Principles of Appreciative Inquiry

“Appreciative inquiry” is an approach to evaluation based on the assumption that an organization wants to improve.  Accordingly, the evaluation has a fundamentally positive focus on what the organization does well, and how it can build on this.

The core principles of Appreciative Inquiry are:

1. Constructionist principle: people’s realities are “constructed” through their social interactions.
2. Simultaneity principle: change and inquiry are simultaneous.  Inquiry can itself effect change.
3. Poetic principle: the “story” of an organization is a product of the ongoing narrative of its members and others.
4. Anticipatory principle: envisioning a positive future can help to guide people towards one.
5. Positive principle: focusing on the positive can help create a positive energy for the future.
6. Wholeness principle: wholeness brings out the best in people, so supporting people to share the whole story from a position of individual wholeness can build a “collective capacity for change.”
7. Enactment principle: positive change occurs when people create the future through their words, images and relationships.
8. Free choice principle: free choice stimulates positive change and liberates personal and organizational power.

---

[1] This explanation of AI is drawn from Howieson, Jill, “A Constructive Inquiry approach: blending Appreciative Inquiry with traditional research and evaluation methods,” Evaluation Journal of Australasia, 11(2) 2011.

Libraries conference - presentation

I gave a presentation yesterday at the NSW Public Libraries Association annual conference, 'SWITCH2015.' I talked about how to conduct 'meaningful measurement' in the libraries context. I have uploaded the presentation here (see the link, below) for those keen on seeing my meandering thought processes up close and personal.

Download Meaningful measurement - library conference presentation November 2015

Social Return on Investment and other stuff

I was just bookmarking a few useful guides and tools for SROI and the like, and then thought, if this is useful for me, why not share it? So here are a few of my favourite links (is that a song...?) and apologies if I am telling you stuff you already know.

• A pretty comprehensive guide to SROI

• A 'how to start' guide to SROI

• Outcome indicators and banks - a guide to where to find them online (a bit out of date but still useful)

• Global Value Exchange - a crowd-sourced outcomes indicator bank

Other awesome websites:

• Better Evaluation (international collaboration with loads of resources about evaluation)

• New Economics Foundation (I am a fan-girl of this UK think tank about economics 'as if people and the environment matter')

How do the arts really change the world? European report recommends experimental research to answer this question.

A European report has just been released which reviews the available evidence and methodologies for cultural and creative 'spillovers' into other areas of society. The researchers, who included the creative industry veteran Tom Fleming, looked for evidence in:

• knowledge spillover (new ideas, innovations etc from the arts and cultural sector which might spillover into the wider economy)
• industry spillover (e.g. productivity and innovation from having dynamic cultural presence)
• network spillover (e.g. impacts on society and economy from the presence of cultural clusters)

This might not surprise you, but Fleming and his team found that there is not enough evidence in all the many research articles out there to demonstrate a causal link between the arts/culture and many claimed spillovers. The research for these areas were persuasive, but fell short of proving causality.

He did find that there were several areas where causality had been established to a scientific standard:

• communications can be boosted within organisations
• culture-led regeneration has a positive impact
• cross-fertilisation occurs between commercial and no=commercial sectors
• investment in design has an impact
• spillovers play a role in boosting uptake of new technology
• networks are important in spreading innovation

We were excited to see Fleming recommend the following to really look at if and how arts/culture can cause social change:

• experimental studies which include control groups
• action research, testing hypotheses through interventions over time
• proxy research approaches

We have our fingers crossed extremely tightly that some funding goes towards this in the academic and cultural space. As we all know, the kind of research to find out if the arts 'changes the world' and if so, exactly how and where and when and who, costs the kind of money most arts organisations don't have.

At the moment what we mostly do in our evaluations is look at the existing research into the determinants of particular social outcomes e.g. wellbeing, inclusion, and see if we can hypothesise a sound theoretical chain of contribution to these determinants by the arts activity. Or not.

This is a perfectly acceptable way of evaluating arts projects (and is used in other areas with similar complexity around causation, such as health promotion). But you do need the big research on which to rely upon when looking for whether your arts project is contributing to the determinants of wellbeing, inclusion, increased productivity, or whatever it might happen to be. So let's hope academics and institutions can support Fleming's recommendations and put the question to rest (or perhaps, to work) - how do the arts really change the world?

Evaluating arts and social inclusion projects

Yesterday I gave a presentation at the Creative Victoria Expert Arts Panel session on evaluating arts impact. Along with me, Deakin Uni's Hilary Glow and Anne Kershaw presented about a recent evaluation they did for Vichealth on arts and wellbeing, and Mark Hogan from Regional Development talked about the Clunes Booktown regional transformation story. Once Creative Victoria upload the full session to the web I will upload a link. In the meantime, I have uploaded my powerpoint presentation here.

Thumbnail guide to evaluation

This is just a little something I thought I would include here. I have been working on it for a client and thought it might be useful to others too. Nothing top secret, so completely fine to share.

Summary of key types of evaluation

Type Description Resources Formative Evaluation

This is a tool to establish the needs and baseline of the community before a project is designed and commenced. It can include:

• A needs assessment

• Community consultations about needs and priorities

• Benchmarking research, e.g. surveys of the community about characteristics which the project will aim to change (e.g. wellbeing, social inclusion, arts access)

Mission Australia Youth Survey

ABS Community Profiles

The Warwick-Edinburgh Mental Wellbeing Scale

Process Evaluation

This looks at how the project was implemented. It can include:

• program fidelity – the extent to which the process was implemented as intended

• feasibility – problems and solutions that were implemented along the way

• reach, equity, intensity of involvement – the level of uptake and participation in the project

UNSW Centre for Social Impact guides

Impact Evaluation

This looks at the impact the project has had over the short, medium and long-term. In this approach, parties agree on a ‘theory of change’ – what they think will happen as a result of the program. Then they look at whether this occurred, and what unintended outcomes may have arisen. They look at impacts for each major stakeholder group e.g. participants, partners, community groups, donors, and the arts organisation. Some evaluators use a 'Program Logic' approach, or a ‘Most Significant Change’ approach, where they try to identify the key change resulting from the project for each stakeholder group.

Methods of measuring impacts will depend on access to participants and what can be feasibly and reliably measured. Methods regularly include a review of participation numbers and data, surveys, interviews, ethnographic observations and focus groups.

UNSW Centre for Social Impact guides

Upcoming BYP Group presentations

We will be presenting at a number of events over the coming months. We will attempt to upload any powerpoint slides to this website, but just in case you think can come to any of the below, here are the details:

Jackie will be a panellist on an Expert Arts Panel on evaluation and the arts, Creative Victoria, Melbourne, 28 October 2015

Jackie will be delivering a Keynote Address - 'Meaningful Measurements - How do we articulate the value of public libraries in a contemporary context?' SWITCH 2015 - NSW Public Libraries Association, Sydney, 17-20 November 2015

Sarah will be delivering a free regional workshop for Creative Victoria - 'Evaluating your Work - Measuring Success in Arts and Culture,' in Kilmore, Victoria, Friday 27 November.

Jackie will be hosting the panel - 'Evaluation and Research - 6 Minute Learnings' at the Australia Council for the Arts Arts Learning Forum, Footscray, Melbourne 25-27 November 2015

Impacts of poetry in schools

Key findings from our evaluation of the Red Room Company's schools poetry education program have now been published as part of the Red Room's annual report. We interviewed students and teachers and conducted a survey of participants from the 2014 program. The company is using the findings in strategic planning and philanthropic work. You can read the key findings and more about the Red Room Company here.

If you would like to talk to us about the evaluation or similar work, please contact Jackie jackie at bypgroup dot com or +61 428 576 372.